... but now I have an issue with excessive retries.
Retransmissions are one of ways for TCP to throttle back. And they indicate that it's not the first leg from transmitter which has (performance) problems.
You can try with UDP connectivity ... start with modest bandwidth setting (e.g. 2Gbps) and gradually increase it. At some certain point you'll start seeing missing packets in receiver's report.
It's hard to tell where the bottleneck lies, but it's quite likely it's the receiver itself, even basic processing of multi-Gbps is not eaxactly peace a cake. Definitely check port stats on switch, if it starts to show dropped packets it will help to narrow down on the location of bottleneck.
So another possibility is to run TCP tests with multiple parallel streams. It gives opportunity to both sender and recriver to process data using multiple CPU cores in parallel. Then you check the sum of all parallel streams.
Re. MTU sizes: switches mostly operate in "store and forward" mode, they first receive full frame, store it in buffer and then forward it through egress port. So switches have to assume some maximum frame size in order to establish some granularity of buffer space. And this granularity is in MT world named L2MTU. Any frame, larger than that, arriving on ingress port will get dropped due to inability to store it. It can be adjusted according zo needs but there's upper limit and it depends on switch chip (some switch chips don't even support jumbo feames, some support sizes up to around 4kB, etc.).
So when going for jumbo frames, one has to check all L2 devices about maximum L2MTU ... and then set thrm to equal values. When considering setting (L3) MTU, one has to consider ethernet headers (minimum of 18 bytes).
Another thing to consider: the most straight-forward thinking would be to set L2MTU to switch chip max supported value everywhere. That would work but would waste buffer space - due to granularity. E.g. setting L2MTU to 9kB instead of 4.5kB while actual frame sizes are up to 4kB means only half the maximum number of buffered frames.
Statistics: Posted by mkx — Wed Feb 05, 2025 10:10 am
