New talk from Alireza Farshin and Luigi Rizzo.
NIC link speeds are moving quickly toward the 1 Tbps per-port range
and higher! These speed increases open up newer bottlenecks in the
communication between system components ( i.e PCIe bus, address
translation (IOMMU), memory fabric, and memory chips) that we never
had to worry about at lower speeds.
In this talk Alireza and Luigi focus on the IOMMU and the associated
IOTLB contributions to the new bottlenecks. The IOTLB size tends to be
small and depending on the pattern of IO requests, it is possible to
overflow the cache and cause a dramatic drop in performance. The
authors refer to this threshold phenomena as the "IOTLB wall". For
instance, the buffer management (e.g., PagePool API in the Linux
kernel) and various offloading capabilities of modern NICs (e.g., LRO
and TSO) could affect the memory request pattern, and consequently the
number of IOTLB misses.
Alireza and Luigi describe the IOTLB behavior and its effects on
recent Intel Xeon Scalable & AMD EPYC processors at 200 Gbps, by
analyzing the impact of different factors contributing to IOTLB misses
and causing a throughput drop. The authors then proceed to prescribe
how the problem can be mitigated with various techniques.
More info: netdevconf.info/0x18/7
cheers,
jamal