New talk from Alireza Farshin and Luigi Rizzo.
NIC link speeds are moving quickly toward the 1 Tbps per-port range and higher! These speed increases open up newer bottlenecks in the communication between system components ( i.e PCIe bus, address translation (IOMMU), memory fabric, and memory chips) that we never had to worry about at lower speeds.
In this talk Alireza and Luigi focus on the IOMMU and the associated IOTLB contributions to the new bottlenecks. The IOTLB size tends to be small and depending on the pattern of IO requests, it is possible to overflow the cache and cause a dramatic drop in performance. The authors refer to this threshold phenomena as the "IOTLB wall". For instance, the buffer management (e.g., PagePool API in the Linux kernel) and various offloading capabilities of modern NICs (e.g., LRO and TSO) could affect the memory request pattern, and consequently the number of IOTLB misses.
Alireza and Luigi describe the IOTLB behavior and its effects on recent Intel Xeon Scalable & AMD EPYC processors at 200 Gbps, by analyzing the impact of different factors contributing to IOTLB misses and causing a throughput drop. The authors then proceed to prescribe how the problem can be mitigated with various techniques.
More info: netdevconf.info/0x18/7
cheers, jamal