Hi Simon,
On Wed, Feb 21, 2024 at 8:48 AM Simon Leinen simon.leinen@switch.ch wrote:
Jamal Hadi Salim writes:
I have a different perspective on what you said on DPUs increasing the power base. Consider a 2x25G BF2 which sucks power from the PCI3 bus at ~45W. _Under load_ there is a clear win in experiments we conducted. A "high load enough" which offloads ACLs and TLS would cost almost ~100W if running on the host. The operative term is "under load".
You could argue even if you used a plain non-smart NIC on PCI3 it would still consume 45 W to turn on.
Well you could *claim* that, but it seems unrealistic. Looking at datasheets, I see 20.8W MAX for a 2x25GE Intel E810 card, 11.1W/12.9W for a Broadcom 957414A4142CC-DS 2x25GE card under 100% traffic load (though the datasheet reads a bit dubious in that it only talks about a single DAC cable/SFP28 transceiver...)
So I claim that there *is* an increased base energy cost that you pay for those smart NICs compared with less smart ones (apologies to the Intel and Broadcom NICs, you're also smart! Just not *that* smart ;-).
You make a good point. The cabling contribution (even for DAC) is never factored fairly in those marketing^Wproduct brochures. There is an argument still to be made for xPUS: they dont need fans which is a big cost. I think our measurement of 100W vs 45W included the fans because we sample the power bar.
To summarize: Paper claims: you benefit when your cluster is in constant churn at > 50% of capacity Simon: Nobody is ever running at > 50% 24/7.
cheers, jamal
You can play with PCI registers to lower the power consumption for the non-smart NIC but that comes with a cost of increasing latency and other issues (I dont remember whether Jesse mentioned mucking with ASPM in PCIE).
OTOH, i have seen: once you start going past PCIE3, these xPUs now provide an extra cable you connect to the motherboard to draw extra power (very similar to GPUs). When asked, vendors would say "it's just insurance in case we need more power on overload" - it's hard to judge if the "truth".
Hmmmm...
Simon.