There was an interesting talk in Kubecon about power management: https://www.youtube.com/watch?v=2GQnzZhmGRc

Apparently the HPC guys have been thinking about this problem for quite some time. There's a tool from Intel (hey Jesse!) called GEOPM that can model power consumption and be used to forecast the tradeoffs! With these AI farms getting bigger and bigger, seems like a lot from HPC can be applied here.