Why goodput matters more than throughput for LLM serving

When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to compare, and it…
Quelle: Originalartikel öffnen




