General

Why goodput matters more than throughput for LLM serving

When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to compare, and it…

Quelle: Originalartikel öffnen

AI Assistant
Context loaded: Why goodput matters more than throughput for LLM serving