Shared GPUs without shared latency: Priority-aware serving in Red Hat AI

Chat, batch, and agentic traffic can run on the same graphics processing unit (GPU) without the chatbot waiting for the batch job to complete. Built on llm-d flow control and generally available in Red Hat AI 3.5, priority-aware serving decides which request goes first when the pool is full, and applications change nothing. On a NVIDIA H100, 95% of premium requests got their first token in under half a second, and batch rejections fell from 48,224 to 0.Every separate pool…

Quelle: Originalartikel öffnen