Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP
Add admission control at ingress so requests are admitted, deferred, or rejected before they consume GPU memory.
- Estimate request cost and check live engine signals; request counts alone do not reflect GPU capacity.
- Return 503 with Retry-After and jitter for temporary deferrals; use 429 for caller limits and 529 for pool overload.
- Benchmark against replayed agent traces using goodput across all arrivals, including rejected requests.
The talk includes a concrete benchmark setup and GPU results that add value beyond the notes. · Best for: Teams self-hosting LLMs for coding agents or other bursty workloads.
The speakers explain why self-hosted inference systems need an ingress admission layer, how to choose signals and policies, how clients should handle decisions, and how to benchmark policies. They demonstrate AdmitPerf and report a small real-GPU test.
Inference capacity and burst traffic2:45
- Inference generates output token by token; KV-cache memory grows with the request and remains reserved while it runs.
- A request's output length is unknown in advance, so admitting it commits resources for its full lifetime.
- Coding agents can generate dozens of calls in seconds, creating bursts that exceed deployment capacity.
Admission control at ingress8:49
- Admission happens before requests take GPU memory and can admit now, defer, or reject.
- The ingress layer can see both request details and engine state without modifying the inference engine.
- Useful signals include input length, tenant/session, deadline, KV-cache use, queue age, cache hits, and replica load.
Responses and client behavior21:09
- On 200, start the timeout clock when the first token arrives; close the connection if abandoning the request to free its cache.
- For 503 deferrals, wait for Retry-After plus random jitter before retrying.
- Treat 429 as caller overuse and reduce concurrency; treat 529 as provider overload. Use exponential backoff, jitter, and a retry budget.
Policy choices and research approaches27:40
- Choose the admission unit (request, session, or tenant), signal, decision rule, and action.
- Request-level admission can let parts of multiple agent tasks in while leaving all tasks incomplete; session-level admission can allow one task to finish.
- Choose signals based on the failure to prevent: KV pressure, queue depth, cache reuse, deadline slack, or predicted output length.
- The speakers compare threshold, dual-gate, optimizer, and formal-bound approaches, noting that published policies do not cleanly fit coding-agent sessions.
Benchmarking and AdmitPerf38:27
- Measure baseline capacity without admission control, replay realistic agent traces, and test loads from half to twice that capacity.
- Score goodput as requests meeting their SLO divided by all arrivals, including rejected requests.
- Keep the deployment constant while comparing policies; require declared signals to be available, and inspect recorded run files.
- In the reported test, a queue policy admitted 63% of requests and lowered P95 time to first token from about 2.5 seconds to about 1 second at the same goodput. KV use peaked at only 0.5%, while the queue reached 24 requests, showing why KV pressure alone was a poor signal for that setup.
Step by step
- Measure the engine's capacity with admission control disabled.
- Select an admission unit, live signal, decision rule, and admit/defer/reject behavior that match the workload.
- At ingress, inspect request and engine state before forwarding requests to the engine.
- On defer, return a retry delay; have clients wait with jitter. On rejection, communicate the reason and enforce retry limits.
- Replay realistic agent traces at loads from 0.5× to 2× baseline capacity, comparing the same deployment with and without the policy.
- Evaluate goodput over all arrivals, latency/SLO results, repeated-run variation, and the disclosed benchmark configuration.
Tools & resources mentioned
- AdmitPerf — ingress admission library and benchmark harness; includes a fake engine for laptop testing.
- vLLM and SGLang — inference engines discussed as deployment targets and benchmark integrations.
- Qwen 2.5 — model used in the reported small-GPU benchmark.
Notes are AI-generated from the video's transcript and may miss nuance.