Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean
Use Optuna with Bayesian optimization to tune VLM serving recipes against token throughput, then deploy the best-performing YAML config.
- Start with the model’s serving YAML and define the study, hardware, and trial count for tuner.py.
- The agent scores each recipe over repeated trials and outputs an HTML report plus an optimized YAML file.
- The showcased runs reported 17.5% more tokens for GPT-OSS and doubled throughput with nearly halved ITL for DeepSeek R1 Distill.
The demo shows the tuning inputs and generated reports, though runs take hours. · Best for: Engineers tuning VLM serving configurations for throughput.
The demo presents an agent that automates the search for efficient VLM serving configurations using Bayesian optimization via Optuna. It iteratively tests recipes and reports performance results.
Goal and optimization approach0:01
- The agent aims to reduce manual effort in finding a good VLM serving recipe.
- Bayesian optimization narrows the search; Optuna manages trials, scores recipes, and updates settings.
Starting recipe and run inputs2:24
- The example tests chat short, chat long, RAG, and batch classify regimes, with configurable input tokens.
- Run tuner.py with a study name, starting YAML, hardware, and trial count; one example uses 24 trials.
Run duration and outputs3:25
- Runs can take hours; the GPT-OSS example took about four to five hours.
- The run produces an HTML report and an optimized YAML recipe.
Reported results4:31
- A 50-trial GPT-OSS run reported 17.5% improvement in generated tokens.
- A nine-trial DeepSeek R1 Distill run reported doubled token throughput and ITL reduced by almost half.
Step by step
- Download the model weights and inspect its starting serving YAML recipe.
- Run tuner.py, providing a study name, starting YAML file, hardware, and number of trials.
- Review the generated HTML report and use the optimized YAML recipe from the run.
Tools & resources mentioned
- Optuna — manages the optimization search loop and trials.
- tuner.py — script used to configure and run tuning.
- Starting YAML — baseline serving recipe provided with the model weights.
- HTML report — summarizes trial results and identifies the best trial.
- Optimized YAML — contains the selected serving recipe settings.
Notes are AI-generated from the video's transcript and may miss nuance.