Generic serving
Qwen3.6-27B
1× H200
Multi-tenant API serving with traffic patterns from the Alibaba Bailian trace.
Gated DeltaNet in Qwen3.6
Humanity's Last System
Steven Gao1, Rulin Shao1, Keisuke Kamahori1, Yuetai Li1, Yufei Gao1, Atindra Jha2, Junjie Oscar Yin1, Shihang Li1, Radha Poovendran1, Jure Leskovec2, Hannaneh Hajishirzi1, Pang Wei Koh1, Baris Kasikci1, Stephanie Wang1, Luke Zettlemoyer1
1University of Washington2Stanford University
Can coding agents build inference systems that are hyperspecialized to the model architecture, hardware, and workload? We gave frontier agents live GPU sandboxes of up to 8 H200s and up to 36 hours to build an inference stack from scratch.
Leaderboard
| 1 | Opus 5max | Claude Code | 71.4% | 81.0% | 1.91 | 0.99 | 1.83 | 1.89 | 25.9 | 2.40 | 4.92 |
| 2 | Kimi K3max | Claude Code | 23.8% | 33.3% | 0 | 0 | 0 | 1.01 | 12.4 | 1.24 | 1.04 |
| 3 | Opus 4.8max | Claude Code | 19.0% | 71.4% | 0.45 | 0.029 | 0 | 0.43 | 7.10 | 1.08 | 0.76 |
| 4 | GLM 5.3max | Claude Code | 14.3% | 28.6% | 0 | 0 | 0 | 0.80 | 13.2 | 0 | 1.20 |
| 5 | GPT 5.6 Solmax | Codex | 9.5% | 42.9% | 0 | 0.013 | 0 | 0.54 | 0 | 1.66 | 0.54 |
| 6 | GPT 5.5xhigh | Codex | 0.0% | 28.6% | 0 | 0 | 0 | 0 | 0.25 | 0 | 0.038 |
Task columns show speedup over the baseline: 1 matches it, 0 is a run that failed. Correct and Faster than baseline are shares of each model's 21 runs.