HLS

Traces

153 runs
Generic servingOpus 5maxClaude Code11.12965$197.17
Generic servingOpus 5maxClaude Code21.91634$222.81
Generic servingOpus 5maxClaude Code30correctness: output distribution888$185.03
Generic servingOpus 4.8maxClaude Code10correctness: output distribution1,622$233.81
Generic servingOpus 4.8maxClaude Code20correctness: output distribution1,252$216.39
Generic servingOpus 4.8maxClaude Code30.451,218$198.67
Generic servingGPT 5.6 SolmaxCodex10correctness: systems1,550$108.01
Generic servingGPT 5.6 SolmaxCodex20correctness: systems3,404$225.69
Generic servingGPT 5.6 SolmaxCodex30correctness: systems2,572$166.59
Generic servingGPT 5.5xhighCodex10correctness: systems4,274$191.34
Generic servingGPT 5.5xhighCodex20no valid submission3,588$149.27
Generic servingGPT 5.5xhighCodex30no valid submission3,188$128.85
Generic servingKimi K3maxClaude Code10correctness: output distribution1,567$123.15est.
Generic servingKimi K3maxClaude Code20correctness: systems1,237$86.19est.
Generic servingKimi K3maxClaude Code30correctness: systems1,213$93.48
Generic servingGLM 5.3maxClaude Code10correctness: systems1,044$71.52est.
Generic servingGLM 5.3maxClaude Code20correctness: systems838$59.93est.
Generic servingGLM 5.3maxClaude Code30correctness: systems953$71.33est.
Generic servingOpus 5maxClaude Code13h horizon0.028329$37.91
Generic servingOpus 5maxClaude Code23h horizon0.79400$42.54
Generic servingOpus 5maxClaude Code33h horizon0.63342$34.32
Generic servingOpus 5maxClaude Code16h horizon0correctness: systems530$115.85
Generic servingOpus 5maxClaude Code26h horizon0.66491$86.73
Generic servingOpus 5maxClaude Code36h horizon0correctness: output distribution575$88.63
Generic servingOpus 5maxClaude Code124h horizon2.411,087$587.09
Generic servingOpus 5maxClaude Code224h horizon2.27907$659.82
Generic servingOpus 5maxClaude Code324h horizon2.041,246$323.31
Generic servingOpus 5maxClaude Code136h horizon2.261,011$932.34
Generic servingOpus 5maxClaude Code236h horizon0correctness: output distribution1,629$406.77
Generic servingOpus 5maxClaude Code336h horizon2.491,332$706.48
Generic servingOpus 5lowClaude Code1low effort0correctness: output distribution834$155.98
Generic servingOpus 5lowClaude Code2low effort1.34947$163.10
Generic servingOpus 5lowClaude Code3low effort1.03883$156.07
Generic servingOpus 5mediumClaude Code1medium effort1.33801$125.07
Generic servingOpus 5mediumClaude Code2medium effort0.42791$127.78
Generic servingOpus 5mediumClaude Code3medium effort1.45989$187.94
Generic servingOpus 5highClaude Code1high effort1.73526$221.91
Generic servingOpus 5highClaude Code2high effort1.50598$241.76
Generic servingOpus 5highClaude Code3high effort1.30766$156.46
Generic servingOpus 5xhighClaude Code1xhigh effort1.56870$162.09
Generic servingOpus 5xhighClaude Code2xhigh effort2.221,044$255.67
Generic servingOpus 5xhighClaude Code3xhigh effort1.69910$193.08
Generic servingOpus 5maxClaude Code1vLLM source1.22604$193.20
Generic servingOpus 5maxClaude Code2vLLM source0correctness: output distribution637$206.56
Generic servingOpus 5maxClaude Code3vLLM source0correctness: output distribution961$150.99
Agentic codingOpus 5maxClaude Code10correctness: systems713$309.00
Agentic codingOpus 5maxClaude Code20.54992$274.03
Agentic codingOpus 5maxClaude Code30.99866$212.31
Agentic codingOpus 4.8maxClaude Code10.0291,550$254.17
Agentic codingOpus 4.8maxClaude Code20.00861,447$250.96
Agentic codingOpus 4.8maxClaude Code30.0121,391$195.82
Agentic codingGPT 5.6 SolmaxCodex10correctness: systems3,085$215.51
Agentic codingGPT 5.6 SolmaxCodex20correctness: systems3,373$243.61
Agentic codingGPT 5.6 SolmaxCodex30.0133,087$231.12
Agentic codingGPT 5.5xhighCodex10correctness: systems3,921$160.64
Agentic codingGPT 5.5xhighCodex20correctness: output distribution3,987$166.12
Agentic codingGPT 5.5xhighCodex30correctness: output distribution4,558$197.12
Agentic codingKimi K3maxClaude Code10correctness: systems775$49.26
Agentic codingKimi K3maxClaude Code20correctness: systems1,484$137.84
Agentic codingKimi K3maxClaude Code30correctness: output distribution1,340$78.47
Agentic codingGLM 5.3maxClaude Code10correctness: systems832$55.90est.
Agentic codingGLM 5.3maxClaude Code20no valid submission3,684$287.99est.
Agentic codingGLM 5.3maxClaude Code30no valid submission1,268$152.92est.
Deep researchOpus 5maxClaude Code11.83778$198.86
Deep researchOpus 5maxClaude Code21.53853$165.45
Deep researchOpus 5maxClaude Code31.57682$222.95
Deep researchOpus 4.8maxClaude Code10correctness: systems1,499$255.85
Deep researchOpus 4.8maxClaude Code20correctness: output distribution1,016$333.35
Deep researchOpus 4.8maxClaude Code30correctness: systems283$21.66
Deep researchGPT 5.6 SolmaxCodex10correctness: systems2,747$181.95
Deep researchGPT 5.6 SolmaxCodex20correctness: systems3,123$209.11
Deep researchGPT 5.6 SolmaxCodex30correctness: systems2,949$202.46
Deep researchGPT 5.5xhighCodex10correctness: systems3,056$119.73
Deep researchGPT 5.5xhighCodex20correctness: systems1,455$67.54
Deep researchGPT 5.5xhighCodex30correctness: systems2,047$76.29
Deep researchKimi K3maxClaude Code10correctness: systems1,420$103.95
Deep researchKimi K3maxClaude Code20correctness: output distribution1,261$93.68
Deep researchKimi K3maxClaude Code30correctness: systems1,586$121.76
Deep researchGLM 5.3maxClaude Code10correctness: systems815$53.78est.
Deep researchGLM 5.3maxClaude Code20correctness: systems570$68.54est.
Deep researchGLM 5.3maxClaude Code30correctness: systems678$56.50est.
Code completionOpus 5maxClaude Code10correctness: systems473$221.18
Code completionOpus 5maxClaude Code21.69709$296.26
Code completionOpus 5maxClaude Code31.89960$210.90
Code completionOpus 4.8maxClaude Code10.431,489$254.63
Code completionOpus 4.8maxClaude Code20.141,464$222.68
Code completionOpus 4.8maxClaude Code30.341,884$246.41
Code completionGPT 5.6 SolmaxCodex10.541,804$113.72
Code completionGPT 5.6 SolmaxCodex20.00432,978$213.81
Code completionGPT 5.6 SolmaxCodex30.0142,749$181.17
Code completionGPT 5.5xhighCodex10correctness: output distribution2,490$107.74
Code completionGPT 5.5xhighCodex20correctness: systems2,564$119.88
Code completionGPT 5.5xhighCodex30correctness: systems3,984$161.84
Code completionKimi K3maxClaude Code11.011,320$101.50
Code completionKimi K3maxClaude Code20correctness: systems2,227$218.45est.
Code completionKimi K3maxClaude Code30.851,301$89.79
Code completionGLM 5.3maxClaude Code10correctness: systems1,808$309.30est.
Code completionGLM 5.3maxClaude Code20.801,294$111.90
Code completionGLM 5.3maxClaude Code30correctness: systems916$121.85est.
Speech recognitionOpus 5maxClaude Code10correctness: output distribution828$348.48
Speech recognitionOpus 5maxClaude Code218.21,039$209.63
Speech recognitionOpus 5maxClaude Code325.9997$215.53
Speech recognitionOpus 4.8maxClaude Code13.25756$261.64
Speech recognitionOpus 4.8maxClaude Code27.10779$247.84
Speech recognitionOpus 4.8maxClaude Code34.74975$303.74
Speech recognitionGPT 5.6 SolmaxCodex10correctness: output distribution3,775$226.03
Speech recognitionGPT 5.6 SolmaxCodex20correctness: output distribution3,232$195.79
Speech recognitionGPT 5.6 SolmaxCodex30correctness: output distribution3,246$204.61
Speech recognitionGPT 5.5xhighCodex10.162,815$110.09
Speech recognitionGPT 5.5xhighCodex20.194,213$165.92
Speech recognitionGPT 5.5xhighCodex30.253,893$167.84
Speech recognitionKimi K3maxClaude Code112.41,471$111.20
Speech recognitionKimi K3maxClaude Code210.91,506$102.53
Speech recognitionKimi K3maxClaude Code30correctness: output distribution1,536$99.13
Speech recognitionGLM 5.3maxClaude Code12.25911$83.99est.
Speech recognitionGLM 5.3maxClaude Code20correctness: systems940$102.79est.
Speech recognitionGLM 5.3maxClaude Code313.2914$53.89
Speech synthesisOpus 5maxClaude Code12.40992$241.64
Speech synthesisOpus 5maxClaude Code22.11912$234.13
Speech synthesisOpus 5maxClaude Code32.231,054$282.61
Speech synthesisOpus 4.8maxClaude Code10.92853$331.47
Speech synthesisOpus 4.8maxClaude Code20.86909$371.44
Speech synthesisOpus 4.8maxClaude Code31.081,006$373.21
Speech synthesisGPT 5.6 SolmaxCodex11.643,013$190.12
Speech synthesisGPT 5.6 SolmaxCodex20.862,438$163.82
Speech synthesisGPT 5.6 SolmaxCodex31.662,735$178.07
Speech synthesisGPT 5.5xhighCodex10correctness: output distribution2,556$99.49
Speech synthesisGPT 5.5xhighCodex20correctness: systems1,993$64.38
Speech synthesisGPT 5.5xhighCodex30no valid submission1,293$55.56
Speech synthesisKimi K3maxClaude Code11.241,306$62.30
Speech synthesisKimi K3maxClaude Code20correctness: output distribution1,128$54.84
Speech synthesisKimi K3maxClaude Code30correctness: systems1,377$116.08
Speech synthesisGLM 5.3maxClaude Code10correctness: output distribution806$86.28est.
Speech synthesisGLM 5.3maxClaude Code20correctness: systems606$66.04est.
Speech synthesisGLM 5.3maxClaude Code30correctness: systems820$73.03est.
Omni chatOpus 5maxClaude Code12.26803$347.39
Omni chatOpus 5maxClaude Code24.08763$284.89
Omni chatOpus 5maxClaude Code34.92895$347.32
Omni chatOpus 4.8maxClaude Code10correctness: systems1,024$344.42
Omni chatOpus 4.8maxClaude Code20.761,098$351.84
Omni chatOpus 4.8maxClaude Code30.75817$293.11
Omni chatGPT 5.6 SolmaxCodex10correctness: output distribution3,410$227.88
Omni chatGPT 5.6 SolmaxCodex20.173,302$209.72
Omni chatGPT 5.6 SolmaxCodex30.542,877$190.90
Omni chatGPT 5.5xhighCodex10.0311,282$57.57
Omni chatGPT 5.5xhighCodex20.038550$24.05
Omni chatGPT 5.5xhighCodex30.00503,220$118.06
Omni chatKimi K3maxClaude Code11.041,555$121.21
Omni chatKimi K3maxClaude Code20.351,222$82.92
Omni chatKimi K3maxClaude Code30correctness: output distribution1,345$72.84
Omni chatGLM 5.3maxClaude Code11.201,080$109.77est.
Omni chatGLM 5.3maxClaude Code20.501,010$56.50
Omni chatGLM 5.3maxClaude Code30.0241,190$162.67est.