What happened

The task is not a chat answer — agents must drive a fixed nanoGPT optimization loop with tool calls, subagents, and scratchpads under one scoring protocol. That makes the table closer to a production agent stress test than to a one-shot exam question.

Per the published summary, Kimi K3kimi-code holds the best validated mark. Opus 4.8claude-code scored 3,018 and GPT-5.6 Solcodex 3,042 — close numbers, with validation still calling out Kimi as the top verified run. Spread across the other models is large; that variance is the point of the harness.

The public bonus is 41 full trajectories: tool sequences, subagent branches, and scratchpad text, not only a final score.

Why it matters

Classic leaderboards reward short answers. Agent coding and tuning fail on planning, tool loops, and recovery. A shared optimizer speedrun surfaces those gaps without a slide deck that says “almost as good as the top on MMLU.”

For teams picking a model for IDE or CI agents, this kind of run is more informative than raw price per million tokens: you see who finishes a long loop with a valid score.

In practice

1. Read the spread across 18 models, not only the top row — a thin board hides instability.
2. Open at least one of the 41 trajectories and compare it to how your agent walks tools.
3. When selecting a coding agent model, add a long-context edit loop to your bake-off, not only HumanEval.
4. Freeze validation the way the speedrun does — otherwise “on our bench” is not reproducible.
5. Keep raw scores and “best validated” labels separate; the write-up treats them differently on purpose.

Takeaway

NanoGPT Speedrun Frontier is a rare public agent × frontier model cut on one task with trajectories. Use it as a compass for long loops — not as a substitute for your own scenarios.