00 · llm benchmark
Frontier model benchmark
Head-to-head evaluations of 37 models — paid frontier models (Claude, GPT, Gemini, Kimi, Codex) plus open-weight models on the OpenRouter free tier (Nemotron, Gemma, GPT-OSS, North, Laguna, Ling) — across physics, game building, security, UI generation, mathematics, and electronics. Each task asks for a single self-contained artifact, run multiple times and scored for correctness, runtime, and cost.
01 · results
Model performance
37 of 37 models
| # | Model | ||||||
|---|---|---|---|---|---|---|---|
| 01 | Laguna XS 2.1 Poolside · Laguna 2.1 · 7 runs | 99.4 | 36.44s | $0 | 144.9k | 100% | |
| 02 | Ling 3.0 Tiny inclusionAI · Ling 3.0 · 7 runs | 99.4 | 42.40s | $0 | 383.2k | 0% | |
| 03 | Gemma 4 26B A4B Google · Gemma 4 · 7 runs | 98.5 | 2m 31s | $0 | 107.9k | 100% | |
| 04 | Nemotron 3 Nano 30B NVIDIA · Nemotron 3 · 7 runs | 98.0 | 1m 01s | $0 | 195.8k | 86% | |
| 05 | Codex GPT-5.6 Sol OpenAI · GPT-5.6 · 7 runs | 97.9 | 7m 43s | $0 | 2.87M | 71% | |
| 06 | Nemotron 3 Super NVIDIA · Nemotron 3 · 7 runs | 97.3 | 1m 44s | $0 | 137.6k | 86% | |
| 07 | Laguna S 2.1 Poolside · Laguna 2.1 · 7 runs | 96.9 | 1m 14s | $0 | 122.2k | 100% | |
| 08 | GPT-6 Astra (extra high) OpenAI · GPT-6 Astra · 11 runs | 95.5 | 10m 40s | $0 | 2.53M | 100% | |
| 09 | Qwen3.8 27B MLX (local) Alibaba · Qwen3.8 · 7 runs | 94.7 | 2m 24s | $0 | 35.3k | 100% | |
| 10 | Nemotron 3 Nano Omni NVIDIA · Nemotron 3 · 7 runs | 94.6 | 1m 01s | $0 | 229.2k | 100% | |
| 11 | GPT-OSS 20B OpenAI · gpt-oss · 7 runs | 94.4 | 2m 32s | $0 | 114.3k | 86% | |
| 12 | Devin SWE-2 (high) Cognition · SWE-2 · 11 runs | 94.3 | 6m 50s | $0 | 148.3k | 91% | |
| 13 | Nemotron 3 Ultra NVIDIA · Nemotron 3 · 7 runs | 93.8 | 3m 46s | $0 | 156.6k | 43% | |
| 14 | Claude 4 Anthropic · Claude 4 · 7 runs | 92.6 | 12.90s | $0.042 | 35.2k | 100% | |
| 15 | Gemini 3.6 Flash Google · Gemini 3.6 · 8 runs | 91.0 | 58.15s | $0.0004 | 411.3k | 100% | |
| 16 | Kimi K3 Moonshot AI · Kimi · 7 runs | 90.5 | 7m 26s | $1.014 | 482.8k | 71% | |
| 17 | Nemotron Nano 9B V2 NVIDIA · Nemotron Nano · 7 runs | 89.7 | 2m 14s | $0 | 151.1k | 86% | |
| 18 | Gemini 2.5 Pro Google · Gemini 2.5 · 7 runs | 89.1 | 9.40s | $0.025 | 35.6k | 100% | |
| 19 | GPT-5 OpenAI · GPT-5 · 7 runs | 85.6 | 6.57s | $0.026 | 32.4k | 100% | |
| 20 | Kimi K2.7 Moonshot AI · Kimi · 7 runs | 83.1 | 2m 37s | $0.173 | 614.1k | 71% | |
| 21 | North Mini Code Cohere · North · 7 runs | 82.7 | 4m 36s | $0 | 259.4k | 71% | |
| 22 | Gemini 3.5 Flash (agy) Google · Gemini 3.5 · 7 runs | 82.5 | 1m 23s | $0.025 | 167.6k | 86% | |
| 23 | Codex GPT-5.5 OpenAI · GPT-5.5 · 7 runs | 81.4 | 2m 32s | $0.194 | 542.3k | 86% | |
| 24 | Codex GPT-5.6 Terra OpenAI · GPT-5.6 · 7 runs | 81.2 | 3m 49s | $0 | 1.59M | 100% | |
| 25 | Ornith 1.5 35B (local) ChirpChirp · Ornith 1.5 · 7 runs | 70.3 | 42.49s | $0 | 36.3k | 100% | |
| 26 | DeepSeek V4 Flash Free DeepSeek · V4 · 7 runs | 69.5 | 4m 48s | $0 | 68.5k | 86% | |
| 27 | Codex GPT-5.6 Luna OpenAI · GPT-5.6 · 7 runs | 67.6 | 3m 42s | $0 | 1.76M | 100% | |
| 28 | Qwen3 8B (local) Alibaba · Qwen3 · 7 runs | 66.7 | 15.36s | $0 | 11.8k | 100% | |
| 29 | Gemma 3 12B (local) Google · Gemma 3 · 7 runs | 64.7 | 31.48s | $0 | 14.5k | 100% | |
| 30 | Nemotron Nano 12B VL NVIDIA · Nemotron Nano · 8 runs | 54.4 | 49.84s | $0 | 85.7k | 75% | |
| 31 | Gemma 3 27B (local) Google · Gemma 3 · 7 runs | 50.3 | 53.64s | $0 | 13.2k | 100% | |
| 32 | Qwen3 14B (local) Alibaba · Qwen3 · 7 runs | 45.9 | 1m 17s | $0 | 27.2k | 100% | |
| 33 | Claude Opus 4.6 Thinking (agy) Anthropic · Claude Opus 4.6 · 7 runs | 42.9 | 1m 12s | $0.194 | 18.9k | 0% | |
| 34 | Gemini 3.6 Flash (agy) Google · Gemini 3.6 · 7 runs | 42.7 | 36.41s | $0.032 | 215.1k | 43% | |
| 35 | Gemma 4 31B Google · Gemma 4 · 7 runs | 41.1 | 16.49s | $0 | 13.3k | 0% | |
| 36 | Gemma 3 4B (local) Google · Gemma 3 · 7 runs | 30.3 | 9.43s | $0 | 11.0k | 100% | |
| 37 | GPT-OSS 120B (agy) OpenAI · GPT-OSS · 7 runs | 20.4 | 10.92s | $0.0076 | 20.3k | 14% |
