cost & model
Where tokens and dollars go across every model and suite in the harness. Cost is the primary efficiency signal; tokens are the budget. Budget alerts flag any model running over the per-task or per-run thresholds.
$5.2
(2 models have no cost data)
15.82M
146
5
$0.07
| Model | Tasks | Pass rate | Total cost | Total tokens | Cost / solved | Avg time | Top tools |
|---|---|---|---|---|---|---|---|
| agy+MiniMax-M3+deepseek-v4-proconsensus | 36 | 100% | $2.6 | 7.88M | $0.07 | 3m 4s | n/a |
| MiniMax-M3minimax | 37 | 11% | $1.4 | 4.09M | $0.35 | 2m 60s | n/a |
| deepseek-v4-prodeepseek | 36 | 11% | $1.2 | 3.85M | $0.30 | 3m 3s | n/a |
| moonshot-v1-128kkimi | 1 | 100% | n/a | n/a | n/a | 56.0s | n/a |
| agyexternal | 36 | 81% | n/a | n/a | n/a | 3m 3s | n/a |
| Suite | Models | Runs | Passed | Total cost | Total tokens | Total time |
|---|---|---|---|---|---|---|
| harder | 4 | 145 | 73/145 | $5.2 | 15.82M | 7h 21m |
| deep-swe | 1 | 1 | 1/1 | n/a | n/a | 56.0s |
| Complexity | Runs | Passed | Total cost | Total tokens | Avg time |
|---|---|---|---|---|---|
| medium | 86 | 44/86 | $3.1 | 9.52M | 2m 38s |
| complex | 48 | 24/48 | $1.5 | 4.51M | 4m 1s |
| simple | 12 | 6/12 | $0.59 | 1.79M | 1m 55s |
| Task | Model | Cost | Tokens | Duration | Status |
|---|---|---|---|---|---|
| Test the debounce implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.24 | 723.4k | 4m 46s | pass |
| Build a token-bucket limiter with exact semanticsharder | agy+MiniMax-M3+deepseek-v4-pro | $0.21 | 632.2k | 8m 50s | pass |
| Test the debounce implementationharder | MiniMax-M3 | $0.20 | 592.3k | 4m 46s | fail |
| Build a token-bucket limiter with exact semanticsharder | MiniMax-M3 | $0.17 | 486.5k | 8m 49s | fail |
| Test the groupBy implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.14 | 431.9k | 3m 45s | pass |
| Test the debounce implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.13 | 396.5k | 3m 6s | pass |
| Test the flatten implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.12 | 377.7k | 1m 57s | pass |
| Test the groupBy implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.11 | 349.1k | 2m 41s | pass |
| Test the groupBy implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.11 | 326.8k | 2m 55s | pass |
| Test the flatten implementationharder | agy+MiniMax-M3+deepseek-v4-pro | $0.10 | 293.3k | 1m 49s | pass |
Methodology
Cost and token data comes from the model provider responses (where the API exposes usage). External CLIs like agy (PTY-based, no metrics parser) and providers that don't return cost in the chat response (e.g. z.ai GLM) show as n/a. Pass-rate is computed from the harness server's own evaluation, not the model self-report.
Budget thresholds: per-task cost > $0.50 triggers a warn, > $2.00 an alert. Total run cost > $5.00 also flags as alert.