V4 Flash on ARC-AGI: 89% for $0.02 a task

In our ChinaAIbench evaluation run, we gave V4 Flash eleven tasks on August 3 and it failed exactly one: "consecutive integers summing to 575." It answered 341. Twice. Then we flipped thinking on and it got 575 three times in a row — same prompt, one flag. That quirk is the right lens for the ARC Prize result published this week, because the verified table shows the same non-monotonicity at scale: more reasoning effort does not always mean more correct.

The short version

ARC Prize verified DeepSeek V4 Flash-0731 at 89.0% on ARC-AGI-1 Semi-Private with max effort, at $0.02 per task, and 61.4% on ARC-AGI-2 at $0.04 per task. The ARC-AGI-2 number is the one to stare at. ARC-AGI-1 is nearly saturated for frontier models, but ARC-AGI-2 is a held-out reasoning benchmark the model couldn't have trained on. A reasoning model crossing 60% on ARC-AGI-2 is interesting. Doing it at four cents per task changes who can afford to experiment.

The verified numbers

ARC Prize runs its own evaluation on a semi-private set and publishes per-task pass/fail tables, which is why we trust this more than a vendor table. The row that matters is not Max. It is Low.

The public eval breakdown tells the same story with harder edges. On ARC-AGI-1's 400 public tasks, Max passed 379 (94.75%), High 358 (89.5%), Low 373 (93.25%) — Low beat High. On ARC-AGI-2's 120 tasks: Max 72 (60.0%), High 65 (54.2%), Low 51 (42.5%). In 9 tasks Max failed while High passed, and in 5 tasks Low passed where Max failed. ARC-AGI-3 was not evaluated.

Three things stand out.

Inference scaling pays off exactly where reasoning is hard. Moving Low to Max gains +14.9 points on ARC-AGI-2 but only +5.0 on ARC-AGI-1. On a benchmark where the model already knows most answers, more thinking buys little. On a genuinely held-out task set, it buys a lot. That matches what we saw in our own 11-task run: thinking off cost us the 575 problem, thinking on fixed it.

The non-monotonicity is real and it has a cost. Our 341 result looked like a fluke until we saw ARC's table: 9 tasks where max effort failed and high effort passed, plus the Low-beats-High anomaly on the public ARC-AGI-1 set. Reasoning effort is not a monotonic dial. For agent builders that means thinking budgets are a search problem, not a slider you max out. You pay for the highest tier and occasionally get worse answers for it.

$0.02 per task changes the economics of evals. We spent $0.0013 on 22 API calls for our whole V4 battery. ARC Prize's per-task cost of $0.02 makes large-scale reasoning evaluation affordable enough to run continuously.

61.4% on ARC-AGI-2 for a Flash-tier model pulls the "reasoning is expensive" argument out from under the closed labs.

For scale, ARC Prize's own results page lists GPT-5.6 Sol at 92.5% and Claude Opus 5 at 90.4% on ARC-AGI-2 at max reasoning effort, roughly thirty points above V4 Flash's 61.4%. The score gap is still wide; the cost gap is now the interesting one.

Where this fits our own testing

Our own logs put a price tag on that gap. The cost gap is now the interesting one, and our numbers make it concrete. We ran V4 Flash-0731 on August 3, three days after release: 10/11 tasks passed, the whole 22-call battery cost $0.0013, average 1.1s per response. The one failure was the 575 math problem with thinking off — the same model that just scored 89% on ARC-AGI-1 verified. Both things are true: this is among the strongest value propositions we've seen for low-cost reasoning workloads, and it will still hand you 341 when you ask the wrong way. ARC Prize's independent verification is the closest thing to a third-party answer on where the cheap tier actually lands.

Test notes & sources

• ARC Prize verified results, published 2026-07-31: arcprize.org/results/deepseek-v4-flash-0731 (semi-private scores, public 400/120-task pass/fail tables, cost figures)
• ARC Prize results index for frontier reference: arcprize.org/results (GPT-5.6 entry 2026-07-09, Claude Opus 5 entry 2026-07-24; 92.5% / 90.4% on ARC-AGI-2 at max reasoning effort)
• Our hands-on run: DeepSeek V4 review, August 3 2026, 11-task battery, both models, raw logs published (deepseek-v4-review-2026)
• What we didn't verify: ARC-AGI-3 (not evaluated by ARC Prize for this model); we did not re-run ARC's semi-private set ourselves; ARC Prize does not publish per-task cost for GPT-5.6 or Claude Opus 5, so no frontier cost comparison is made
• Pricing context: V4 Flash $0.14/$0.28 per million tokens (vendor docs, as of July 31, 2026)