NIST CAISI vs DeepSeek V4: the 8-month gap

Vendor benchmark tables are useful if you read them as advertising. DeepSeek's own chart makes V4 Pro look like a peer of GPT-5.4 and Claude Opus 4.6 — released roughly two months earlier, which is a normal product cycle gap. Then the US government's evaluation shop ran the same model on benchmarks it had never seen, and the gap turned out to be about eight months.

This is the story of that discrepancy, and why I think it's the most important number in the V4 release.

What CAISI actually is

NIST's Center for AI Standards and Innovation is the US federal evaluation shop for commercial AI. It runs unclassified evals across five domains: cyber, software engineering, natural sciences, abstract reasoning, and math. Most of its benchmarks are closed, deliberately — the model can't have trained on them. That's the entire point of the exercise, and it's the thing no vendor leaderboard can offer you.

CAISI evaluated DeepSeek V4 Pro in April 2026 and published on May 1. It also reproduced DeepSeek's self-reported GPQA-Diamond results before running its own suite — a sanity check that rules out serving or configuration errors. I wish every benchmark article did that, including mine.

The three findings

First, V4 lags the US frontier by roughly eight months. That's an IRT-style aggregate over 16 benchmarks and 35 models. DeepSeek's table makes V4 look like a peer of models released two months earlier; CAISI's closed suite puts V4 at roughly GPT-5's level, released eight months earlier. That gap between the two pictures is the contamination gap, quantified.

Second, V4 is still the most capable Chinese model CAISI has ever evaluated. V3 trailed the frontier by 12-14 months. V4 trails by 8. The trend is closing, and math, software engineering, and natural sciences are its strongest domains.

Third, it wins on capability per dollar. On 5 of 7 benchmark tasks, V4 was cheaper than GPT-5.4 mini — from 53% less expensive to 41% more, depending on the task. Cheap-and-close beats expensive-and-frontier when the frontier part isn't what you need.

The weaknesses are the interesting part

The failures CAISI found are exactly the ones DeepSeek's report doesn't feature: ARC-AGI-2 semi-private, PortBench (a held-out software engineering eval CAISI built itself), CTF-Archive-Diamond (cyber). These are reasoning-under-novel-conditions tasks, not canned coding.

My own smoke test on August 3 agreed with the shape of this: V4 Pro passed 11/11 practical tasks, but the knowledge and novel-reasoning dimensions are where the vendor table and reality diverge. SimpleQA tells the same story — 57.9 for V4 vs 75.6 for Gemini 3.1 Pro. Ask V4 about facts and it drifts.

What the 8-month gap buys you

If you're building on a Chinese model today, the gap is the price of admission: you get frontier-adjacent coding at 1/54th of the output price, and you give up frontier knowledge and novel reasoning. That's a trade a lot of teams should take — my whole 22-call battery cost $0.0013, which is itself the argument.

What you shouldn't do is quote the vendor table in a procurement meeting. Quote CAISI. The government eval is the number that survives a second opinion.

One thing I want to see next: whether the trend keeps closing. V3 trailed by 12-14 months, V4 by 8. If the next release cuts that to 4-5, the "frontier gap" stops being a technical argument and becomes a pricing one. That's when the interesting procurement conversations start.

This is a field note, not a full review — the numbers here are cross-checked in my hands-on review: DeepSeek V4 review 2026: benchmarks, pricing, my test run. Sources: NIST CAISI "Evaluation of DeepSeek V4 Pro" (2026-05-01), DeepSeek technical report arXiv:2606.19348, and my API test logs from 2026-08-03.