China vs US frontier: 9 models, 11 tasks, same battery

Editor's note: model names and dates reflect information available as of 2026-08-07/08; verify with official sources.

Every "China vs US frontier" piece I read this year was someone else's benchmark table, re-colored and republished. So we built the comparison nobody else had: nine models, five Chinese and four American, the exact same eleven tasks, the same judge, one run per model, and every raw log published.

The headline result isn't a score. It's the failure pattern. Six of nine models passed all eleven tasks, and nobody dropped below ten. The only miss on the American side was Claude Opus 5, and it wasn't a capability miss. It ran out of output budget before writing anything. The two Chinese misses were different in kind: thinking that never converges even with a bigger budget (a reasoning behavior issue), and a sign error in high-school math (a plain capability error).

The compiled run

Same prompts, same grading module. The five Chinese models ran on direct vendor APIs: Zhipu, Volcano Engine, SiliconFlow. The four US models ran through OpenRouter (non-affiliate) on the same battery, with byte-identical prompts pre-registered in prompts-2026-08-07.md before any call went out. The judge is a deterministic grading module previously used in DeepSeek V4 evaluations [7].

The battery: an IPv4 validator, a bug-fix task for finding the second-largest element, two math problems (quadratic roots; consecutive integers summing to 575), a syllogism, two knowledge questions (36th US president, Burkina Faso's capital), a three-sentence product copy, JSON extraction, JSON-array summary, and a 426-token needle test. Code, math, logic, knowledge, writing, structured output, long context. Eleven tasks, six dimensions [1].

One caveat up front: this is an N=1 smoke-test battery, not a statistical evaluation. One run per model per task, on August 7, 2026, with the settings below. It shows what these models did that day. It doesn't tell you which model is better, and nobody should rank anything off eleven samples.

The 9×11 table

• [a] Step 3.5 Flash: forces reasoning on SiliconFlow and ignores the thinking parameter. First run burned 1024 max_tokens with empty content; a 4096 retry spent ~15.5K reasoning chars and still didn't converge. Battery average: 11.96s latency, 10,612 tokens.
• [b] Hunyuan A13B: answered "3, -4" on the quadratic x² − x − 12 = 0. The roots are 4 and -3 (3²-3-12=-6≠0). Sign check skipped. A capability error, reported as-is.
• [c] Claude Opus 5: with OpenRouter's default reasoning on, spent its 256 max_tokens on the reasoning trace and returned empty content. Our retry with thinking disabled was blocked by the free-tier balance (HTTP 402). Output budget exhaustion, not a capability miss.
• [d] Gemini 3.6 Flash: first run truncated at 1024 max_tokens (reasoning ate 993 chars, content cut to two sentences). Rerun at 2048 passed; the first-run record is kept in first_run_truncated.

Chinese rows ran on direct vendor APIs, US rows via OpenRouter. Scores are counts of passed tasks, not percentages.

What the scores say

Six of nine cleared the full battery: GLM-5.2, Doubao Seed 2.1, Ling 2.0, GPT-5.6 Luna Pro, Gemini 3.6 Flash, Grok 4.5. Three landed at 10/11: Step 3.5 Flash, Hunyuan A13B, Claude Opus 5.

Read the failure column, not the scores. The two Chinese misses were different in kind. Step 3.5 Flash's ipv4 task is the "simple task, thinking doesn't converge" case — a reasoning behavior issue, not a capability miss: more output budget didn't help. It's the same family as the overthinking complaints people post about reasoning models, except here it's reproducible with raw logs. Hunyuan A13B's quadratic answer is a plain capability error, the kind a code reviewer would bounce back in thirty seconds.

The US side's only miss is the mirror image: Claude Opus 5 failed a task it can obviously do, because the default configuration let the reasoning trace eat the entire output budget. Claude Opus 5's budget exhaustion mirrors Step 3.5 Flash's forced reasoning behavior, from the other side of the ocean. We report the 10/11 as it ran, not as we'd like it to look. The retry (thinking disabled, 512 tokens) would have cost about $0.013, and OpenRouter's free-tier balance rejected it with HTTP 402. The $0.15 grant was already spent.

Two models got a config fix under our truncation policy: Gemini's first run was truncated and the 2048 rerun passed. Step 3.5 Flash's 4096 retry also followed policy. It just never converged. One policy note, because it changes how you read the table: Gemini's reasoning trace isn't scored (same policy as Step 3.5 Flash), and Claude is scored on default behavior, because that's what you get out of the box.

The cost story

Battery cost is what we actually spent on the eleven calls, read from the cost field in the raw logs. Listed price is the per-1M-token snapshot from August 7, 2026: US rows from OpenRouter [4], Chinese rows from vendor pricing pages [5]. Two different things, not to be mixed.

The spread is roughly a hundredfold from Ling to Gemini. Three things stand out:

• GPT-5.6 Luna Pro is the price killer. It passed everything for $0.004, about 8x less than Claude Opus 5 cost for one fewer pass, and its listed input price ($0.1) sits 50x below Opus's ($5).
• Ling 2.0, a Chinese flash model most Western readers have never heard of, ran the whole battery for less than half a cent. 01.AI's model line (the company behind the Yi series) doesn't appear in most "China AI" coverage, and it matched the US flagships at roughly 1/90th of Gemini's spend.
• Doubao Seed 2.1 going 11/11 matters because ByteDance's flagship is rarely independently tested in English. At ≈$0.85/$4.23 listed, it cleared the full battery where Claude Opus 5, at $5/$25, dropped one task.

Where the Chinese flagships actually lag

First, what's not on it: the Chinese flagships didn't trail GPT-5.6 Luna Pro on this battery. GLM-5.2, Seed 2.1 and Ling 2.0 all matched its 11/11, and each cost less to run than Luna's $0.004. The lag this batch showed is narrower and more specific.

Step 3.5 Flash is cheap and slow. 11.96s average latency and 10,612 tokens for eleven small tasks, 8 to 17 times the token burn of the other eight models. Forced reasoning is the vendor's config on SiliconFlow, not the test harness, and it's a real operational trait: batch small tasks through it and you pay in wall-clock time.

Hunyuan A13B's math isn't trustworthy yet. 0.72s and $0.0005 for the whole battery is remarkable, but a sign error on a quadratic is still a sign error on a quadratic.

Claude Opus 5 on OpenRouter ships a configuration trap: reasoning on by default, and with a small max_tokens budget you get empty content. It's on the retest list with thinking disabled.

To be clear about what this run does and doesn't say: it is not "China beats America." Both sides clear the bar on this battery, and the differences that showed up are mostly configuration and behavior, not intelligence. The one number that looks genuinely bad is Step 3.5 Flash's 10,612 tokens: token efficiency on trivial tasks is where this batch of Chinese flagships trails, and it's specific, reproducible, and logged.

Test notes & sources

Reproduce it yourself. Every call is in the raw logs with prompt, output, verdict, latency, tokens, and cost: per-model JSON files, mirrored to the HF dataset [2]. Task definitions and runner scripts live in the evaluation toolkit on GitHub [3], task battery locked at commit 4d467ac, prompts pre-registered before any API call.

Config, because conclusions without config are noise:

• Five Chinese models on direct vendor APIs; the four US models via OpenRouter free tier (grant $0.15 [6], fully spent across this run).
• Thinking explicitly disabled for GLM-5.2 and Seed 2.1 (same as the DeepSeek V4 runs); vendor default for the SiliconFlow trio; default reasoning for Claude and Gemini.
• max_tokens 64–512 per task on the Chinese vendor batch, 1024–2048 where reasoning models needed headroom; only Gemini got the relaxed budget, disclosed per task in the logs.
• Latency figures are channel-scoped: direct-API rows and OpenRouter rows are not comparable (gateway overhead), so we make no cross-channel latency claims here. Within the US zone, calls ran 0.7–7s through the gateway, with Gemini the steadiest at ~1.3s.

On the retest list: Claude's writing task with thinking disabled (~$0.013, expected pass), quarterly battery re-runs, and re-runs after major releases. When Claude passes, this table gets a twelfth green cell.

Same battery on the ChinaAI Bench leaderboard → International reference zone

Sources

1. Raw logs, 2026-08-07 — per-model calls-2026-08-07.json under raw-logs/ on the HF dataset (the compiled run, N=1; prompt, output, verdict, latency, tokens, cost per call): https://huggingface.co/datasets/EliChen-ai/china-ai-bench-benchmarks/tree/main/raw-logs
2. HF dataset mirror, huggingface.co/datasets/EliChen-ai/china-ai-bench-benchmarks (raw logs full data, 2026-08-07, the compiled run)
3. Evaluation toolkit, github.com/EliChen-ai/china-ai-bench (task-battery.json @ 4d467ac, runner scripts, the compiled run)
4. OpenRouter listed prices (non-affiliate), snapshot 2026-08-07, openrouter.ai/models (per-1M-token, in/out; US rows only)
5. Chinese-vendor listed prices, vendor-reported snapshot Aug 7 2026 (Chinese rows only): Zhipu bigmodel.cn/pricing · Volcano Engine volcengine.com/pricing?product=ark&tab=2 · SiliconFlow siliconflow.cn/pricing
6. OpenRouter free tier policy, snapshot Aug 7 2026, openrouter.ai/docs/api/reference/limits
7. DeepSeek V4 re-run raw logs, 2026-08-07 (same grading module scored the 22-call V4 rerun): https://huggingface.co/datasets/EliChen-ai/china-ai-bench-benchmarks/tree/main/raw-logs/deepseek-v4-pro