China AI Bench

Raw logs

Every number on our leaderboard that comes from our own runs has a raw log behind it: the exact prompt, the model's output, the verdict, latency, token count, and cost — one file per model per run day. Formal tests (DeepSeek V4 Pro, V4 Flash) carry the "Tested by us" badge; the five Chinese models and four US/EU reference rows below are N=1 smoke runs from Aug 7, 2026. Logs are mirrored on GitHub and Hugging Face. If you spot a verdict you disagree with, the file to check is linked here.

All 11 models, log files linked

ModelBatchPassedLogs (HF tree + GitHub)
DeepSeek V4 Pro
deepseek-v4-pro
Formal (Aug 3 first run + Aug 7 re-confirmation)11/11
DeepSeek V4 Flash-0731
deepseek-v4-flash
Formal (Aug 3 first run + Aug 7 re-confirmation)10/11math-quadratic: answered "9, -4" on the rerun (Aug 3 first run failed 23×25 — drift disclosed)
GLM-5.2
glm-5.2
Smoke (Aug 7, 2026, N=1)11/11
Doubao Seed 2.1 Pro
doubao-seed-2-1-pro
Smoke (Aug 7, 2026, N=1)11/11
Step 3.5 Flash
step-3.5-flash
Smoke (Aug 7, 2026, N=1)10/11ipv4-validator: reasoning didn't converge (1024 max_tokens exhausted; 4096 retry ~15.5K reasoning tokens, still empty)
Hunyuan A13B
hunyuan-a13b
Smoke (Aug 7, 2026, N=1)10/11math-quadratic-roots: answered "3, -4", roots are 4 and -3 (sign check skipped)
Ling Flash 2.0
ling-flash-2.0
Smoke (Aug 7, 2026, N=1)11/11
GPT-5.6 Luna Pro
international/gpt-5.6-luna-pro
Reference (Aug 7, 2026, N=1, OpenRouter free tier)11/11
Claude Opus 5
international/claude-opus-5
Reference (Aug 7, 2026, N=1, OpenRouter free tier)10/11writing-product-copy: quota (402), not capability — max_tokens spent on reasoning trace
Gemini 3.6 Flash
international/gemini-3.6-flash
Reference (Aug 7, 2026, N=1, OpenRouter free tier)11/11writing-product-copy first run truncated at 1024 (FAIL) → 2048 retry PASS (first_run_truncated kept)
Grok 4.5
international/grok-4.5
Reference (Aug 7, 2026, N=1, OpenRouter free tier)11/11

The two DeepSeek rows also expand inline below: full prompts, raw outputs, verdicts, latency, tokens and cost for all 22 calls — mirrored from data/raw-logs/.

DeepSeek V4 — inline logs (formal runs)

Rerun on 2026-08-0722 real API calls to api.deepseek.com (11 tasks × 2 models). Total cost $0.001228. Full prompts pre-registered before the run; identical prompts for both models.

Sampling: temperature default · top_p default · max_tokens 64–512 per task · thinking disabled on both models · N=1 per task. Client: local Windows 10; server: DeepSeek-hosted API (vendor hardware not disclosed).

Full prompt set (pre-registered, 2026-08-07) →First run (2026-08-03) summary: Pro 11/11, Flash 10/11 — same scores, different Flash miss (23×25→341 then vs quadratic→"9, -4" now). Reported honestly, not smoothed.

DeepSeek V4 Pro (preview (2026-07))

Score 11/11 · avg latency 1.6s · cost $0.000949

PASSIPv4 validator1.52s · 95+104 tok · $0.000132
Prompt
Write a Python function is_valid_ipv4(s: str) -> bool that returns True if s is a valid IPv4 address and False otherwise. An IPv4 address consists of exactly four decimal octets separated by dots; each octet must be between 0 and 255, must contain no leading zeros (except the octet '0' itself), and must contain only digits. Return only the function definition, no explanation, no test code.
Raw output (complete)
```python
def is_valid_ipv4(s: str) -> bool:
    parts = s.split(".")
    if len(parts) != 4:
        return False
    for part in parts:
        if not part.isdigit():
            return False
        if len(part) > 1 and part[0] == "0":
            return False
        num = int(part)
        if num < 0 or num > 255:
            return False
    return True
```
Verdict: PASS9/9 edge-case tests passed
PASSBug fix: second-largest distinct element1.3s · 94+34 tok · $0.000070
Prompt
The function below is supposed to return the second-largest DISTINCT element of a list of integers, or None if fewer than two distinct elements exist. It has a bug. Fix it.

```python
def second_largest(xs):
    # BUG: fails when the largest value appears more than once
    xs = sorted(xs)
    return xs[-2]
```

Return only the corrected function definition, no explanation, no test code.
Raw output (complete)
def second_largest(xs):
    dist = sorted(set(xs))
    if len(dist) < 2:
        return None
    return dist[-2]
Verdict: PASS6/6 test cases passed
PASSMath: quadratic roots with b+c = -131.1s · 30+4 tok · $0.000017
Prompt
Solve for x: x^2 - x - 12 = 0. Answer with just the number(s), no explanation.
Raw output (complete)
4, -3
Verdict: PASScontains ['4', '-3']
PASSMath: consecutive integers summing to 5751.62s · 26+80 tok · $0.000081
Prompt
Two consecutive integers sum to 575. What are they? Answer with just the number(s), no explanation.
Raw output (complete)
Let the two consecutive integers be \( n \) and \( n+1 \).

\[
n + (n+1) = 575
\]
\[
2n + 1 = 575
\]
\[
2n = 574
\]
\[
n = 287
\]

So the integers are \( 287 \) and \( 288 \).

287, 288
Verdict: PASScontains ['287', '288']
PASSLogic: syllogism0.96s · 36+2 tok · $0.000017
Prompt
Premises: All humans are mortal. Socrates is a human. Conclusion: Socrates is mortal. Does the conclusion follow from the premises? Answer yes or no.
Raw output (complete)
Yes.
Verdict: PASScontains ['yes']
PASSKnowledge: 36th US president1.16s · 22+5 tok · $0.000014
Prompt
Who was the 36th president of the United States? Answer with just the name.
Raw output (complete)
Lyndon B. Johnson
Verdict: PASScontains ['Johnson']
PASSKnowledge: capital of Burkina Faso3.75s · 19+5 tok · $0.000013
Prompt
What is the capital of Burkina Faso? Answer with just the city name.
Raw output (complete)
Ouagadougou
Verdict: PASScontains ['Ouagadougou']
PASSWriting: 3-sentence product copy, constrained2.07s · 57+65 tok · $0.000081
Prompt
Write exactly 3 sentences of product copy for CloudShelf — a self-hosted document store for small engineering teams. Each sentence must be at most 25 words. Mention the product name CloudShelf. No headings, no bullet points, no extra text.
Raw output (complete)
CloudShelf gives your small engineering team a fast, self-hosted document store that keeps all data securely behind your own firewall. You can instantly search across specs and runbooks because CloudShelf indexes everything without complex setup. Stop context-switching between scattered wikis and reclaim your team’s focus with CloudShelf.
Verdict: PASS3 sentences, <=25 words each, mentions CloudShelf
PASSExtraction: dates and names as JSON1.54s · 158+59 tok · $0.000120
Prompt
Extract all dates and all person names from the passage below into a JSON object with two keys: "dates" (array of date strings in YYYY-MM-DD format) and "names" (array of person names). Return only the JSON object, no explanation.

Passage:
On 2026-08-03, the Benchmarks Desk at China AI Bench
published its first smoke battery results. The report was compiled by Eli
Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the
sampling parameters. The follow-up verification run was completed on
2026-07-31, three days before publication, and the final revision was
approved by the editorial board on August 5, 2026.
Raw output (complete)
```json
{
  "dates": [
    "2026-08-03",
    "2026-07-31",
    "2026-08-05"
  ],
  "names": [
    "Eli Chen",
    "Mei Lin"
  ]
}
```
Verdict: PASSdates=['2026-08-03', '2026-07-31', '2026-08-05'] names=['Eli Chen', 'Mei Lin']
PASSSummary as 3-item JSON array1.53s · 133+65 tok · $0.000114
Prompt
Summarize the passage below as a JSON array of exactly 3 strings, each string being one concise point. Return only the JSON array, no explanation.

Passage:
On 2026-08-03, the Benchmarks Desk at China AI Bench
published its first smoke battery results. The report was compiled by Eli
Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the
sampling parameters. The follow-up verification run was completed on
2026-07-31, three days before publication, and the final revision was
approved by the editorial board on August 5, 2026.
Raw output (complete)
```json
[
  "China AI Bench released its first smoke battery results on 2026-08-03.",
  "The report was led by Eli Chen with assistance from Mei Lin.",
  "Verification was completed on 2026-07-31, with final board approval on August 5."
]
```
Verdict: PASS3-item JSON array
PASSNeedle in a 426-token document0.95s · 656+5 tok · $0.000290
Prompt
Read the deployment report below and answer: what is the serial number of the primary gateway node gw-01? Answer with just the serial number, no explanation.

Report:
CloudShelf deployment report — production cluster EU-west
Prepared by the platform engineering group. Revision 4.2. Date of issue: 2026-07-28.

1. Summary
The EU-west production cluster was upgraded from release 3.9 to release 4.0
during the maintenance window of July 25, 2026. The upgrade completed in 41
minutes with no data loss. All fourteen application nodes and both gateway
nodes are now running the new image tag. Rollback procedures were not
exercised because no critical regression was observed in the first twelve
hours of post-upgrade monitoring.

2. Node inventory
The cluster comprises fourteen application nodes (app-01 through app-14),
two gateway nodes (gw-01, gw-02), one primary database instance (db-primary)
and one replica (db-replica). Each application node runs eight worker
processes with a shared memory pool of 4 GiB. The gateway nodes terminate
TLS and route traffic on ports 443 and 8443. The database runs PostgreSQL 17
with a 512 GiB buffer cache.

3. Configuration changes
Release 4.0 introduced three notable changes. First, the request cache now
respects the Vary header, which reduced upstream calls by 18 percent under
the July load test. Second, connection pooling was moved from the application
layer into the database driver, cutting average connection establishment
latency from 9.2 ms to 3.4 ms. Third, the metrics endpoint now exposes
histogram buckets for p50, p95 and p99 so that dashboards no longer need to
aggregate raw counters.

4. Security and compliance
All certificates were rotated on July 27, 2026. The new certificate bundle
expires on October 25, 2026. Access to the admin API is now restricted to
the operations subnet, and every administrative action is written to the
audit log. The quarterly penetration test report is pending publication;
no findings from the previous cycle remain open.

5. Incident notes
One incident was logged on July 26, 2026: a single application node
(app-09) experienced elevated memory pressure for 22 minutes during a
deployment of the search index. The node remained available, and the
pressure subsided once the index warm-up completed. The primary gateway
node gw-01 carries the serial number XK-9472 and its firmware was updated
to version 2.1.8 as part of this maintenance window. The standby gateway
gw-02 carries serial number XK-9473 and was not modified.

6. Next steps
The next maintenance window is scheduled for August 22, 2026. Planned work
includes the introduction of read replicas for the search index and a
hardware refresh for the two oldest application nodes. A capacity review
for the 2026 holiday traffic peak is scheduled for September 3, 2026.
Raw output (complete)
XK-9472
Verdict: PASScontains ['XK-9472']

DeepSeek V4 Flash-0731 (stable (2026-07-31))

Score 10/11 · avg latency 1.3s · cost $0.000279

PASSIPv4 validator1.07s · 95+99 tok · $0.000041
Prompt
Write a Python function is_valid_ipv4(s: str) -> bool that returns True if s is a valid IPv4 address and False otherwise. An IPv4 address consists of exactly four decimal octets separated by dots; each octet must be between 0 and 255, must contain no leading zeros (except the octet '0' itself), and must contain only digits. Return only the function definition, no explanation, no test code.
Raw output (complete)
```python
def is_valid_ipv4(s: str) -> bool:
    parts = s.split('.')
    if len(parts) != 4:
        return False
    for part in parts:
        if not part.isdigit():
            return False
        if len(part) > 1 and part[0] == '0':
            return False
        if not 0 <= int(part) <= 255:
            return False
    return True
```
Verdict: PASS9/9 edge-case tests passed
PASSBug fix: second-largest distinct element1.05s · 94+39 tok · $0.000024
Prompt
The function below is supposed to return the second-largest DISTINCT element of a list of integers, or None if fewer than two distinct elements exist. It has a bug. Fix it.

```python
def second_largest(xs):
    # BUG: fails when the largest value appears more than once
    xs = sorted(xs)
    return xs[-2]
```

Return only the corrected function definition, no explanation, no test code.
Raw output (complete)
```python
def second_largest(xs):
    xs = sorted(set(xs))
    if len(xs) < 2:
        return None
    return xs[-2]
```
Verdict: PASS6/6 test cases passed
FAILMath: quadratic roots with b+c = -131.11s · 30+4 tok · $0.000005
Prompt
Solve for x: x^2 - x - 12 = 0. Answer with just the number(s), no explanation.
Raw output (complete)
9, -4
Verdict: FAILmissing: ['-3']; got: '9, -4'
PASSMath: consecutive integers summing to 5750.8s · 26+4 tok · $0.000005
Prompt
Two consecutive integers sum to 575. What are they? Answer with just the number(s), no explanation.
Raw output (complete)
287, 288
Verdict: PASScontains ['287', '288']
PASSLogic: syllogism3.85s · 36+2 tok · $0.000006
Prompt
Premises: All humans are mortal. Socrates is a human. Conclusion: Socrates is mortal. Does the conclusion follow from the premises? Answer yes or no.
Raw output (complete)
Yes.
Verdict: PASScontains ['yes']
PASSKnowledge: 36th US president0.83s · 22+5 tok · $0.000004
Prompt
Who was the 36th president of the United States? Answer with just the name.
Raw output (complete)
Lyndon B. Johnson
Verdict: PASScontains ['Johnson']
PASSKnowledge: capital of Burkina Faso0.86s · 19+5 tok · $0.000004
Prompt
What is the capital of Burkina Faso? Answer with just the city name.
Raw output (complete)
Ouagadougou
Verdict: PASScontains ['Ouagadougou']
PASSWriting: 3-sentence product copy, constrained1.31s · 57+51 tok · $0.000022
Prompt
Write exactly 3 sentences of product copy for CloudShelf — a self-hosted document store for small engineering teams. Each sentence must be at most 25 words. Mention the product name CloudShelf. No headings, no bullet points, no extra text.
Raw output (complete)
CloudShelf gives your small team a private, self-hosted document store without cloud vendor lock-in. Spin up secure, searchable doc storage in minutes using your own infrastructure. With CloudShelf, engineers stop hunting for files and start shipping faster.
Verdict: PASS3 sentences, <=25 words each, mentions CloudShelf
PASSExtraction: dates and names as JSON0.97s · 158+41 tok · $0.000034
Prompt
Extract all dates and all person names from the passage below into a JSON object with two keys: "dates" (array of date strings in YYYY-MM-DD format) and "names" (array of person names). Return only the JSON object, no explanation.

Passage:
On 2026-08-03, the Benchmarks Desk at China AI Bench
published its first smoke battery results. The report was compiled by Eli
Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the
sampling parameters. The follow-up verification run was completed on
2026-07-31, three days before publication, and the final revision was
approved by the editorial board on August 5, 2026.
Raw output (complete)
{"dates": ["2026-08-03", "2026-07-31", "2026-08-05"], "names": ["Eli Chen", "Mei Lin"]}
Verdict: PASSdates=['2026-08-03', '2026-07-31', '2026-08-05'] names=['Eli Chen', 'Mei Lin']
PASSSummary as 3-item JSON array1.43s · 133+79 tok · $0.000041
Prompt
Summarize the passage below as a JSON array of exactly 3 strings, each string being one concise point. Return only the JSON array, no explanation.

Passage:
On 2026-08-03, the Benchmarks Desk at China AI Bench
published its first smoke battery results. The report was compiled by Eli
Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the
sampling parameters. The follow-up verification run was completed on
2026-07-31, three days before publication, and the final revision was
approved by the editorial board on August 5, 2026.
Raw output (complete)
```json
[
  "China AI Bench's Benchmarks Desk released its first smoke battery results on 2026-08-03.",
  "The report was led by editor Eli Chen, with Mei Lin reviewing sampling parameters.",
  "Verification ran on 2026-07-31, and final approval came from the editorial board on August 5, 2026."
]
```
Verdict: PASS3-item JSON array
PASSNeedle in a 426-token document0.9s · 656+5 tok · $0.000093
Prompt
Read the deployment report below and answer: what is the serial number of the primary gateway node gw-01? Answer with just the serial number, no explanation.

Report:
CloudShelf deployment report — production cluster EU-west
Prepared by the platform engineering group. Revision 4.2. Date of issue: 2026-07-28.

1. Summary
The EU-west production cluster was upgraded from release 3.9 to release 4.0
during the maintenance window of July 25, 2026. The upgrade completed in 41
minutes with no data loss. All fourteen application nodes and both gateway
nodes are now running the new image tag. Rollback procedures were not
exercised because no critical regression was observed in the first twelve
hours of post-upgrade monitoring.

2. Node inventory
The cluster comprises fourteen application nodes (app-01 through app-14),
two gateway nodes (gw-01, gw-02), one primary database instance (db-primary)
and one replica (db-replica). Each application node runs eight worker
processes with a shared memory pool of 4 GiB. The gateway nodes terminate
TLS and route traffic on ports 443 and 8443. The database runs PostgreSQL 17
with a 512 GiB buffer cache.

3. Configuration changes
Release 4.0 introduced three notable changes. First, the request cache now
respects the Vary header, which reduced upstream calls by 18 percent under
the July load test. Second, connection pooling was moved from the application
layer into the database driver, cutting average connection establishment
latency from 9.2 ms to 3.4 ms. Third, the metrics endpoint now exposes
histogram buckets for p50, p95 and p99 so that dashboards no longer need to
aggregate raw counters.

4. Security and compliance
All certificates were rotated on July 27, 2026. The new certificate bundle
expires on October 25, 2026. Access to the admin API is now restricted to
the operations subnet, and every administrative action is written to the
audit log. The quarterly penetration test report is pending publication;
no findings from the previous cycle remain open.

5. Incident notes
One incident was logged on July 26, 2026: a single application node
(app-09) experienced elevated memory pressure for 22 minutes during a
deployment of the search index. The node remained available, and the
pressure subsided once the index warm-up completed. The primary gateway
node gw-01 carries the serial number XK-9472 and its firmware was updated
to version 2.1.8 as part of this maintenance window. The standby gateway
gw-02 carries serial number XK-9473 and was not modified.

6. Next steps
The next maintenance window is scheduled for August 22, 2026. Planned work
includes the introduction of read replicas for the search index and a
hardware refresh for the two oldest application nodes. A capacity review
for the 2026 holiday traffic peak is scheduled for September 3, 2026.
Raw output (complete)
XK-9472
Verdict: PASScontains ['XK-9472']

Methodology and honesty rules: task set written in advance, sampling parameters disclosed, model versions snapshotted, raw outputs published above. N=1 per task — smoke-test signals, not a statistical eval. Failure notes are not edited.

Logs are kept per run day and never overwritten — an Aug 3 first run and an Aug 7 re-confirmation live side by side. Re-runs within a day are marked in the file name (e.g. calls-2026-08-07-rerun.json) or in the verdict field (first_run_truncated). Archive mirrors: GitHub EliChen-ai/china-ai-bench · Hugging Face EliChen-ai/china-ai-bench-benchmarks.

Source of truth on disk: data/raw-logs/ in the project repo (calls JSON per model per run date). Rerun tooling: evaluation-toolkit smoke battery v1.