pfffp Research
AI Benchmarks & Frontier Leaderboards
Tracking real capability, evaluation contamination, and the gap between claimed performance and independent reality.
Curated snapshot · October 2026
SWE-bench Verified SOTA
95.0%
Best published resolve rate · headline benchmark
Text Arena leader
1502
Top Elo across reasoning & conversation
Open-weight leader
1500
Text Arena · first open model inside the frontier band
Design Arena leader
1489
UI generation · visual reasoning · critique
Largest claimed-vs-real gap
69%
SWE-bench Verified vs contamination-controlled Pro
00 — All arenas
Capability over time
Best published SWE-bench Verified score per frontier lab family
1.96% → 95% in 35 months, led by Claude Fable 5.5. The best open-weight score is now 82.4% (GLM-5.3). Verified is saturated and widely considered contaminated. The dashed open-frontier line tracks the best open-weight score — it closed most of the 2025 gap in 2026. Treat headline scores as an upper bound and read them next to the disparity index below.
01 — Text & reasoning
Conversational leaderboard
Blind human preference — the arena that started it all
Text Arena
Elo- 1Gemini 4 ArgonPreview1,502
- 2Kimi K3Open1,500
- 3Claude Opus 5.51,497
- 4Claude Fable 5.51,491
- 5Qwen3.8 MaxOpen1,490
- 6GPT-6 Astra1,488
- 7GLM-5.2Open1,483
- 8DeepSeek V4.1Open1,462
- 9Grok 4.71,461
- 10MiniMax M3Open1,436
Blind human preference over open conversation and reasoning prompts. Open-weight models (● Open) now sit inside the frontier Elo band for the first time; preview models are flagged until general availability.
Gemini 4 Argon holds the top position as a preview model — a signal worth watching, not yet a shipping verdict.
Blind human preference over open conversation and reasoning prompts. Open-weight models (● Open) now sit inside the frontier Elo band for the first time; preview models are flagged until general availability.
02 — Design & multimodal
Visual intelligence arena
Same-prompt builds, visual reasoning and design critique
GPT-6 Astra
UI generation · visual reasoning · design critique
1,489
- 2Claude Opus 5.51,471
- 3Gemini 4 ArgonPreview1,466
- 4Claude Fable 5.51,452
- 5Kimi K3Open1,440
- 6GLM-5.3Open1,432
- 7Grok 4.71,418
- 8DeepSeek V4.1Open1,395
Same-prompt UI builds, layout reproduction and visual reasoning, judged blind by cross-vendor panels.
GPT-6 Astra dominates this category — the widest lead of any arena we track, and the clearest signal that multimodal performance is now a first-class frontier axis.
Open-weight challengers are no longer absent from the visual arena: Kimi K3's native vision lands it fifth, with GLM-5.3 close behind.
Same-prompt UI builds, layout reproduction and visual reasoning, judged blind by cross-vendor panels.
03 — SWE & coding
Engineering arena
Independent Elo ratings and held-out resolve rates
BridgeBench — Elo-style rating
Editorial Elo-style ratings across nine software-building axes, not pass rates. Methodology is proprietary to reduce benchmark gaming. Open-weight leaders cluster 460–565 behind the closed frontier.
DeepSWE 1.1 — resolve rate
mini-swe-agent harness, program-based verifiers, isolated environments. DeepSeek V4.1 and Kimi K3 trade the top spots with the closed frontier — the open-weight gap is now under one point.
04 — Autonomy & hallucinations
How far, and how truthful
Long-horizon task completion and hallucination resistance
METR — 50% task horizon (log scale)
Doubling time (all time)
196 days
Doubling time (since 2023)
131 days
80% success horizon
~1.5h
Time Horizon 1.1 suite. Measurements above 16 hours are unreliable; the frontier is saturating the task suite.
Hallucination rate — lower is better
Artificial Analysis Omniscience measures the share of incorrect answers among responses that were not correct. Bar color marks the lab; lower is better. Open-weight models hold five of the six best positions.
05 — Disparity index
Claimed vs real-world
Vendor-reported Verified scores against contamination-controlled Pro
| Model | Claimed | Independent | Gap | Disparity index |
|---|---|---|---|---|
| GPT-5OpenAI | 74.9% | 23.1% | −51.8 pts | 69%High gap |
| Claude Opus 4.1Anthropic | 74.5% | 23.1% | −51.4 pts | 69%High gap |
| DeepSeek V4 FlashDeepSeekOpen | 79% | 52.6% | −26.4 pts | 33%High gap |
| MiniMax M2.5MiniMaxOpen | 80.2% | 55.4% | −24.8 pts | 31%High gap |
| GLM-5Z.ai (GLM)Open | 77.8% | 55.1% | −22.7 pts | 29% |
| Claude Opus 4.7Anthropic | 87.6% | 64.3% | −23.3 pts | 27% |
| Claude Fable 5.5Anthropic | 95% | 80% | −15 pts | 16% |
Verified rewards memorised public issues; SWE-bench Pro uses held-out and commercial splits with contamination control. The 2025 generation lost ~69% of its headline score; the 2026 generation is closing the gap as independent suites mature.
06 — Methodology
Sources & method
Where these numbers come from
- SWE-bench official leaderboardsVerified / Pro splits, resolve rates, contamination notes.
- LMArenaBlind human preference Elo used for the Text Arena snapshot.
- BridgeBenchIndependent vibe-coding ratings and Design Bench gallery.
- DeepSWE 1.1 (llm-stats)Independent mini-swe-agent harness results.
- METR time horizons50% and 80% task-completion horizons, TH 1.1 methodology.
- Artificial Analysis OmniscienceHallucination rates across frontier and open models.
Values are a curated editorial snapshot taken in October 2026 from the public sources above. Benchmarks differ in harness, prompting and contamination controls; preview models are flagged. Treat cross-benchmark comparisons as directional, not exact. No vendor sponsorship.