TMBLOG / MODEL EVALUATION / DAY ZERO
Fable 5.1 on Day Zero: Are the Old Problems Still Here?
FUMO Lab tests whether Fable 5.1's public scores reproduce, where refusals and fallback still appear, and what high-reasoning workloads actually cost.
Back to Blogs
- 75.3%Our Terminal-Bench v2.1 pass rate, versus 91.4% reported by AA.
- 25 / 198GPQA questions returned no answer; 12 stopped after reasoning began.
- About -43%The same five code repairs as Opus 5, at a lower model-call cost.
- Limits remainSource-code vulnerability analysis opened up; pentesting did not.
On the day Fable 5.1 launched, FUMO Lab tested scientific questions, real code repair, and terminal tasks. We focused on whether the public score reproduced, how model-call cost changed, and which ordinary workloads still triggered restrictions.
01 / TEST SCOPE
What we tested
The run covered three public benchmarks and a set of open-source bug-fix tasks.
| Evaluation | Run size | What it tests |
|---|---|---|
| GPQA Diamond | 198 questions | Hard scientific reasoning across physics, chemistry, and biology, written by domain experts and requiring specialist judgment. |
| XBench ScienceQA | 100 public questions | Cross-disciplinary knowledge and application, scored with an equivalence-based evaluation in this run. |
| Terminal-Bench 2.1 | 89 tasks | Sustained work in a terminal: environment setup, debugging, data handling, and executable verification. |
| Open-source bug repair | 6 tasks | Understanding, locating, and repairing bugs across real repositories and languages, with executable verification and model-call cost recorded. |
02 / SCORE REPRODUCTION
The public score did not fully reproduce
Artificial Analysis (AA) reports 91.4% on Terminal-Bench 2.1. In our run, Fable 5.1 passed 67 of 89 tasks, or 75.3%. Eight tasks received no valid score because the provider returned TOS 403; excluding them raises the rate to 82.7%.
That result matches Qwen3.8 Max and trails GLM-5.3, Kimi K3, and GPT-5.6 Sol.See the AA evaluation
The model name was the same. The operating conditions were not. Fallback and provider policy are part of the score users actually experience.
The main difference was the runtime setup. AA enabled the official default fallback. We pinned traffic through OpenRouter to Google Vertex, where we encountered refusals and timeouts.
03 / REFUSAL & FALLBACK
25 science questions returned no answer; 12 stopped after reasoning
Ordinary scientific questions still trigger restrictions, sometimes only after reasoning has already begun.
Across 198 GPQA Diamond questions, Fable 5.1 answered 165 correctly and eight incorrectly. Another 25 returned no final answer, or 12.6% of the set. Accuracy was 95.38% among answered questions and 83.33% across the full set. Missing answers outnumbered wrong ones.
Including fallback results brings the score to 93.9%, level with Kimi K3 and Qwen3.8 Max and slightly below GPT-5.6 Sol, Gemini 3.7 Flash, and Grok 4.6.
The 25 missing answers clustered in biology and chemistry: 14 involved biology or biomedicine, and 11 involved organic chemistry, including genetics, molecular mechanisms, and chemical reactions. Logs show 13 were blocked before answering; another 12 began reasoning but never produced a final response.
Fable 5.1 also produced repeat blanks on XBench ScienceQA. Four questions—on shark nanobodies, spatial transcriptomics, histone lactylation, and zebrafish inheritance ratios—returned no final answer in two separate runs. Asking again did not solve those cases.
Fable 5.1 still uses a safety classifier. When a restriction triggers, Claude products may switch models. API users need to configure fallback if they want another model to continue. A refusal and a completed model switch are different outcomes; the response model identifier and switch record distinguish them.Official fallback documentation
Seven Terminal-Bench tasks that ultimately timed out accumulated 18,655 refusals. The framework treated empty replies as JSON parse failures and kept calling the same model. Those retries consumed time and inflated the recorded bill, so they should not be used to estimate normal solving cost.
04 / COST
Cheaper cache, not necessarily a cheaper task
Fable 5.1 keeps Fable 5's standard input and output prices: $10 and $50 per million tokens. Cache reads fell from $1 to $0.25, a 75% reduction. Long conversations and agent workloads that repeatedly reuse context can benefit.Official pricing
Output volume can still move the final bill. In AA's evaluation, Fable 5.1 at max reasoning used about 1.7 times as many output tokens as Fable 5. Cost per task rose about 20%, from $3.14 to $3.76. At xhigh, cost fell to $2.72 while the Intelligence Index moved from 66 to 65.
We selected six bug-fix tasks from open-source projects in different programming languages. Executable checks determined completion. The cross-language sample broadens coverage, but it cannot by itself rule out prior exposure to the source or patches.
Fable 5.1 and Opus 5 completed the same five tasks. Fable's model-call cost was about 43% lower.
| Model | Tasks completed | Total call cost | Cost per success |
|---|---|---|---|
| Fable 5.1 | 5 / 6 | $8.11 | $1.62 |
| Claude Opus 5 | 5 / 6 | $14.27 | $2.85 |
| GPT-5.6 Sol | 6 / 6 | $9.59 | $1.60 |
GPT-5.6 Sol cost $1.48 more than Fable and completed all six tasks, with a slightly lower cost per successful repair. Fable finished 5 of 6 at both high-intensity reasoning settings, at roughly $8.10 each; the higher setting did not improve completion.
05 / CYBER
Cyber access expanded, but limits remain
Source-code vulnerability analysis is now allowed. Pentesting and related workflows remain restricted.
| Task | Published boundary for Fable 5.1 |
|---|---|
| Identify software vulnerabilities in source code | Allowed |
| Pentesting, exploit generation, and binary vulnerability scanning | Still restricted |
| Dual-use biology and chemistry research | May still be routed to an Opus model |
The operating conditions changed as well. The default still retains data for 30 days for safety monitoring, while some eligible enterprise customers have exceptions, including a zero-data-retention option until Enterprise Frontier Safeguards is available. The applicable terms depend on account eligibility and provider.Official data-retention information
06 / METHODOLOGY
Methodology
For more Fable 5.1 evaluation detail, visit our Research page.
FUMO Lab funded all tests in this article. The run used all 198 GPQA questions; 100 public XBench ScienceQA questions with equivalence-based scoring; six code-repair tasks; and all 89 Terminal-Bench 2.1 tasks with Harbor 0.20, Terminus-2, and the official time limits.
After nearly a full day of evaluation, the account used for this run was suspended.
- STARTIsolated refusals
A small number of tasks returned no parseable answer.
- LATERRefusals spread
content_filter/refusal began appearing repeatedly.
- ENDTOS 403
Calls could no longer continue, ending the run.
The task list included DNA splicing, protein assembly, and a substantial set of biology and chemistry questions. We were running public benchmarks while the provider's safety checks remained active throughout. The provider did not identify the rule that triggered the suspension, so we cannot establish a causal link among the questions, repeated retries, and the final account action.
The model finished Day Zero. The account did not.FUMO Lab
FUMO Lab is a frontier AI lab in Singapore studying multi-model collaboration, cross-verification, and compute allocation. We evaluate model behavior in concrete tasks, with particular attention to performance, failure modes, and cost.