FUMO / Research · Model evaluation and behavior analysis
A comparative view of model scores, reasoning-trace text similarity, item-level outcomes, and version differences under documented evaluation conditions.
How to read: Each outer model is compared only with the center. Distances are rescaled within the selected center model and show relative order only; use the printed 0–1 values for absolute differences.
Current finding
Performance on the same tasks / GPT 5.5 → 5.6 Sol → 6 Astra
Across the same 68 tasks and 3 attempts per task, task pass rate followed 81.37% → 86.27% → 89.71% across versions. 5.5 → 5.6 Sol: coding implementation records 4 more passes; extraction & compression records 2 more passes; translation & rewriting records 2 fewer passes. 5.6 Sol → 6 Astra: code analysis records 6 more passes; translation & rewriting records 2 more passes; agent tool use records 2 fewer passes.
Pass rate · three attempts per task
Sol strengthens implementation and information processing; Astra improves code analysis and restores translation performance. Implementation gains are smaller, while tool execution slips slightly.
Observed · FUMO Capability Evaluation · Each model answers the same 68 tasks three times. Bars show passes by category; changes compare 6 Astra with 5.6 Sol. Five categories with the largest differences are shown. See methods
Capability complementarity
Whether totals are similar or far apart, models may share some passes and each solve items the other misses. Both-pass, one-sided-pass, and both-fail counts reveal capability overlap, complementarity, and gaps hidden by totals.
24 configurations are eligible. Each pair uses only shared tasks with three graded attempts per model. Score cards retain each model’s own coverage; some models still have missing results.
Across 68 matched items, 6 are passed only by DeepSeek V4 Flash and 15 only by DeepSeek V4 Pro; both pass 29, and neither passes 18.
FUMO Capability Evaluation · 24 models. Seven quality dimensions use case-family-equal aggregation, not task pass rates.
openai · xhigh
Check-level labels can coexist with a task pass. These exploratory, non-exclusive counts are neither failed-answer totals nor prevalence estimates.
All reported values and conclusions are conditional on the listed tasks, model versions, run configurations, and scoring rules.
Similarity, category-level differences, and failure labels describe observations under specified conditions. Estimates may change with the task distribution, reasoning settings, tool policy, or model version.
Shared passes indicate overlap, while one-sided passes indicate relative strengths; together with shared failures, they describe complementarity and gaps within the evaluated task set.
Task coverage, repetition count, and model-version changes all affect the estimates. Rare failures and cross-domain conclusions require additional independent replication.
Methods & data
From raw runs to displayed metrics, each step retains an explicit comparison unit and denominator. The rules below describe the data shown on this page, rather than a generic methodology statement.
Lock model version, reasoning configuration, tasks, and repeats.
Identify infrastructure failures, incomplete runs, and unresolved grades.
Set the denominator actually used by each metric.
Compute attempt success and stable-task pass separately.
Partition outcomes only where both models have valid results.
A run-level measure whose denominator excludes invalid observations.
Used only for repeated-run benchmarks and kept separate from attempt success.
The four cells sum to the eligible matched items for the active model pair.
Trace-prefix intervention intervals use task-level resampling, keeping repeated observations grouped within each task.
Two 0–1 text scales are reported separately; results with different task coverage are not pooled.
One answer may carry multiple check labels even when it passes; counts are not prevalence estimates.
FUMO Capability Evaluation data updated 2026-09-12 · Evaluation methods. Evaluation data is continually reviewed. Additional runs and scoring-rule revisions may change scores and eligible sample counts.
TM