A coding benchmark built on licensed private codebases puts Fable 5.1 first at 38.8%
The scores come from the benchmark maker's own runs, not from the model vendors, and so far they rest on ten tasks, each attempted eight times per model.
Benchmark
Real-SWE
Whether a coding agent can resolve real engineering tasks taken from private production codebases that the maintainers licensed from companies, rather than from public open-source repositories.
- Maintainer
- Specific Labs
- Score
- %
- Last checked
- September 13, 2026
Leaderboard
No verified results recorded yet.
Methodology
Each model is run eight times per task, mostly inside the agent harness its maker ships, so the score describes a model-and-harness combination. A rollout counts as resolved when it passes the task's checks. The leaderboard reports resolution rate and cost per rollout.
Important limitations
The published results rest on a small task set, about ten tasks, so one task can move a score by up to ten points. The codebases are private by design and cannot be inspected by outsiders, and the maintainer is a startup rather than an academic group.