Two evaluations, both OpenAI’s
OpenAI has published two coding evaluations aimed at the languages its customers say models handle badly, and the results that go with them, in a research post on 19 October. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in each of twenty named languages. A second evaluation asks it to migrate COBOL programs to another language without changing what they do.
Every number below is OpenAI’s, produced by OpenAI running its own models. The evaluation setup and test suite for MultiLangBench have been released on GitHub with per-language results, so the first half of that is checkable; the scores themselves have not yet been reproduced by anyone else.
COBOL is the floor
At maximum reasoning effort, OpenAI reports GPT-6 Astra passing an average of 87.4% of tests across the twenty languages, against 26.8% for GPT-5.6 Sol. The company says the model scores above 83% in nineteen of the twenty — and that the twentieth is COBOL, at 72.1%.
That is still a large move from where it started. OpenAI puts GPT-5.6 Sol at 10.9% on COBOL, so the jump is roughly sixty-one points. The other low-resource languages move further: Fortran from 21.4% to 83.5%, OCaml from 25.6% to 85.8%, Visual Basic from 27.5% to 89.4%.
The cheaper model tells a different story. GPT-6 Luna averages 40.3% across the twenty languages at maximum reasoning effort — better than GPT-5.6 Sol in every language tested, but far below Astra, and on COBOL it reaches only 17.1%. What it buys instead is price: OpenAI estimates an average API cost of $0.77 a sample against $3.48 for GPT-5.6 Sol, roughly 78% lower.
The migration test is nearly finished
The second evaluation is the one with the operational point. It asks a model to migrate COBOL programs to C++, Java, Kotlin, Python or Rust while preserving behaviour, and it is graded strictly: hidden tests require byte-for-byte parity on observable outputs, and a migration counts only if every test passes.
OpenAI reports GPT-6.1 Sol at 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. Turn the reasoning effort up and accuracy reaches 97.9% — which OpenAI itself describes as “nearly saturating the evaluation.” A benchmark published at 97.9% has roughly one release of headroom left in it.
The reason strict grading matters here is arithmetic. COBOL’s fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested, so a rewrite that rounds instead will run cleanly and produce different financial results. OpenAI notes that applying this to production software “still requires expert review and testing against the system’s business rules and dependencies.”
Why these languages
OpenAI’s framing is that these languages hold up infrastructure nobody can switch off. Citing outside sources, it says COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, and that Fortran is used in weather forecasting and nuclear reactor safety analysis. The difficulty, it says, is that there is far less code and documentation in these languages for a model to learn from, documentation is often incomplete, the original developers have left, and important business rules exist only in the code.
What to watch
The useful test of this release is not the headline average but what happens when someone outside OpenAI runs MultiLangBench. The suite is public, the per-language results are published, and the gap between 87.4% on average and 72.1% on COBOL is specific enough to check. The other thing to watch is whether a benchmark its author has already pushed to 97.9% gets replaced with a harder one, or quietly retired.
