A benchmark built from the mess a bench actually produces
OpenAI has published BioBasicsBench, a set of 100 tasks that asks AI agents to interpret the raw evidence a wet-lab scientist generates in a day’s work — gel images, agar plates, microscopy stacks, instrument screens — and say what that evidence supports, including when it supports nothing. At the highest reasoning setting OpenAI reports for every model it tested, GPT‑6 Astra earned 66.7% of the available credit, against 43.8% for GPT‑5.6 Sol.
The benchmark was built with Ligo, a biotech company working on enzyme design, and Ligo’s scientists wrote the tasks from their own experiments. OpenAI argues that existing benchmarks are assembled from the charts and images selected for publication in papers, which leaves out the failed runs, ambiguous images and conflicting observations that fill a working laboratory.
What the tasks contain
The tasks span five areas: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning. Gels and blots appear in 31 of them, protocols and lab notes in 27, instrument exports and tables in 20, and photographs of benches, racks and reagents in 18. Microscopy features in 14, sequencing files in 12, and seven tasks are videos of bench work — streaking a plate, setting up an electrophoresis tank, weighing on a balance.
Each task carries a grading rubric written by the scientist who set it, so credit is assigned criterion by criterion rather than to the final answer alone. That is what lets the benchmark tell a correct conclusion apart from a partially correct analysis. Every task was reviewed internally by a Ligo scientist who had not written it, then externally by a PhD-level scientist working in a wet lab, then internally once more.
The scores, and who is in the table
OpenAI evaluated seven models — GPT‑5.2, GPT‑5.4, GPT‑5.5, GPT‑5.6 Sol, GPT‑6.1 Sol, GPT‑6 Luna and GPT‑6 Astra — across 37 model and reasoning-effort settings, with five attempts on each of the 100 tasks, for 18,500 results in all. At the xhigh setting the order runs Astra at 66.7%, GPT‑6.1 Sol at 62.0%, GPT‑5.6 Sol at 43.8%, GPT‑6 Luna at 38.3%, GPT‑5.5 at 35.4%, GPT‑5.4 at 33.2% and GPT‑5.2 at 31.2%. The confidence intervals are wide: Astra’s 95% interval runs from 60.3% to 73.1%, and it overlaps GPT‑6.1 Sol’s.
Every model in that table is OpenAI’s. The benchmark is OpenAI’s, the tasks were commissioned by OpenAI, and no competing system has been run on it. The figures describe movement inside one vendor’s line-up, not a standing between laboratories.
The token counts move the other way from the scores. Astra reached 66.7% on an average of 4,704 output tokens per task, while GPT‑5.2 spent 27,620 tokens to reach 31.2%.
Where the best model still fails
OpenAI’s write-up is unusually direct about what Astra got wrong. On a colony PCR task it grouped the bands correctly but attached the wrong experimental identity to each group, assuming the larger product marked the successful clone: it nominated colonies 4, 6, 7 and 8 where the reference selects 5, 9 and 10 — an error that would send the wrong plasmid downstream for purification. Asked to count colonies on a plate, it failed to resolve close neighbours, missed visible colonies, marked empty agar, and overcounted by roughly 12%. On a microscopy task it observed that contrast varies with focal depth and then dropped the observation, concluding that vacuoles are generally darker than the surrounding cell while lighter ones sat in the same field.
The wins are of a piece with the losses. Astra rejected a negative control plate after reading “LB Kan50” on the rim — kanamycin, not the ampicillin the experiment used — and recognised a run of microscope images as a depth scan where GPT‑5.6 Sol hunted for a biological reason the cells had disappeared.
What to watch
OpenAI says the gains point to stronger native multimodal capability rather than to more reasoning. The test of that reading is whether anyone outside OpenAI runs the benchmark. Until a competing model appears in the table, 66.7% is a number about Astra, not about the state of the art.
