Benchmark
Whether a model can look at photographs of part-assembled IKEA furniture and identify the steps in which a mistake was made, together with a usable description of what went wrong.
No verified results recorded yet.
Sixty photographs span three IKEA builds, each photographed part-way through assembly with mistakes introduced on purpose. The model is given the assembly manual alongside the images, plus a zoom tool and a Python interpreter. Grading runs in stages: the model must name the steps that contain a mistake, and a language-model grader, GPT-5.6 Sol, judges whether the description of each mistake is correct, to a deliberately lenient standard. Epoch AI also records the median time a model takes per task.
The set covers three pieces of furniture, which Epoch AI itself notes limits how far a score generalises to physical reasoning in general. The grader is a language model rather than a person. Epoch AI both built and runs the benchmark, so the evaluation is independent of the labs whose models it scores but not of its own methodology.
An AI can now spot most mistakes in a half-built IKEA shelf, and ten months ago it could not
The set holds 60 images across three builds. Models get the manual, a zoom tool and a Python interpreter, and have to name the step that went wrong and say what went wrong in it.
Privacy preferences
We use necessary cookies to run this site. With your consent we also use advertising cookies. You can change your choice at any time. Cookie Policy · Privacy Policy
