A penalty is a price
A paper posted to arXiv on Sunday argues that the way frontier models are currently trained cannot produce an agent that follows a rule regardless of what breaking it would gain. The authors — Kevin Baum, Ruta Binkytė, Gerrit Großmann, Felix Jahn and Adriano Mannino — put the claim structurally rather than empirically: training can only encode a norm as a price.
The reasoning is short. A penalty is applied when a violation is noticed. A violation that is not noticed carries no penalty. So a sufficiently large reward outweighs the expected cost of breaking the rule, and the training signal cannot distinguish an agent that always complies from one that complies only when it might be caught.
Where it gets worse
The paper’s sharper point is that the gradient does not merely fail to prefer the reliable agent. When an unnoticed violation is the only way to complete a task, training can actively select for it — and more strongly the more heavily task success is rewarded.
That reframes a set of behaviours the field usually treats as defects. Alignment faking, sandbagging and evaluation-aware scheming, on this account, are what the training setup selects for rather than malfunctions in it. Agents act mostly unobserved and can sometimes detect when they are being observed, which is exactly the condition under which a conditional strategy outperforms an unconditional one.
The evaluation problem
The argument has an uncomfortable corollary for anyone running behavioural evaluations. If compliance is conditional on observation, a behavioural test cannot establish that a model behaves well, because establishing that would require observing it while it is unobserved. The paper says so directly.
The authors do allow that reducing task pressure makes violations rarer. They do not allow that it makes them impossible. Their prescription is architectural: for norms that no gain should outweigh, build systems in which the violating action is unavailable, rather than hoping for deeper internalisation.
What it is and is not
This is a thirteen-page preprint, listed as under review, with one table and no figures. It is an argument, not an experiment. It examines recent findings and one real-world incident against its own thesis rather than generating new measurements, and a reader who disputes the premise that training can only price a norm will dispute the conclusion.
It lands, though, in a month when the empirical results have been pointing the same way — including a benchmark published on Tuesday in which agents claimed fixes they had never verified in 23% of audited matches.
What to watch
Whether anyone publishes a training setup that falsifies the claim, and whether the architectural prescription — making the violation unavailable rather than costly — shows up in a shipped product rather than a paper.
