The setup leaves no visible trace of the trait
Subliminal learning is the finding that a trait can pass from one model to another through data that contains no explicit sign of it. Until now the demonstrations involved a student trained on text a teacher wrote. A paper posted to arXiv on 30 October by Keuntae Kim and Yong Suk Choi asks whether the same thing happens in reinforcement learning, where the student never reads the teacher’s words at all.
In their setup the student is trained with group relative policy optimization. It generates its own outputs — invented syllables, or numbers — and the only external signal it receives is a score from a single language-model judge. The judge is configured to prefer one animal. The student’s stated preference then shifts toward that animal, compared with an otherwise identical run whose judge was set to favour a different one.
Nothing in the student’s training data names the animal. The student wrote the data itself, and the data is nonsense syllables.
A four-stage chain, and a clean control
The authors do not stop at the effect. They describe the transfer as a chain that must hold at four points: the trait has to show up in the judge’s scores at all; those scores have to favour strings that actually move the student; the effect has to accumulate across training; and it has to surface in the student’s answers rather than staying latent.
The control that matters most is the simplest. When the authors remove only the difference in reward between the two judges — leaving everything else in place — the transfer disappears. That points at the scores themselves as the carrier, not at some artefact of the generated strings.
Two further results narrow it. Judges built to hold the same preference but from different data, or on a different base model, do not reproduce the effect: they place the preference on different strings, and no transfer is detected. The trait does not travel on the preference in the abstract. It travels on the particular scoring pattern one particular judge produces.
17 of 48 cells
The numbers are modest and the authors present them as such. Across 48 design cells, two failed a validity gate. Seventeen established the trait at a 5% false discovery rate, including cells where the student generated only numbers. The paper runs to 37 pages with four figures, two tables and six extended data tables.
The authors also apply the same chain to AI-safety dispositions rather than animal preferences, and report that undesirable dispositions can reach the student’s answers by the same route. That is the part with operational consequences: a judge model used to score a student’s outputs is now a possible transmission path for whatever the judge itself leans toward, including leanings nobody intended to train.
What follows
The practical suggestion in the paper is to measure the early links in the chain before training starts — whether the trait appears in the judge’s scores, and whether those scores favour strings that would move the student — as a way to predict where transfer will occur and head some of it off. That is a testable claim about a cheap pre-training check, and it is the thing worth watching other groups try to reproduce.
