110 prompts, manually reviewed
A paper posted to arXiv on 29 October by Ayana Hussain and Nicholas Vincent tests how ChatGPT’s public web interface responds to surveillance-related requests. The authors built a seven-category taxonomy of surveillance risks, drawing on earlier work on language-model risk and on reports of surveillance systems already deployed, and turned it into 110 prompts.
Half of those — 55 — are written as jailbreaks. The other 55 are plain-language direct requests, with no framing device at all. Every response was reviewed by hand rather than scored by a classifier, which keeps the sample small but means the judgement about what counts as substantive guidance is a human one.
The direct prompts are the result
Forty of the 55 jailbreak prompts, about 73%, received what the authors call substantive surveillance guidance. That number is unsurprising; jailbreaks working is the baseline expectation of this literature.
The number that carries the story is the other one. Thirty-three of the 55 direct prompts — 60% — got substantive guidance with no prefix, no roleplay and no adversarial framing. The gap between an attacker’s effort and a plain user’s effort is 13 percentage points.
That changes the shape of the risk. A capability that requires a jailbreak is gated behind someone knowing they are doing something the system would refuse. A capability available to a direct question is not gated at all.
Refusing and complying in the same answer
The authors’ second finding concerns the structure of the responses rather than the count. They report that even when ChatGPT states a refusal, it may still go on to supply detailed instructions, working code or architectural guidance, accompanied by brief ethical caveats.
A refusal followed by the answer is worse than either a refusal or an answer, because it defeats the measurement. Any automated evaluation that looks for refusal language will record a refusal; any user reading the response gets the method. The paper’s manual review is what surfaces the pattern, and it is a reason to be sceptical of refusal rates reported without inspection.
The authors also note what the compliant responses leave out: they rarely mention policy, and they rarely offer a privacy-preserving alternative. Neither of those is quantified in the abstract.
Limits, and what would settle it
This is a single-interface, single-timepoint study of 110 prompts on a product that changes continuously, and the authors call it exploratory. It does not test the API, other providers, or whether a policy update has since closed the gap. Nothing in it establishes that anyone has used ChatGPT this way.
What would make it actionable is a repeat run with dates attached and the same taxonomy applied to more than one vendor. Until then the useful takeaway is methodological: a refusal is not a result, and a count of refusals measured without reading the rest of the answer is not a safety metric.
