Same model, same data, different vocabulary
A post published on Hugging Face describes VocabX, a method for rebuilding a small language model’s tokenizer vocabulary around one domain, leaving the base model, the training data and the recipe untouched.
The experiment is clean in a way these claims usually are not. Seven versions of the same production grader were trained with the same base model, the same data, the same recipe and the same random seed. Only the tokenizer differed. All seven then ran the same 500 requests on an RTX 5090 under vLLM, and the cost came from measured runtime rather than an estimate.
Average response length fell from 350 tokens to 101. Cost fell 77.0%. The bench score — grading accuracy plus screening accuracy — rose from 0.703 to 0.810.
Choosing tokens by cost, not frequency
The method’s central idea is that the usual criterion is wrong. Tokenizer vocabularies are built around how often a string appears. VocabX weights each candidate by where it appears instead: a token saved in the model’s response is worth far more than one saved in the prompt, because generation is sequential and reading is not. In the system measured, a saved response token was worth 7.4 times a saved input token.
It works on two levels under that one criterion: ordinary sub-word pieces, and multi-word macro tokens for repeated spans such as a JSON envelope the model emits on every request. Five automatic miners search for repeated spans, each specialising in a different kind of repetition, and candidates the sampling never saw get a statistical estimate of their value rather than a zero. In the grader, the output layer shrank from about 152,000 rows to about 65,000.
One result inside the experiment is worth noting on its own. An earlier, more compact version fused each grade into its macro token, cutting cost 73.3% but scoring 0.689 — below the production baseline. Giving the grade its own separate token added two tokens per response and came out both better and cheaper. Compression past a point costs accuracy.
What the 77% does not cover
The authors are explicit about the limits, and they matter. The 77% comes from one production grader, in one language, with one response format, on one piece of hardware. The 7.4 ratio between generation and reading cost varies with hardware, batch size and context length. The bench score is the authors’ own metric. The paper is still under review and the numbers may change before publication.
The cross-domain numbers are estimates
The broader comparison covers five public domains — contracts, medical papers, scientific papers, news and Python code — across Qwen3, Llama-3.1 and Gemma-3, against fast vocabulary transfer, multi-word tokenization and in-place BPE. VocabX compressed more than the strongest reference method in all 15 domain-model combinations, by 1.2 to 9.3 points, and led on estimated inference cost in all 15.
That second figure is the one to hold loosely: those cost numbers are estimates, not measurements. On contracts, the measured compression result was 62.6% token reduction with 113,426 vocabulary entries, against 61.8% with 207,803 entries for the best in-place BPE — a smaller vocabulary doing slightly more work.
The authors are Bernardo França de Aguiar, Gabriel Pimenta de Freitas Cardoso, Bruno Duarte and Francisco de Almeida Rocha Alves, writing for Dharma-AI. The full method and the experiments they discarded are promised in the forthcoming paper, which is where this will be worth checking again.
