Tokenizer compatibility

—

Models checked—— distinct tokenizer files

Same in transformers.js 4—of models, on all 42 texts

Same, by downloads—share of monthly downloads

Differ on everyday text—models, in transformers.js 4

Matches Python, by runtime

Models whose ids and decoded text equal Python's on every text.

same everywherediffers on some textcannot load

Why they differ

Models affected by each cause. A model can have more than one.

Cause	transformers.js 3	transformers.js 4	tokenizers.js

Which texts trip them up

Models that differ from Python on each text. The last column counts models where the two Python references disagree with each other on it; those are not counted as JavaScript differences.

Text	Kind	transformers.js 3	transformers.js 4	tokenizers.js	Python disagrees

Every model

Each difference

One row per tokenizer file, runtime and text, under the most-downloaded model that uses the file. Tokens are shown from the first difference on.

Model	Runtime	Text	Cause	Python	JavaScript

How this was checked

The most-downloaded models on the Hugging Face Hub, taking each distinct tokenizer.json once (by its file hash). Models without one, and gated models, are left out.
Each tokenizer encodes 42 texts (everyday text, 15 languages and scripts, Unicode edge cases, stress cases) with special tokens added, then decodes Python's ids with special tokens skipped and clean-up off.
Python has two references: the tokenizers library reading the file as written, and transformers' AutoTokenizer, which adds model-specific behaviour. A difference counts only when JavaScript differs from both, or from the only one that can load the model. Where the two disagree with each other, JavaScript always sided with one of them.
Causes are matched from what each side produced and are best guesses. "Final sigma": lowercasing a capital Σ at the end of a word gives ς in JavaScript and σ in Python. "Whitespace": JavaScript treats U+FEFF as whitespace and U+0085 as not. "Byte fallback ignored": characters outside the vocabulary become one unknown token instead of bytes.
Run with python -m tokcpu compat from the tokcpu repository (daily, by a workflow, whenever a runtime has a new release), with —, on Node — (V8 —), against Python transformers — and tokenizers —, on —.