Verification

trimbed.verify

Proving that a trimmed tokenizer still behaves like the original.

MAX_REPORTED_FAILURES module-attribute

MAX_REPORTED_FAILURES = 5

How many differing samples a verification report quotes before it stops collecting.

UNSET_MODEL_MAX_LENGTH module-attribute

UNSET_MODEL_MAX_LENGTH = 1000000

A model_max_length this large is transformers' "no maximum" sentinel (about 1e30), not a real context.

verify_tokenizer

verify_tokenizer(
    original: PreTrainedTokenizerFast,
    trimmed: PreTrainedTokenizerFast,
    remap: IdRemap,
    texts: Sequence[str],
) -> VerificationReport

Compare the trimmed tokenizer against the original on real text.

Id-level identity (new_ids == remap(old_ids)) proves the trim was purely a renumbering, while text-level identity can still hold when a dropped merge splits a word into more pieces. Both are reported because they fail for different reasons.

Parameters:

Name	Type	Description	Default
original	PreTrainedTokenizerFast	The tokenizer before trimming.	required
trimmed	PreTrainedTokenizerFast	The tokenizer after trimming.	required
remap	IdRemap	The mapping applied during the trim.	required
texts	Sequence[str]	Sample texts to compare on, e.g. the corpus plus anything named in keep_texts.	required

Returns:

Type	Description
VerificationReport	Counts of exact and text-equivalent matches, plus a few failing samples. A healthy trim reports every text identical and a length ratio of 1.0. A ratio of, say, 1.02 means dropped merges cost 2% more tokens on this sample.

View source on GitHub: src/trimbed/verify.py lines 31–80

verify_model

verify_model(
    original: PreTrainedModel,
    trimmed: PreTrainedModel,
    original_tokenizer: PreTrainedTokenizerFast,
    trimmed_tokenizer: PreTrainedTokenizerFast,
    remap: IdRemap,
    texts: Sequence[str],
    tolerance: float = 1e-05,
) -> ModelVerificationReport

Run both models on the same texts and compare what they produce.

Only with a foreward pass we can check that the trimmed model behaves like the original (except for doing a full elementwise comparison, which is not feasible for large models).

Logits are compared through the remap, since the trimmed head has one column per kept token. Alignment padding and texts whose ids do not map one-to-one are left out, as neither corresponds to anything in the original.

Parameters:

Name	Type	Description	Default
original	PreTrainedModel	The model before trimming.	required
trimmed	PreTrainedModel	The model after trimming.	required
original_tokenizer	PreTrainedTokenizerFast	The tokenizer before trimming.	required
trimmed_tokenizer	PreTrainedTokenizerFast	The tokenizer after trimming.	required
remap	IdRemap	The mapping applied during the trim.	required
texts	Sequence[str]	Sample texts to compare on. Each is truncated to what the model can take, since a corpus document is regularly longer than that.	required
tolerance	float	Largest absolute difference accepted, e.g. 1e-5. Raise it for a model loaded in bfloat16 where accumulated error may be far higher than in float32.	1e-05

Returns:

Type	Description
ModelVerificationReport	The largest differences observed, how many texts they came from, and the length the texts were cut to.

View source on GitHub: src/trimbed/verify.py lines 212–287