Compare¶
A structural diff between two tokenizers, typically a base and a trimmed version.
A trimmed checkpoint's README tells you the recipe (which corpus, which target size) but not the result. This answers the other half: whether the smaller vocabulary really is a subset of the larger one, which structural guarantees survived, which presets and which Unicode scripts were preserved or gutted, and what the same text now costs in tokens.
Nothing here loads weights or reads a corpus. Two tokenizer.json documents go in, one
report comes out.
module-attribute
¶
How many equal slices of the base id range the removed-token profile is cut into.
module-attribute
¶
How many fragmenting sample texts the encoding comparison quotes.
module-attribute
¶
Script bucket for a token whose surface form holds no alphabetic character at all.
module-attribute
¶
Script bucket for a token whose letters come from more than one script.
module-attribute
¶
Bucket for a token that stands for no well-formed text, e.g. half a UTF-8 sequence.
pydantic-model
¶
Bases: _Base
How one group of base-vocabulary tokens fared, e.g. one Unicode script.
Show JSON schema:
{
"additionalProperties": false,
"description": "How one group of base-vocabulary tokens fared, e.g. one Unicode script.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
}
},
"required": [
"name",
"base_tokens",
"kept"
],
"title": "GroupDiff",
"type": "object"
}
Fields:
-
name(str) -
base_tokens(int) -
kept(int)
pydantic-model
¶
Bases: GroupDiff
How one named preset fared, e.g. digits or script:Latin.
Show JSON schema:
{
"additionalProperties": false,
"description": "How one named preset fared, e.g. `digits` or `script:Latin`.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
},
"always_kept": {
"description": "Whether a trim keeps this preset's tokens whether or not it is named, so any loss here means a broken tokenizer rather than a policy choice.",
"title": "Always Kept",
"type": "boolean"
}
},
"required": [
"name",
"base_tokens",
"kept",
"always_kept"
],
"title": "PresetDiff",
"type": "object"
}
Fields:
-
name(str) -
base_tokens(int) -
kept(int) -
always_kept(bool)
pydantic-field
¶
Whether a trim keeps this preset's tokens whether or not it is named, so any loss here means a broken tokenizer rather than a policy choice.
pydantic-model
¶
Bases: _Base
One token the other tokenizer no longer has.
Show JSON schema:
{
"additionalProperties": false,
"description": "One token the other tokenizer no longer has.",
"properties": {
"token": {
"description": "The token as the base vocabulary stores it, e.g. '\u0120bureaucratie'.",
"title": "Token",
"type": "string"
},
"token_id": {
"description": "Its id in the base vocabulary, e.g. 1204.",
"title": "Token Id",
"type": "integer"
},
"surface": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The text it stands for, e.g. ' bureaucratie'; null for a partial byte sequence.",
"title": "Surface"
}
},
"required": [
"token",
"token_id",
"surface"
],
"title": "RemovedToken",
"type": "object"
}
Fields:
pydantic-field
¶
The text it stands for, e.g. ' bureaucratie'; null for a partial byte sequence.
pydantic-model
¶
Bases: _Base
How the two vocabularies relate as sets of tokens.
Show JSON schema:
{
"additionalProperties": false,
"description": "How the two vocabularies relate as sets of tokens.",
"properties": {
"base_size": {
"description": "Tokens in the base vocabulary, e.g. 119547.",
"title": "Base Size",
"type": "integer"
},
"other_size": {
"description": "Tokens in the other vocabulary, e.g. 50000.",
"title": "Other Size",
"type": "integer"
},
"shared": {
"description": "Tokens both have.",
"title": "Shared",
"type": "integer"
},
"removed": {
"description": "Tokens only the base has.",
"title": "Removed",
"type": "integer"
},
"introduced": {
"description": "Tokens only the other has; zero for a genuine trim.",
"title": "Introduced",
"type": "integer"
},
"is_subset": {
"description": "Whether the other vocabulary introduces nothing new.",
"title": "Is Subset",
"type": "boolean"
},
"ids_contiguous": {
"description": "Whether the other vocabulary is numbered 0..n-1 without gaps.",
"title": "Ids Contiguous",
"type": "boolean"
},
"order_preserved": {
"description": "Whether the shared tokens keep their relative order. A trim renumbers the survivors in place, so a false here means the vocabulary was rebuilt rather than trimmed.",
"title": "Order Preserved",
"type": "boolean"
},
"introduced_examples": {
"description": "A few of the tokens only the other tokenizer has.",
"items": {
"type": "string"
},
"title": "Introduced Examples",
"type": "array"
}
},
"required": [
"base_size",
"other_size",
"shared",
"removed",
"introduced",
"is_subset",
"ids_contiguous",
"order_preserved"
],
"title": "VocabularyDiff",
"type": "object"
}
Fields:
-
base_size(int) -
other_size(int) -
shared(int) -
removed(int) -
introduced(int) -
is_subset(bool) -
ids_contiguous(bool) -
order_preserved(bool) -
introduced_examples(list[str])
pydantic-field
¶
Whether the other vocabulary is numbered 0..n-1 without gaps.
pydantic-field
¶
Whether the shared tokens keep their relative order. A trim renumbers the survivors in place, so a false here means the vocabulary was rebuilt rather than trimmed.
pydantic-field
¶
A few of the tokens only the other tokenizer has.
property
¶
Return the share of the base vocabulary that is gone, e.g. 0.582.
pydantic-model
¶
Bases: _Base
How the parts of the tokenizer around the vocabulary compare.
Show JSON schema:
{
"additionalProperties": false,
"description": "How the parts of the tokenizer around the vocabulary compare.",
"properties": {
"base_model_type": {
"description": "Backend family of the base tokenizer, e.g. 'WordPiece'.",
"title": "Base Model Type",
"type": "string"
},
"other_model_type": {
"description": "Backend family of the other tokenizer.",
"title": "Other Model Type",
"type": "string"
},
"base_uses_byte_level": {
"description": "Whether the base maps text through the ByteLevel alphabet.",
"title": "Base Uses Byte Level",
"type": "boolean"
},
"other_uses_byte_level": {
"description": "The same for the other tokenizer.",
"title": "Other Uses Byte Level",
"type": "boolean"
},
"base_unk_token": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The base's unknown token, e.g. '[UNK]', or null.",
"title": "Base Unk Token"
},
"other_unk_token": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The other's unknown token.",
"title": "Other Unk Token"
},
"base_added_tokens": {
"description": "Entries in the base's `added_tokens` list, e.g. 5.",
"title": "Base Added Tokens",
"type": "integer"
},
"other_added_tokens": {
"description": "Entries in the other's `added_tokens` list.",
"title": "Other Added Tokens",
"type": "integer"
},
"base_special_tokens": {
"description": "How many of the base's added tokens are flagged special.",
"title": "Base Special Tokens",
"type": "integer"
},
"other_special_tokens": {
"description": "The same for the other tokenizer.",
"title": "Other Special Tokens",
"type": "integer"
},
"removed_added_tokens": {
"description": "Added tokens the other tokenizer no longer has.",
"items": {
"type": "string"
},
"title": "Removed Added Tokens",
"type": "array"
},
"removed_special_tokens": {
"description": "Special tokens the other tokenizer no longer has.",
"items": {
"type": "string"
},
"title": "Removed Special Tokens",
"type": "array"
},
"removed_post_processor_tokens": {
"description": "Tokens the base's post-processor names that the other tokenizer no longer has, e.g. ['[SEP]']. Every one of them breaks encoding.",
"items": {
"type": "string"
},
"title": "Removed Post Processor Tokens",
"type": "array"
},
"chat_template": {
"description": "How the two chat templates relate.",
"enum": [
"identical",
"changed",
"only in base",
"only in other",
"absent"
],
"title": "Chat Template",
"type": "string"
}
},
"required": [
"base_model_type",
"other_model_type",
"base_uses_byte_level",
"other_uses_byte_level",
"base_unk_token",
"other_unk_token",
"base_added_tokens",
"other_added_tokens",
"base_special_tokens",
"other_special_tokens",
"chat_template"
],
"title": "ComponentDiff",
"type": "object"
}
Fields:
-
base_model_type(str) -
other_model_type(str) -
base_uses_byte_level(bool) -
other_uses_byte_level(bool) -
base_unk_token(str | None) -
other_unk_token(str | None) -
base_added_tokens(int) -
other_added_tokens(int) -
base_special_tokens(int) -
other_special_tokens(int) -
removed_added_tokens(list[str]) -
removed_special_tokens(list[str]) -
removed_post_processor_tokens(list[str]) -
chat_template(Literal['identical', 'changed', 'only in base', 'only in other', 'absent'])
pydantic-field
¶
Backend family of the base tokenizer, e.g. 'WordPiece'.
pydantic-field
¶
Whether the base maps text through the ByteLevel alphabet.
pydantic-field
¶
The same for the other tokenizer.
pydantic-field
¶
The base's unknown token, e.g. '[UNK]', or null.
pydantic-field
¶
Entries in the base's added_tokens list, e.g. 5.
pydantic-field
¶
Entries in the other's added_tokens list.
pydantic-field
¶
How many of the base's added tokens are flagged special.
pydantic-field
¶
Added tokens the other tokenizer no longer has.
pydantic-field
¶
Special tokens the other tokenizer no longer has.
pydantic-field
¶
Tokens the base's post-processor names that the other tokenizer no longer has, e.g. ['[SEP]']. Every one of them breaks encoding.
pydantic-field
¶
How the two chat templates relate.
property
¶
Return whether the other tokenizer lost a token no tokenizer can do without.
pydantic-model
¶
Bases: _Base
What kind of tokens were dropped, attributed three ways.
Show JSON schema:
{
"$defs": {
"GroupDiff": {
"additionalProperties": false,
"description": "How one group of base-vocabulary tokens fared, e.g. one Unicode script.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
}
},
"required": [
"name",
"base_tokens",
"kept"
],
"title": "GroupDiff",
"type": "object"
},
"PresetDiff": {
"additionalProperties": false,
"description": "How one named preset fared, e.g. `digits` or `script:Latin`.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
},
"always_kept": {
"description": "Whether a trim keeps this preset's tokens whether or not it is named, so any loss here means a broken tokenizer rather than a policy choice.",
"title": "Always Kept",
"type": "boolean"
}
},
"required": [
"name",
"base_tokens",
"kept",
"always_kept"
],
"title": "PresetDiff",
"type": "object"
},
"RemovedToken": {
"additionalProperties": false,
"description": "One token the other tokenizer no longer has.",
"properties": {
"token": {
"description": "The token as the base vocabulary stores it, e.g. '\u0120bureaucratie'.",
"title": "Token",
"type": "string"
},
"token_id": {
"description": "Its id in the base vocabulary, e.g. 1204.",
"title": "Token Id",
"type": "integer"
},
"surface": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The text it stands for, e.g. ' bureaucratie'; null for a partial byte sequence.",
"title": "Surface"
}
},
"required": [
"token",
"token_id",
"surface"
],
"title": "RemovedToken",
"type": "object"
}
},
"additionalProperties": false,
"description": "What kind of tokens were dropped, attributed three ways.",
"properties": {
"presets": {
"description": "One entry per registered preset that matches anything in the base vocabulary.",
"items": {
"$ref": "#/$defs/PresetDiff"
},
"title": "Presets",
"type": "array"
},
"scripts": {
"description": "One entry per dominant Unicode script, largest group first, e.g. 'LATIN', 'CYRILLIC'.",
"items": {
"$ref": "#/$defs/GroupDiff"
},
"title": "Scripts",
"type": "array"
},
"categories": {
"description": "One entry per majority Unicode category, e.g. 'letter', 'number', 'whitespace'.",
"items": {
"$ref": "#/$defs/GroupDiff"
},
"title": "Categories",
"type": "array"
},
"removed_by_decile": {
"description": "Removed tokens per tenth of the base id range, lowest ids first. A trim that only cut the tail leaves the early deciles near zero.",
"items": {
"type": "integer"
},
"title": "Removed By Decile",
"type": "array"
},
"removed_examples": {
"description": "The lowest-id removed tokens, which are the ones a trim rarely gives up.",
"items": {
"$ref": "#/$defs/RemovedToken"
},
"title": "Removed Examples",
"type": "array"
}
},
"required": [
"presets",
"scripts",
"categories",
"removed_by_decile"
],
"title": "ProfileDiff",
"type": "object"
}
Fields:
-
presets(list[PresetDiff]) -
scripts(list[GroupDiff]) -
categories(list[GroupDiff]) -
removed_by_decile(list[int]) -
removed_examples(list[RemovedToken])
pydantic-field
¶
presets: list[PresetDiff]
One entry per registered preset that matches anything in the base vocabulary.
pydantic-field
¶
scripts: list[GroupDiff]
One entry per dominant Unicode script, largest group first, e.g. 'LATIN', 'CYRILLIC'.
pydantic-field
¶
categories: list[GroupDiff]
One entry per majority Unicode category, e.g. 'letter', 'number', 'whitespace'.
pydantic-field
¶
Removed tokens per tenth of the base id range, lowest ids first. A trim that only cut the tail leaves the early deciles near zero.
pydantic-field
¶
removed_examples: list[RemovedToken]
The lowest-id removed tokens, which are the ones a trim rarely gives up.
pydantic-model
¶
Bases: _Base
What the two tokenizers do to the same text.
Show JSON schema:
{
"additionalProperties": false,
"description": "What the two tokenizers do to the same text.",
"properties": {
"checked": {
"default": 0,
"description": "Texts encoded with both tokenizers.",
"title": "Checked",
"type": "integer"
},
"identical": {
"default": 0,
"description": "Texts both split into exactly the same tokens.",
"title": "Identical",
"type": "integer"
},
"base_tokens": {
"default": 0,
"description": "Tokens the base produced over all texts.",
"title": "Base Tokens",
"type": "integer"
},
"other_tokens": {
"default": 0,
"description": "Tokens the other produced over the same texts.",
"title": "Other Tokens",
"type": "integer"
},
"examples": {
"description": "The texts that fragment the most, truncated for readability.",
"items": {
"type": "string"
},
"title": "Examples",
"type": "array"
}
},
"title": "EncodingDiff",
"type": "object"
}
Fields:
-
checked(int) -
identical(int) -
base_tokens(int) -
other_tokens(int) -
examples(list[str])
pydantic-model
¶
Bases: _Base
The complete diff between two tokenizers.
Show JSON schema:
{
"$defs": {
"ComponentDiff": {
"additionalProperties": false,
"description": "How the parts of the tokenizer around the vocabulary compare.",
"properties": {
"base_model_type": {
"description": "Backend family of the base tokenizer, e.g. 'WordPiece'.",
"title": "Base Model Type",
"type": "string"
},
"other_model_type": {
"description": "Backend family of the other tokenizer.",
"title": "Other Model Type",
"type": "string"
},
"base_uses_byte_level": {
"description": "Whether the base maps text through the ByteLevel alphabet.",
"title": "Base Uses Byte Level",
"type": "boolean"
},
"other_uses_byte_level": {
"description": "The same for the other tokenizer.",
"title": "Other Uses Byte Level",
"type": "boolean"
},
"base_unk_token": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The base's unknown token, e.g. '[UNK]', or null.",
"title": "Base Unk Token"
},
"other_unk_token": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The other's unknown token.",
"title": "Other Unk Token"
},
"base_added_tokens": {
"description": "Entries in the base's `added_tokens` list, e.g. 5.",
"title": "Base Added Tokens",
"type": "integer"
},
"other_added_tokens": {
"description": "Entries in the other's `added_tokens` list.",
"title": "Other Added Tokens",
"type": "integer"
},
"base_special_tokens": {
"description": "How many of the base's added tokens are flagged special.",
"title": "Base Special Tokens",
"type": "integer"
},
"other_special_tokens": {
"description": "The same for the other tokenizer.",
"title": "Other Special Tokens",
"type": "integer"
},
"removed_added_tokens": {
"description": "Added tokens the other tokenizer no longer has.",
"items": {
"type": "string"
},
"title": "Removed Added Tokens",
"type": "array"
},
"removed_special_tokens": {
"description": "Special tokens the other tokenizer no longer has.",
"items": {
"type": "string"
},
"title": "Removed Special Tokens",
"type": "array"
},
"removed_post_processor_tokens": {
"description": "Tokens the base's post-processor names that the other tokenizer no longer has, e.g. ['[SEP]']. Every one of them breaks encoding.",
"items": {
"type": "string"
},
"title": "Removed Post Processor Tokens",
"type": "array"
},
"chat_template": {
"description": "How the two chat templates relate.",
"enum": [
"identical",
"changed",
"only in base",
"only in other",
"absent"
],
"title": "Chat Template",
"type": "string"
}
},
"required": [
"base_model_type",
"other_model_type",
"base_uses_byte_level",
"other_uses_byte_level",
"base_unk_token",
"other_unk_token",
"base_added_tokens",
"other_added_tokens",
"base_special_tokens",
"other_special_tokens",
"chat_template"
],
"title": "ComponentDiff",
"type": "object"
},
"EncodingDiff": {
"additionalProperties": false,
"description": "What the two tokenizers do to the same text.",
"properties": {
"checked": {
"default": 0,
"description": "Texts encoded with both tokenizers.",
"title": "Checked",
"type": "integer"
},
"identical": {
"default": 0,
"description": "Texts both split into exactly the same tokens.",
"title": "Identical",
"type": "integer"
},
"base_tokens": {
"default": 0,
"description": "Tokens the base produced over all texts.",
"title": "Base Tokens",
"type": "integer"
},
"other_tokens": {
"default": 0,
"description": "Tokens the other produced over the same texts.",
"title": "Other Tokens",
"type": "integer"
},
"examples": {
"description": "The texts that fragment the most, truncated for readability.",
"items": {
"type": "string"
},
"title": "Examples",
"type": "array"
}
},
"title": "EncodingDiff",
"type": "object"
},
"GroupDiff": {
"additionalProperties": false,
"description": "How one group of base-vocabulary tokens fared, e.g. one Unicode script.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
}
},
"required": [
"name",
"base_tokens",
"kept"
],
"title": "GroupDiff",
"type": "object"
},
"PresetDiff": {
"additionalProperties": false,
"description": "How one named preset fared, e.g. `digits` or `script:Latin`.",
"properties": {
"name": {
"description": "What the group is, e.g. 'LATIN', 'letter' or 'digits'.",
"title": "Name",
"type": "string"
},
"base_tokens": {
"description": "Tokens the base tokenizer has in this group, e.g. 60003.",
"title": "Base Tokens",
"type": "integer"
},
"kept": {
"description": "How many of those the other tokenizer still has, e.g. 45102.",
"title": "Kept",
"type": "integer"
},
"always_kept": {
"description": "Whether a trim keeps this preset's tokens whether or not it is named, so any loss here means a broken tokenizer rather than a policy choice.",
"title": "Always Kept",
"type": "boolean"
}
},
"required": [
"name",
"base_tokens",
"kept",
"always_kept"
],
"title": "PresetDiff",
"type": "object"
},
"ProfileDiff": {
"additionalProperties": false,
"description": "What kind of tokens were dropped, attributed three ways.",
"properties": {
"presets": {
"description": "One entry per registered preset that matches anything in the base vocabulary.",
"items": {
"$ref": "#/$defs/PresetDiff"
},
"title": "Presets",
"type": "array"
},
"scripts": {
"description": "One entry per dominant Unicode script, largest group first, e.g. 'LATIN', 'CYRILLIC'.",
"items": {
"$ref": "#/$defs/GroupDiff"
},
"title": "Scripts",
"type": "array"
},
"categories": {
"description": "One entry per majority Unicode category, e.g. 'letter', 'number', 'whitespace'.",
"items": {
"$ref": "#/$defs/GroupDiff"
},
"title": "Categories",
"type": "array"
},
"removed_by_decile": {
"description": "Removed tokens per tenth of the base id range, lowest ids first. A trim that only cut the tail leaves the early deciles near zero.",
"items": {
"type": "integer"
},
"title": "Removed By Decile",
"type": "array"
},
"removed_examples": {
"description": "The lowest-id removed tokens, which are the ones a trim rarely gives up.",
"items": {
"$ref": "#/$defs/RemovedToken"
},
"title": "Removed Examples",
"type": "array"
}
},
"required": [
"presets",
"scripts",
"categories",
"removed_by_decile"
],
"title": "ProfileDiff",
"type": "object"
},
"RemovedToken": {
"additionalProperties": false,
"description": "One token the other tokenizer no longer has.",
"properties": {
"token": {
"description": "The token as the base vocabulary stores it, e.g. '\u0120bureaucratie'.",
"title": "Token",
"type": "string"
},
"token_id": {
"description": "Its id in the base vocabulary, e.g. 1204.",
"title": "Token Id",
"type": "integer"
},
"surface": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The text it stands for, e.g. ' bureaucratie'; null for a partial byte sequence.",
"title": "Surface"
}
},
"required": [
"token",
"token_id",
"surface"
],
"title": "RemovedToken",
"type": "object"
},
"VocabularyDiff": {
"additionalProperties": false,
"description": "How the two vocabularies relate as sets of tokens.",
"properties": {
"base_size": {
"description": "Tokens in the base vocabulary, e.g. 119547.",
"title": "Base Size",
"type": "integer"
},
"other_size": {
"description": "Tokens in the other vocabulary, e.g. 50000.",
"title": "Other Size",
"type": "integer"
},
"shared": {
"description": "Tokens both have.",
"title": "Shared",
"type": "integer"
},
"removed": {
"description": "Tokens only the base has.",
"title": "Removed",
"type": "integer"
},
"introduced": {
"description": "Tokens only the other has; zero for a genuine trim.",
"title": "Introduced",
"type": "integer"
},
"is_subset": {
"description": "Whether the other vocabulary introduces nothing new.",
"title": "Is Subset",
"type": "boolean"
},
"ids_contiguous": {
"description": "Whether the other vocabulary is numbered 0..n-1 without gaps.",
"title": "Ids Contiguous",
"type": "boolean"
},
"order_preserved": {
"description": "Whether the shared tokens keep their relative order. A trim renumbers the survivors in place, so a false here means the vocabulary was rebuilt rather than trimmed.",
"title": "Order Preserved",
"type": "boolean"
},
"introduced_examples": {
"description": "A few of the tokens only the other tokenizer has.",
"items": {
"type": "string"
},
"title": "Introduced Examples",
"type": "array"
}
},
"required": [
"base_size",
"other_size",
"shared",
"removed",
"introduced",
"is_subset",
"ids_contiguous",
"order_preserved"
],
"title": "VocabularyDiff",
"type": "object"
}
},
"additionalProperties": false,
"description": "The complete diff between two tokenizers.",
"properties": {
"base": {
"description": "Where the base tokenizer came from, e.g. 'clips/e5-small-trm-nl'.",
"title": "Base",
"type": "string"
},
"other": {
"description": "Where the other tokenizer came from, e.g. 'clips/e5-small-trm'.",
"title": "Other",
"type": "string"
},
"vocabulary": {
"$ref": "#/$defs/VocabularyDiff"
},
"components": {
"$ref": "#/$defs/ComponentDiff"
},
"profile": {
"$ref": "#/$defs/ProfileDiff"
},
"encoding": {
"anyOf": [
{
"$ref": "#/$defs/EncodingDiff"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"base",
"other",
"vocabulary",
"components",
"profile"
],
"title": "ComparisonReport",
"type": "object"
}
Fields:
-
base(str) -
other(str) -
vocabulary(VocabularyDiff) -
components(ComponentDiff) -
profile(ProfileDiff) -
encoding(EncodingDiff | None)
Write the report as JSON.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
File to write, e.g. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
The path written. |
Return the human-readable diff, one aligned label value line per topic.
Returns:
| Type | Description |
|---|---|
The text `trimbed compare` prints, e.g.
|
compare_tokenizers(
base: TokenizerSpec,
other: TokenizerSpec,
texts: Sequence[str] = (),
presets: Sequence[str] = (),
examples: int = 10,
) -> ComparisonReport
Diff two tokenizers, reading nothing but their two documents.
The comparison is directional: everything is attributed against base, so a group
reported as 0/2,341 means the base had 2,341 such tokens and other has none of
them left.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
TokenizerSpec
|
The larger, original tokenizer, e.g. clips/e5-small-trm-nl. |
required |
other
|
TokenizerSpec
|
The tokenizer to compare against it, e.g. its trimmed sibling clips/e5-small-trm. |
required |
texts
|
Sequence[str]
|
Sample texts to encode with both, e.g. a handful of sentences in the language the trim targeted. Leave empty to skip the encoding comparison. |
()
|
presets
|
Sequence[str]
|
Extra preset names to resolve on top of the registered plain ones,
which is how the parametrised ones are reached, e.g. |
()
|
examples
|
int
|
How many removed tokens and introduced tokens to quote. |
10
|
Returns:
| Type | Description |
|---|---|
The diff. For a healthy trim
|
|