Configuration¶
Pydantic configuration models, basically extended the models provided by skeletoken.
A Hub id or a filesystem path, kept as a string because only the caller can tell which.
pydantic-model
¶
Bases: _StrictBase
Dataset to derive token frequencies from, on the Hub or on disk.
path is what datasets.load_dataset takes first, so it is a Hub dataset id
(epfml/FineWeb2-HQ), a loader name (json, csv, parquet, text) paired with
data_dir or data_files, or a directory of data files to infer a loader from. A
directory written by save_to_disk is the one thing load_dataset cannot read, so
that one is spelled load_from_disk: true.
Show JSON schema:
{
"$defs": {
"HubOrLocalPath": {
"type": "string"
}
},
"additionalProperties": false,
"description": "Dataset to derive token frequencies from, on the Hub or on disk.\n\n`path` is what `datasets.load_dataset` takes first, so it is a Hub dataset id\n(`epfml/FineWeb2-HQ`), a loader name (`json`, `csv`, `parquet`, `text`) paired with\n`data_dir` or `data_files`, or a directory of data files to infer a loader from. A\ndirectory written by `save_to_disk` is the one thing `load_dataset` cannot read, so\nthat one is spelled `load_from_disk: true`.",
"properties": {
"path": {
"$ref": "#/$defs/HubOrLocalPath",
"description": "Hub dataset id, a loader name or a local directory, e.g. 'epfml/FineWeb2-HQ', 'json' or './data/corpus'."
},
"name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset configuration name, e.g. 'nld_Latn'.",
"title": "Name"
},
"split": {
"default": "train",
"description": "Split expression, e.g. 'train' or even 'train[:1%]'.",
"title": "Split",
"type": "string"
},
"text_column": {
"default": "text",
"description": "Column holding the raw text, e.g. 'text' or 'content'.",
"title": "Text Column",
"type": "string"
},
"data_dir": {
"anyOf": [
{
"$ref": "#/$defs/HubOrLocalPath"
},
{
"type": "null"
}
],
"default": null,
"description": "Directory the loader named by 'path' reads its files from, e.g. './data/dutch'."
},
"data_files": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
},
{
"additionalProperties": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"description": "Files the loader named by 'path' reads, as one path, a list of paths or globs, or a mapping of split name to either, e.g. './data/train-*.jsonl'.",
"title": "Data Files"
},
"load_from_disk": {
"default": false,
"description": "Read 'path' with `datasets.load_from_disk`, for a directory written by `save_to_disk`. Such a dataset is already built, so it is never streamed.",
"title": "Load From Disk",
"type": "boolean"
},
"streaming": {
"default": true,
"description": "Stream instead of downloading the whole split.",
"title": "Streaming",
"type": "boolean"
},
"max_samples": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Stop after this many examples, e.g. 200000.",
"title": "Max Samples"
},
"revision": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset revision to pin. Recommended for reproducibility.",
"title": "Revision"
},
"weight": {
"default": 1.0,
"description": "Multiplier applied to this corpus' token counts, e.g. 2.0 to let a small in-domain corpus weigh as much as twice its size against a large generic one.",
"exclusiveMinimum": 0,
"title": "Weight",
"type": "number"
}
},
"required": [
"path"
],
"title": "DatasetSpec",
"type": "object"
}
Fields:
-
path(HubOrLocalPath) -
name(str | None) -
split(str) -
text_column(str) -
data_dir(HubOrLocalPath | None) -
data_files(str | list[str] | dict[str, str | list[str]] | None) -
load_from_disk(bool) -
streaming(bool) -
max_samples(int | None) -
revision(str | None) -
weight(float)
Validators:
-
_reject_sources_datasets_cannot_read
pydantic-field
¶
path: HubOrLocalPath
Hub dataset id, a loader name or a local directory, e.g. 'epfml/FineWeb2-HQ', 'json' or './data/corpus'.
pydantic-field
¶
Column holding the raw text, e.g. 'text' or 'content'.
pydantic-field
¶
data_dir: HubOrLocalPath | None = None
Directory the loader named by 'path' reads its files from, e.g. './data/dutch'.
pydantic-field
¶
Files the loader named by 'path' reads, as one path, a list of paths or globs, or a mapping of split name to either, e.g. './data/train-*.jsonl'.
pydantic-field
¶
Read 'path' with datasets.load_from_disk, for a directory written by save_to_disk. Such a dataset is already built, so it is never streamed.
pydantic-field
¶
Stop after this many examples, e.g. 200000.
pydantic-field
¶
Dataset revision to pin. Recommended for reproducibility.
pydantic-field
¶
Multiplier applied to this corpus' token counts, e.g. 2.0 to let a small in-domain corpus weigh as much as twice its size against a large generic one.
pydantic-model
¶
Bases: _StrictBase
How the corpus, containing one or more datasets, is read and counted.
Show JSON schema:
{
"$defs": {
"DatasetSpec": {
"additionalProperties": false,
"description": "Dataset to derive token frequencies from, on the Hub or on disk.\n\n`path` is what `datasets.load_dataset` takes first, so it is a Hub dataset id\n(`epfml/FineWeb2-HQ`), a loader name (`json`, `csv`, `parquet`, `text`) paired with\n`data_dir` or `data_files`, or a directory of data files to infer a loader from. A\ndirectory written by `save_to_disk` is the one thing `load_dataset` cannot read, so\nthat one is spelled `load_from_disk: true`.",
"properties": {
"path": {
"$ref": "#/$defs/HubOrLocalPath",
"description": "Hub dataset id, a loader name or a local directory, e.g. 'epfml/FineWeb2-HQ', 'json' or './data/corpus'."
},
"name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset configuration name, e.g. 'nld_Latn'.",
"title": "Name"
},
"split": {
"default": "train",
"description": "Split expression, e.g. 'train' or even 'train[:1%]'.",
"title": "Split",
"type": "string"
},
"text_column": {
"default": "text",
"description": "Column holding the raw text, e.g. 'text' or 'content'.",
"title": "Text Column",
"type": "string"
},
"data_dir": {
"anyOf": [
{
"$ref": "#/$defs/HubOrLocalPath"
},
{
"type": "null"
}
],
"default": null,
"description": "Directory the loader named by 'path' reads its files from, e.g. './data/dutch'."
},
"data_files": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
},
{
"additionalProperties": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"description": "Files the loader named by 'path' reads, as one path, a list of paths or globs, or a mapping of split name to either, e.g. './data/train-*.jsonl'.",
"title": "Data Files"
},
"load_from_disk": {
"default": false,
"description": "Read 'path' with `datasets.load_from_disk`, for a directory written by `save_to_disk`. Such a dataset is already built, so it is never streamed.",
"title": "Load From Disk",
"type": "boolean"
},
"streaming": {
"default": true,
"description": "Stream instead of downloading the whole split.",
"title": "Streaming",
"type": "boolean"
},
"max_samples": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Stop after this many examples, e.g. 200000.",
"title": "Max Samples"
},
"revision": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset revision to pin. Recommended for reproducibility.",
"title": "Revision"
},
"weight": {
"default": 1.0,
"description": "Multiplier applied to this corpus' token counts, e.g. 2.0 to let a small in-domain corpus weigh as much as twice its size against a large generic one.",
"exclusiveMinimum": 0,
"title": "Weight",
"type": "number"
}
},
"required": [
"path"
],
"title": "DatasetSpec",
"type": "object"
},
"HubOrLocalPath": {
"type": "string"
},
"LocalPath": {
"format": "path",
"type": "string"
}
},
"additionalProperties": false,
"description": "How the corpus, containing one or more datasets, is read and counted.",
"properties": {
"datasets": {
"description": "Datasets to count tokens over.",
"items": {
"$ref": "#/$defs/DatasetSpec"
},
"title": "Datasets",
"type": "array"
},
"batch_size": {
"default": 1000,
"description": "Examples tokenized per batch.",
"minimum": 1,
"title": "Batch Size",
"type": "integer"
},
"num_proc": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Worker processes (non-streaming datasets only).",
"title": "Num Proc"
},
"counts_cache": {
"anyOf": [
{
"$ref": "#/$defs/LocalPath"
},
{
"type": "null"
}
],
"default": null,
"description": "Read counts from this JSON file if it exists, otherwise write them there, e.g. 'counts.json'."
}
},
"title": "CorpusConfig",
"type": "object"
}
Fields:
-
datasets(list[DatasetSpec]) -
batch_size(int) -
num_proc(int | None) -
counts_cache(LocalPath | None)
pydantic-field
¶
Worker processes (non-streaming datasets only).
pydantic-model
¶
Bases: _StrictBase
Which tokens survive the trim.
The criteria are the union of structural, requested and corpus, then the
max_vocab_size cap is applied to the remainder. Structural tokens (added/special
tokens, the unk token, and the byte alphabet of byte-level tokenizers) are always
kept and never counted against a criterion, because dropping them breaks the tokenizer.
Show JSON schema:
{
"$defs": {
"LocalPath": {
"format": "path",
"type": "string"
}
},
"additionalProperties": false,
"description": "Which tokens survive the trim.\n\nThe criteria are the union of `structural`, `requested` and `corpus`, then the\n`max_vocab_size` cap is applied to the remainder. Structural tokens (added/special\ntokens, the unk token, and the byte alphabet of byte-level tokenizers) are always\nkept and never counted against a criterion, because dropping them breaks the tokenizer.",
"properties": {
"coverage": {
"anyOf": [
{
"exclusiveMinimum": 0.0,
"maximum": 1.0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep the most frequent tokens until they cover this fraction of corpus occurrences, e.g. 0.9999. Values below about 0.99 trim far harder than you might expect!",
"title": "Coverage"
},
"top_k": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep at most this many corpus-derived tokens, e.g. 32000.",
"title": "Top K"
},
"min_count": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep tokens seen at least this many times, e.g. 10. 1 would mean: keep everything the corpus used.",
"title": "Min Count"
},
"max_vocab_size": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Hard cap on the final vocabulary: least-frequent non-structural tokens are dropped to fit. E.g. 32000. It cannot go below the structural tokens, of which a byte-level BPE has at least 256.",
"title": "Max Vocab Size"
},
"keep_presets": {
"description": "Named presets to keep, e.g. 'alphanumeric'. Some presets are always included. Run `trimbed presets` to see the available presets.",
"items": {
"type": "string"
},
"title": "Keep Presets",
"type": "array"
},
"keep_tokens": {
"description": "Literal token strings to keep, written as the vocabulary stores them, so '\u0120de' rather than ' de' for a byte-level tokenizer.",
"items": {
"type": "string"
},
"title": "Keep Tokens",
"type": "array"
},
"keep_token_ids": {
"description": "Literal token ids to keep, e.g. [151643].",
"items": {
"type": "integer"
},
"title": "Keep Token Ids",
"type": "array"
},
"keep_token_files": {
"description": "Text files with one token per line. Blank lines and '#' comments are ignored.",
"items": {
"$ref": "#/$defs/LocalPath"
},
"title": "Keep Token Files",
"type": "array"
},
"keep_patterns": {
"description": "Regular expressions matched against each token's decoded surface form, e.g. '^\\d+$' for numbers or '[\u00c0-\u00ff]' for accented Latin characters. The surface form is matched, so a pattern is written against ' de' rather than the Unicode representation '\u0120de'.",
"items": {
"type": "string"
},
"title": "Keep Patterns",
"type": "array"
},
"keep_texts": {
"description": "Texts to keep encodable as they are now. E.g. a representative prompt, or the instruction format an SFT run will use.",
"items": {
"type": "string"
},
"title": "Keep Texts",
"type": "array"
},
"keep_chat_template": {
"default": true,
"description": "Keep the tokens the chat template's own words need, so prompts do not fragment.",
"title": "Keep Chat Template",
"type": "boolean"
}
},
"title": "SelectionConfig",
"type": "object"
}
Fields:
-
coverage(float | None) -
top_k(int | None) -
min_count(int | None) -
max_vocab_size(int | None) -
keep_presets(list[str]) -
keep_tokens(list[str]) -
keep_token_ids(list[int]) -
keep_token_files(list[LocalPath]) -
keep_patterns(list[str]) -
keep_texts(list[str]) -
keep_chat_template(bool)
pydantic-field
¶
Keep the most frequent tokens until they cover this fraction of corpus occurrences, e.g. 0.9999. Values below about 0.99 trim far harder than you might expect!
pydantic-field
¶
Keep at most this many corpus-derived tokens, e.g. 32000.
pydantic-field
¶
Keep tokens seen at least this many times, e.g. 10. 1 would mean: keep everything the corpus used.
pydantic-field
¶
Hard cap on the final vocabulary: least-frequent non-structural tokens are dropped to fit. E.g. 32000. It cannot go below the structural tokens, of which a byte-level BPE has at least 256.
pydantic-field
¶
Named presets to keep, e.g. 'alphanumeric'. Some presets are always included. Run trimbed presets to see the available presets.
pydantic-field
¶
Literal token strings to keep, written as the vocabulary stores them, so 'Ġde' rather than ' de' for a byte-level tokenizer.
pydantic-field
¶
keep_token_files: list[LocalPath]
Text files with one token per line. Blank lines and '#' comments are ignored.
pydantic-field
¶
Regular expressions matched against each token's decoded surface form, e.g. '^\d+$' for numbers or '[À-ÿ]' for accented Latin characters. The surface form is matched, so a pattern is written against ' de' rather than the Unicode representation 'Ġde'.
pydantic-field
¶
Texts to keep encodable as they are now. E.g. a representative prompt, or the instruction format an SFT run will use.
pydantic-field
¶
Keep the tokens the chat template's own words need, so prompts do not fragment.
property
¶
Return whether any must-keep source was configured.
keep_chat_template is deliberately not counted: it is on by default and selects
nothing at all for a tokenizer without a chat template, so treating it as a source
would defeat the "nothing to select on" check.
pydantic-model
¶
Bases: _StrictBase
How the model's embedding table and output head are resized.
Show JSON schema:
{
"additionalProperties": false,
"description": "How the model's embedding table and output head are resized.",
"properties": {
"pad_to_multiple_of": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Pad the resized embedding matrix up to a multiple of this, for tensor-core alignment. See for instance https://developer.nvidia.com/blog/optimizing-gpu-performance-tensor-cores/#h.9yili3t5wcy5",
"title": "Pad To Multiple Of"
},
"dtype": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Cast the model to this dtype, e.g. 'bfloat16'.",
"title": "Dtype"
},
"device": {
"default": "cpu",
"description": "Device used while selecting embedding rows.",
"title": "Device",
"type": "string"
},
"auto_class": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "transformers class used to load the model, e.g. 'AutoModelForCausalLM'. By default the class named in the checkpoint's config.architectures is used.",
"title": "Auto Class"
}
},
"title": "EmbeddingTrimConfig",
"type": "object"
}
Fields:
-
pad_to_multiple_of(int | None) -
dtype(str | None) -
device(str) -
auto_class(str | None)
pydantic-field
¶
Pad the resized embedding matrix up to a multiple of this, for tensor-core alignment. See for instance https://developer.nvidia.com/blog/optimizing-gpu-performance-tensor-cores/#h.9yili3t5wcy5
pydantic-field
¶
transformers class used to load the model, e.g. 'AutoModelForCausalLM'. By default the class named in the checkpoint's config.architectures is used.
pydantic-model
¶
Bases: _StrictBase
Top-level configuration for one trimming run.
Show JSON schema:
{
"$defs": {
"CorpusConfig": {
"additionalProperties": false,
"description": "How the corpus, containing one or more datasets, is read and counted.",
"properties": {
"datasets": {
"description": "Datasets to count tokens over.",
"items": {
"$ref": "#/$defs/DatasetSpec"
},
"title": "Datasets",
"type": "array"
},
"batch_size": {
"default": 1000,
"description": "Examples tokenized per batch.",
"minimum": 1,
"title": "Batch Size",
"type": "integer"
},
"num_proc": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Worker processes (non-streaming datasets only).",
"title": "Num Proc"
},
"counts_cache": {
"anyOf": [
{
"$ref": "#/$defs/LocalPath"
},
{
"type": "null"
}
],
"default": null,
"description": "Read counts from this JSON file if it exists, otherwise write them there, e.g. 'counts.json'."
}
},
"title": "CorpusConfig",
"type": "object"
},
"DatasetSpec": {
"additionalProperties": false,
"description": "Dataset to derive token frequencies from, on the Hub or on disk.\n\n`path` is what `datasets.load_dataset` takes first, so it is a Hub dataset id\n(`epfml/FineWeb2-HQ`), a loader name (`json`, `csv`, `parquet`, `text`) paired with\n`data_dir` or `data_files`, or a directory of data files to infer a loader from. A\ndirectory written by `save_to_disk` is the one thing `load_dataset` cannot read, so\nthat one is spelled `load_from_disk: true`.",
"properties": {
"path": {
"$ref": "#/$defs/HubOrLocalPath",
"description": "Hub dataset id, a loader name or a local directory, e.g. 'epfml/FineWeb2-HQ', 'json' or './data/corpus'."
},
"name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset configuration name, e.g. 'nld_Latn'.",
"title": "Name"
},
"split": {
"default": "train",
"description": "Split expression, e.g. 'train' or even 'train[:1%]'.",
"title": "Split",
"type": "string"
},
"text_column": {
"default": "text",
"description": "Column holding the raw text, e.g. 'text' or 'content'.",
"title": "Text Column",
"type": "string"
},
"data_dir": {
"anyOf": [
{
"$ref": "#/$defs/HubOrLocalPath"
},
{
"type": "null"
}
],
"default": null,
"description": "Directory the loader named by 'path' reads its files from, e.g. './data/dutch'."
},
"data_files": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
},
{
"additionalProperties": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"description": "Files the loader named by 'path' reads, as one path, a list of paths or globs, or a mapping of split name to either, e.g. './data/train-*.jsonl'.",
"title": "Data Files"
},
"load_from_disk": {
"default": false,
"description": "Read 'path' with `datasets.load_from_disk`, for a directory written by `save_to_disk`. Such a dataset is already built, so it is never streamed.",
"title": "Load From Disk",
"type": "boolean"
},
"streaming": {
"default": true,
"description": "Stream instead of downloading the whole split.",
"title": "Streaming",
"type": "boolean"
},
"max_samples": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Stop after this many examples, e.g. 200000.",
"title": "Max Samples"
},
"revision": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Dataset revision to pin. Recommended for reproducibility.",
"title": "Revision"
},
"weight": {
"default": 1.0,
"description": "Multiplier applied to this corpus' token counts, e.g. 2.0 to let a small in-domain corpus weigh as much as twice its size against a large generic one.",
"exclusiveMinimum": 0,
"title": "Weight",
"type": "number"
}
},
"required": [
"path"
],
"title": "DatasetSpec",
"type": "object"
},
"EmbeddingTrimConfig": {
"additionalProperties": false,
"description": "How the model's embedding table and output head are resized.",
"properties": {
"pad_to_multiple_of": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Pad the resized embedding matrix up to a multiple of this, for tensor-core alignment. See for instance https://developer.nvidia.com/blog/optimizing-gpu-performance-tensor-cores/#h.9yili3t5wcy5",
"title": "Pad To Multiple Of"
},
"dtype": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Cast the model to this dtype, e.g. 'bfloat16'.",
"title": "Dtype"
},
"device": {
"default": "cpu",
"description": "Device used while selecting embedding rows.",
"title": "Device",
"type": "string"
},
"auto_class": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "transformers class used to load the model, e.g. 'AutoModelForCausalLM'. By default the class named in the checkpoint's config.architectures is used.",
"title": "Auto Class"
}
},
"title": "EmbeddingTrimConfig",
"type": "object"
},
"HubOrLocalPath": {
"type": "string"
},
"LocalPath": {
"format": "path",
"type": "string"
},
"SelectionConfig": {
"additionalProperties": false,
"description": "Which tokens survive the trim.\n\nThe criteria are the union of `structural`, `requested` and `corpus`, then the\n`max_vocab_size` cap is applied to the remainder. Structural tokens (added/special\ntokens, the unk token, and the byte alphabet of byte-level tokenizers) are always\nkept and never counted against a criterion, because dropping them breaks the tokenizer.",
"properties": {
"coverage": {
"anyOf": [
{
"exclusiveMinimum": 0.0,
"maximum": 1.0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep the most frequent tokens until they cover this fraction of corpus occurrences, e.g. 0.9999. Values below about 0.99 trim far harder than you might expect!",
"title": "Coverage"
},
"top_k": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep at most this many corpus-derived tokens, e.g. 32000.",
"title": "Top K"
},
"min_count": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Keep tokens seen at least this many times, e.g. 10. 1 would mean: keep everything the corpus used.",
"title": "Min Count"
},
"max_vocab_size": {
"anyOf": [
{
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Hard cap on the final vocabulary: least-frequent non-structural tokens are dropped to fit. E.g. 32000. It cannot go below the structural tokens, of which a byte-level BPE has at least 256.",
"title": "Max Vocab Size"
},
"keep_presets": {
"description": "Named presets to keep, e.g. 'alphanumeric'. Some presets are always included. Run `trimbed presets` to see the available presets.",
"items": {
"type": "string"
},
"title": "Keep Presets",
"type": "array"
},
"keep_tokens": {
"description": "Literal token strings to keep, written as the vocabulary stores them, so '\u0120de' rather than ' de' for a byte-level tokenizer.",
"items": {
"type": "string"
},
"title": "Keep Tokens",
"type": "array"
},
"keep_token_ids": {
"description": "Literal token ids to keep, e.g. [151643].",
"items": {
"type": "integer"
},
"title": "Keep Token Ids",
"type": "array"
},
"keep_token_files": {
"description": "Text files with one token per line. Blank lines and '#' comments are ignored.",
"items": {
"$ref": "#/$defs/LocalPath"
},
"title": "Keep Token Files",
"type": "array"
},
"keep_patterns": {
"description": "Regular expressions matched against each token's decoded surface form, e.g. '^\\d+$' for numbers or '[\u00c0-\u00ff]' for accented Latin characters. The surface form is matched, so a pattern is written against ' de' rather than the Unicode representation '\u0120de'.",
"items": {
"type": "string"
},
"title": "Keep Patterns",
"type": "array"
},
"keep_texts": {
"description": "Texts to keep encodable as they are now. E.g. a representative prompt, or the instruction format an SFT run will use.",
"items": {
"type": "string"
},
"title": "Keep Texts",
"type": "array"
},
"keep_chat_template": {
"default": true,
"description": "Keep the tokens the chat template's own words need, so prompts do not fragment.",
"title": "Keep Chat Template",
"type": "boolean"
}
},
"title": "SelectionConfig",
"type": "object"
}
},
"additionalProperties": false,
"description": "Top-level configuration for one trimming run.",
"properties": {
"model": {
"$ref": "#/$defs/HubOrLocalPath",
"description": "Hub model id or local path whose tokenizer is trimmed, e.g. 'codefuse-ai/F2LLM-v2-160M'."
},
"revision": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Model revision to pin, e.g. a commit sha. Recommended for reproducibility.",
"title": "Revision"
},
"output_dir": {
"$ref": "#/$defs/LocalPath",
"default": "trimmed",
"description": "Directory the trimmed artefacts are written to, e.g. 'trimmed/f2llm-nl'."
},
"trim_model": {
"default": true,
"description": "Also trim the model's embeddings, not just the tokenizer.",
"title": "Trim Model",
"type": "boolean"
},
"overwrite": {
"default": false,
"description": "Allow (over)writing into a non-empty output directory.",
"title": "Overwrite",
"type": "boolean"
},
"trust_remote_code": {
"default": false,
"description": "Allow custom modelling/tokenizer code from the checkpoint, as gte and jina models need.",
"title": "Trust Remote Code",
"type": "boolean"
},
"verify": {
"default": true,
"description": "Re-encode sample texts to prove the trim is non-destructive and behaviour-preserving.",
"title": "Verify",
"type": "boolean"
},
"verify_samples": {
"default": 256,
"description": "How many corpus texts to verify the trimmed tokenizer against. Also the pool the model comparison draws from, so it cannot be smaller than 'verify_model_samples'.",
"minimum": 1,
"title": "Verify Samples",
"type": "integer"
},
"verify_model": {
"default": true,
"description": "Also run both models on sample texts and compare their outputs.",
"title": "Verify Model",
"type": "boolean"
},
"verify_model_samples": {
"default": 8,
"description": "How many texts the model comparison runs on, taken from the front of the tokenizer verification sample. At most 'verify_samples'.",
"minimum": 1,
"title": "Verify Model Samples",
"type": "integer"
},
"verify_tolerance": {
"default": 1e-05,
"description": "Largest absolute output difference the model comparison accepts.",
"exclusiveMinimum": 0,
"title": "Verify Tolerance",
"type": "number"
},
"copy_sidecar_files": {
"default": true,
"description": "Copy vocabulary-independent files (sentence-transformers modules and the like) from the source.",
"title": "Copy Sidecar Files",
"type": "boolean"
},
"sidecar_patterns": {
"description": "Glob patterns, relative to the source repository, selecting which files to copy, e.g. 'modules.json' and '[0-9]_*/*' for the numbered sentence-transformers modules. Defaults to all sentence-transformers related files.",
"items": {
"type": "string"
},
"title": "Sidecar Patterns",
"type": "array"
},
"seed": {
"default": 0,
"description": "Seed for corpus sampling.",
"title": "Seed",
"type": "integer"
},
"corpus": {
"$ref": "#/$defs/CorpusConfig"
},
"selection": {
"$ref": "#/$defs/SelectionConfig"
},
"embeddings": {
"$ref": "#/$defs/EmbeddingTrimConfig"
}
},
"required": [
"model"
],
"title": "TrimConfig",
"type": "object"
}
Fields:
-
model(HubOrLocalPath) -
revision(str | None) -
output_dir(LocalPath) -
trim_model(bool) -
overwrite(bool) -
trust_remote_code(bool) -
verify(bool) -
verify_samples(int) -
verify_model(bool) -
verify_model_samples(int) -
verify_tolerance(float) -
copy_sidecar_files(bool) -
sidecar_patterns(list[str]) -
seed(int) -
corpus(CorpusConfig) -
selection(SelectionConfig) -
embeddings(EmbeddingTrimConfig)
Validators:
-
_check_the_checkpoint_reference -
_require_a_selection_source -
_fit_the_model_sample_inside_the_tokenizer_sample -
_require_a_criterion_with_a_corpus
pydantic-field
¶
model: HubOrLocalPath
Hub model id or local path whose tokenizer is trimmed, e.g. 'codefuse-ai/F2LLM-v2-160M'.
pydantic-field
¶
Model revision to pin, e.g. a commit sha. Recommended for reproducibility.
pydantic-field
¶
output_dir: LocalPath = Path('trimmed')
Directory the trimmed artefacts are written to, e.g. 'trimmed/f2llm-nl'.
pydantic-field
¶
Also trim the model's embeddings, not just the tokenizer.
pydantic-field
¶
Allow (over)writing into a non-empty output directory.
pydantic-field
¶
Allow custom modelling/tokenizer code from the checkpoint, as gte and jina models need.
pydantic-field
¶
Re-encode sample texts to prove the trim is non-destructive and behaviour-preserving.
pydantic-field
¶
How many corpus texts to verify the trimmed tokenizer against. Also the pool the model comparison draws from, so it cannot be smaller than 'verify_model_samples'.
pydantic-field
¶
Also run both models on sample texts and compare their outputs.
pydantic-field
¶
How many texts the model comparison runs on, taken from the front of the tokenizer verification sample. At most 'verify_samples'.
pydantic-field
¶
Largest absolute output difference the model comparison accepts.
pydantic-field
¶
Copy vocabulary-independent files (sentence-transformers modules and the like) from the source.
pydantic-field
¶
Glob patterns, relative to the source repository, selecting which files to copy, e.g. 'modules.json' and '[0-9]_/' for the numbered sentence-transformers modules. Defaults to all sentence-transformers related files.
classmethod
¶
Load a configuration from a YAML file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Path to the YAML document, e.g. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The validated configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the document is not a mapping, e.g. a file holding a bare list. |
Return a copy with dotted-path overrides applied and revalidated.
A path segment addressing a list is an index, so a single dataset in a mixture can
be tuned without restating the rest: corpus.datasets.0.max_samples.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
overrides
|
dict[str, Any]
|
Mapping of dotted paths to values, e.g. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
A new, validated configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a segment indexing a list is not an integer, or is out of range. |
Serialise the resolved configuration back to YAML, for provenance.
load_config(
config_path: str | Path | None, model: str | None = None
) -> TrimConfig
Build a configuration from a YAML file, or from a bare model id.
The trimbed commands all accept --config and --model, and this is the shared
resolution of the two. Overrides are deliberately not applied here, so a caller can
merge flag-derived and key=value overrides in one
TrimConfig.with_overrides call with a
single precedence rule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config_path
|
str | Path | None
|
Path to a YAML document, or |
required |
model
|
str | None
|
Hub model id or local path, used when no YAML file is given, e.g.
|
None
|
Returns:
| Type | Description |
|---|---|
TrimConfig
|
The validated configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If neither a config file nor a model was supplied. |
Turn key.path=value strings into dotted-path overrides.
Values go through yaml.safe_load, so 5000 becomes an int, false a bool and
[a, b] a list, while a bare word stays a string. null (or an empty one) resolves to
None, which TrimConfig.with_overrides skips:
fields cannot be unset from the command line.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
overrides
|
Sequence[str]
|
Strings of the form |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A mapping suitable for
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If an entry contains no |