Pipeline¶

trimbed.pipeline ¶

End-to-end trimming pipeline.

TrimPipeline(config: TrimConfig)

Runs a configured trim from corpus counting through to saved artefacts.

The stages are exposed as methods so callers can drive them individually, like our entrypoints trimbed count and trimbed inspect do.

Build a pipeline.

Parameters:

Name	Type	Description	Default
`config`	`TrimConfig`	The validated run configuration.	required

load() -> tuple[PreTrainedTokenizerFast, TokenizerSpec]

Load the tokenizer and parse its backend document (typically tokenizer.json).

Returns:

Type	Description
`tuple[PreTrainedTokenizerFast, TokenizerSpec]`	The fast tokenizer and a spec over its tokenizer.json, e.g. a `Qwen2Tokenizer` and a BPE spec of 151,669 tokens.

count(
    tokenizer: PreTrainedTokenizerFast,
) -> CorpusCounts | None

Count token frequencies over the configured corpus.

Parameters:

Name	Type	Description	Default
`tokenizer`	`PreTrainedTokenizerFast`	The tokenizer used to encode the corpus.	required

Returns:

Type	Description
`CorpusCounts \| None`	The corpus statistics, or `None` when no dataset was configured.

run(dry_run: bool = False) -> TrimReport

Execute the whole pipeline.

Parameters:

Name	Type	Description	Default
`dry_run`	`bool`	Select and report but write nothing to disk. Useful for sweeping different `top_k` values against a cached count file to see what each value would keep.	`False`

Returns:

Type	Description
`TrimReport`	The report describing what happened.