Examples¶
examples/ has one runnable
script per usage pattern. Each takes -m/--model, so you can point any of them at a local
checkpoint, and each returns its report or result rather than printing and discarding it,
so the body copies straight into your own code.
| Script | What it shows |
|---|---|
01_inspect_tokenizer.py |
Build a TokenizerSpec and describe a tokenizer without changing anything |
02_trim_tokenizer_only.py |
Trim from must-keep rules alone, with no corpus and no model |
03_trim_with_corpus.py |
The full TrimPipeline over a corpus, model included |
04_custom_preset.py |
Register your own preset with register_preset |
05_low_level_api.py |
Drive the stages yourself: count, select, trim, verify |
06_local_corpus.py |
Count a corpus of local files with a DatasetSpec, offline |
The low-level path is worth a look if you want to slot trimbed into a larger job:
CorpusCounter →
select_tokens →
trim_tokenizer →
trim_model →
verify_tokenizer.
Every example runs in the test suite against tiny in-process fixtures, so none of them can drift from the API without the suite going red.