Corpus counting

trimbed.counting

Counting how often each token id occurs across one or more Hugging Face datasets.

CorpusCounts dataclass

CorpusCounts(
    counts: Counter[int] = Counter(),
    total_num_tokens: int = 0,
    num_documents: int = 0,
    samples: list[str] = list(),
)

Token-frequency statistics gathered over a corpus.

Attributes:

Name	Type	Description
counts	Counter[int]	Occurrences per token id, weighted per dataset, e.g. Counter({409: 91204, 13: 88317, ...}).
total_num_tokens	int	Sum of all counts.
num_documents	int	Number of examples read.
samples	list[str]	A reservoir sample of raw texts, reused to verify the trimmed tokenizer.

distinct_tokens property

distinct_tokens: int

Return how many distinct token ids the corpus used at least once.

coverage_of

coverage_of(token_ids: Iterable[int]) -> float

Return the fraction of corpus occurrences covered by a set of ids.

Parameters:

Name	Type	Description	Default
token_ids	Iterable[int]	The ids that survive the trim.	required

Returns:

Type	Description
float	A value in [0, 1], e.g. 0.9993. Token frequency is skewed enough that even an aggressive trim lands near 1, so the interesting digits are the last ones. That is why the report prints four decimal places rather than two.

View source on GitHub: src/trimbed/counting.py lines 46–60

ranked_ids

ranked_ids() -> list[int]

Return token ids sorted by descending frequency, ties broken by id to ensure determinism.

View source on GitHub: src/trimbed/counting.py lines 62–64

to_dict

to_dict() -> dict[str, Any]

Return a JSON-serialisable representation.

The counter's integer keys become strings, since JSON has no integer keys, e.g. {"counts": {"13": 88317, "409": 91204}, "total_num_tokens": ..., "num_documents": ..., "samples": [...]}. from_dict converts them back.

View source on GitHub: src/trimbed/counting.py lines 66–78

from_dict classmethod

from_dict(payload: dict[str, Any]) -> Self

Rebuild counts from to_dict output.

Parameters:

Name	Type	Description	Default
payload	dict[str, Any]	The decoded JSON mapping.	required

Returns:

Type	Description
Self	The reconstructed counts.

View source on GitHub: src/trimbed/counting.py lines 80–95

save

save(path: str | Path) -> None

Write the counts to a JSON file, creating parent directories as needed.

Parameters:

Name	Type	Description	Default
path	str | Path	Destination file.	required

View source on GitHub: src/trimbed/counting.py lines 97–106

load_from_file classmethod

load_from_file(path: str | Path) -> Self

Read counts back from a JSON file.

Parameters:

Name	Type	Description	Default
path	str | Path	Source file written by save.	required

Returns:

Type	Description
Self	The reconstructed counts.

View source on GitHub: src/trimbed/counting.py lines 108–118

CorpusCounter

CorpusCounter(
    tokenizer: PreTrainedTokenizerFast,
    config: CorpusConfig,
    *,
    seed: int = 0,
    sample_size: int = 256
)

Tokenizes configured datasets and accumulates per-id frequencies.

Build a counter.

Parameters:

Name	Type	Description	Default
tokenizer	PreTrainedTokenizerFast	A fast tokenizer used to encode the corpus.	required
config	CorpusConfig	A CorpusConfig which specifies the datasets to read and how.	required
seed	int	Seed for the verification-sample reservoir.	0
sample_size	int	How many raw texts to retain for later verification, e.g. 256, to verify the tokenizer and the model comparison then uses only the first handful.	256

View source on GitHub: src/trimbed/counting.py lines 124–139

count

count() -> CorpusCounts

Count token occurrences across every configured dataset.

Reads from CorpusConfig.counts_cache when that file exists, and writes to it otherwise, so an expensive pass can be reused across trimming runs.

Returns:

Type	Description
CorpusCounts	The accumulated statistics.

View source on GitHub: src/trimbed/counting.py lines 141–165