Id remapping

trimbed.remap

Contiguous, order-preserving remapping of token ids.

IdRemap dataclass

IdRemap(
    new_to_old: tuple[int, ...], old_to_new: dict[int, int]
)

A mapping from a kept subset of old token ids onto 0..n-1.

Old ids are sorted ascending before renumbering, so the surviving tokens keep their relative order. That is what makes new_to_old usable as-is when pruning the embeddings: it is the gather index that picks out the rows to keep.

E.g. keeping old ids {0, 5, 9} out of ten gives new_to_old == (0, 5, 9) and old_to_new == {0: 0, 5: 1, 9: 2}.

from_kept classmethod

from_kept(kept_ids: Iterable[int]) -> Self

Build a remap from the set of old ids that survive.

Parameters:

Name	Type	Description	Default
kept_ids	Iterable[int]	Old token ids to keep, e.g. {9, 0, 5}. Duplicates are collapsed and the order is irrelevant.	required

Returns:

Type	Description
Self	A remap numbering the sorted kept ids from zero, so that example yields new_to_old == (0, 5, 9).

Raises:

Type	Description
ValueError	If no ids were kept or an id is negative.

View source on GitHub: src/trimbed/remap.py lines 25–45

from_vocabularies classmethod

from_vocabularies(
    old: dict[str, int], new: dict[str, int]
) -> Self

Build a mapping between two vocabularies, e.g. the original and the trimmed one.

Only the tokens present in both are kept, numbered contiguously from zero.

Parameters:

Name	Type	Description	Default
old	dict[str, int]	Token -> id map before trimming, e.g. codefuse-ai/F2LLM-v2-160M's 151,669 entries.	required
new	dict[str, int]	Token -> id map after trimming, numbered contiguously from zero, e.g. the 32,000 entries skeletoken left behind.	required

Returns:

Type	Description
Self	A remap covering every token present in both vocabularies.

Raises:

Type	Description
ValueError	If the trimmed vocabulary introduces tokens the original lacked, or is not numbered contiguously from zero.

View source on GitHub: src/trimbed/remap.py lines 47–76

__len__

__len__() -> int

Return the number of surviving tokens.

View source on GitHub: src/trimbed/remap.py lines 78–80

__contains__

__contains__(old_id: int) -> bool

Return whether old_id survives the trim.

View source on GitHub: src/trimbed/remap.py lines 82–84

__iter__

__iter__() -> Iterator[int]

Iterate over the kept old ids in ascending order.

View source on GitHub: src/trimbed/remap.py lines 86–88

to_new

to_new(old_id: int) -> int

Map an old id to its new id.

Parameters:

Name	Type	Description	Default
old_id	int	Id in the original vocabulary.	required

Returns:

Type	Description
int	The corresponding id in the trimmed vocabulary.

Raises:

Type	Description
KeyError	If old_id did not survive the trim.

View source on GitHub: src/trimbed/remap.py lines 90–102

to_old

to_old(new_id: int) -> int

Map a new id back to the id it had in the original vocabulary.

Parameters:

Name	Type	Description	Default
new_id	int	Id in the trimmed vocabulary.	required

Returns:

Type	Description
int	The original id.

View source on GitHub: src/trimbed/remap.py lines 104–113

map_sequence

map_sequence(old_ids: Iterable[int]) -> list[int] | None

Map a whole sequence or report that it cannot be mapped.

Parameters:

Name	Type	Description	Default
old_ids	Iterable[int]	Ids produced by the original tokenizer, e.g. [9707, 1879] for "Hello world" under codefuse-ai/F2LLM-v2-160M.	required

Returns:

Type	Description
list[int] | None	The remapped ids, or None if any id was dropped by the trim. None is the signal verify_model uses to skip a text rather than compare two sequences that no longer correspond.

View source on GitHub: src/trimbed/remap.py lines 115–133