Byte-level codec¶

trimbed.bytelevel ¶

The GPT-2-style byte-to-unicode mapping used by ByteLevel tokenizers.

bytes_to_unicode() -> dict[int, str]

Return the reversible byte -> printable-character map used by ByteLevel.

Borrowed mostly from GPT-2's tokenizer, also see the transformers implementation.

Returns:

Type	Description
`dict[int, str]`	A mapping of all 256 byte values to distinct printable characters, e.g. `32 -> "Ġ"` (space), `10 -> "Ċ"` (newline), `0 -> "Ā"`, and `65 -> "A"` for the bytes that are already printable.

unicode_to_bytes() -> dict[str, int]

Return the inverse of bytes_to_unicode.

E.g. "Ġ" -> 32 and "Ċ" -> 10.

byte_level_alphabet() -> frozenset[str]

Return the 256 characters a ByteLevel pre-tokenizer can produce.

Every one of them must stay in the vocabulary of a byte-level tokenizer, otherwise some byte sequences become unencodable.

decode_byte_level(token: str) -> str | None

Turn a byte-level token back into the text it represents.

Parameters:

Name	Type	Description	Default
`token`	`str`	A token as stored in a ByteLevel vocabulary, e.g. `"Ġde"`, which will produce `" de"`. Multi-byte characters arrive as several alphabet characters, so `"Ã©"` produces `"é"`.	required

Returns:

Type	Description
`str \| None`	The decoded text, or `None` if the token is a partial UTF-8 sequence or contains characters outside the byte-level alphabet.