WordLevel¶
Selection constraints for WordLevel tokenizers.
Bases: VocabBackend
Keeps a flat word-level vocabulary usable.
The simplest case: no merges, no subwords, so no dependencies. Anything dropped becomes unk, which makes the unk token the only structural requirement.