WordPiece¶
Selection constraints for WordPiece tokenizers (BERT and friends).
Bases: VocabBackend
Keeps a WordPiece vocabulary usable.
WordPiece greedily matches the longest prefix in the vocabulary and needs no merge
table, so tokens carry no dependencies: dropping "##ing" costs coverage but
never makes another vocabulary token unreachable.