Lexer¶

bcql_py.parser.lexer ¶

Tokenize BCQL source text into a typed token stream.

BCQLLexer(source: str)

A hand-written lexer for BCQL query strings, converting a given string into a sequence of tokens.

Attributes:

Name	Type	Description
`source`	`str`	The original BCQL query string (read-only property).
`pos`	`int`	Current character position in source during tokenization (read-only property).
`tokens`	`tuple[Token, ...]`	Immutable tuple of tokens produced by tokenization (read-only property).

Initialize the empty lexer state for source.

Parameters:

Name	Type	Description	Default
`source`	`str`	BCQL query text to tokenize.	required

source: str

Return the original BCQL source string.

pos: int

Return the current character position in source.

tokens: tuple[Token, ...]

Return the tokenized source as an immutable token tuple.

tokenize() -> tuple[Token, ...]

We assume the base case where the token can be a sequence of [] tokens or "" or '' or a sequence of alphanumeric characters and underscores.