Typst math: spelling and lexing of tokens
render spells a token list as Typst math; lex
reads it back. Typst has one
parenthesis: the printer uses the grouping token lpg for every (, and the
renderer writes no space between a function name and its ( (Typst calls a function
only then) and a space between an operand and a following ( (juxtaposition).
Spellings: numbers in the canonical spelling, scientific ones as
1.8 times 10^(-12) (read back as one literal, with any spacing:
1.8times 10^( -12 ) too; Esm.Document.Math.Surface
); names as italic("…"), formulas and
name subscripts as runs of adjacent upright("…") pieces each with an optional digit
subscript (upright("CH")_2upright("O"); a space between two pieces separates two
names); the scope dot as "." (a bare . after a Greek name is a symbol modifier in
Typst); call-form operators op("…"); text "…"; JSON values of an escape node as
# and the Typst value (Esm.Document.TypstDict
); a unit block
thin upright("m") thin upright("s")^(-1) after a number (Esm.Units.Typeset
).
Accepted on input and never written: a bare . as the scope dot.
Spelling
Spacing
def merges (t : Tok) (c : Char) : Bool
A space is needed between t and a following token starting with c.
def callEnd (t : Tok) : Bool
t ends with a character after which ( is a call parenthesis.
def sp (p : Option Tok) (t1 t2 : Tok) : Bool
No space before a call parenthesis, nor before the [ of an index (k[1]).
Lexing
def idRun : List Char → List Char × List Char
The maximal run of identifier characters.
def quotedArg (r : List Char) : Option (String × List Char)
("…") after a function name.
def pieces : Nat → List Char → List (String × String) × List Char
Adjacent upright("…") pieces, each with an optional digit subscript.
def sciExp (r : List Char) : Option (Int × List Char)
The scientific tail times 10^exponent after a mantissa (the exponent bare or in
parentheses), with any white space (none included) between its parts.
def uprightE : List Char → Option (Tok × List Char)
upright(e), Euler’s number written without quotes (an accepted variant, read as
upright("e"); the printer always quotes).
def lexIdent (cs : List Char) : Option (Tok × List Char)
One identifier-started token.
def lexOne (prev : Option Char) (cs : List Char) : Option (Tok × List Char)
One token at the front of the input; prev is the character just before it, if no
space came between (( right after a name, a digit, ) or " is a call
parenthesis).
def lexAux : Nat → Option Char → List Char → Option (List Tok)
The tokens of the input; prev is the last character of the previous token’s
spelling when no space came between (the canonical spelling: every token’s input
spellings end in a character of the same kind for the call-parenthesis test).