Version: v0.14.0

The parser: tokens to expressions

Lean module Esm.Document.Math.Parse, formal/Esm/Document/Math/Parse.lean at ffd5aee2f868
Imports Esm.Document.Math.Print

Recursive descent with one function per precedence level (Esm.Document.Math.Print lists the levels), each taking fuel. The +, *, and and or chains are read into one n-ary node; binary -, the comparisons and the Typst fraction / group to the left; unary - takes a whole product (-a b is -(a b), spec §4.1.1); ^ takes a postfix-level base and one exponent.

Accepted variants are read here too (the printer never writes them), each with a lemma in Esm.Document.Math.Variants .

Small pieces

def close (o : String) : List Json → Json

Close an n-ary chain: one operand is itself, more make a node.

def cmpOf : Kw → Option String

The comparison a token spells.

def fnOf : Kw → Option String

The unary function a keyword spells.

def atomStart : Tok → Bool

Tokens that can start an operand of a product: juxtaposition continues a product when the next token is one of these.

def isLp : Tok → Bool

An opening parenthesis: ( or \left(; the parser reads them alike.

def isRp : Tok → Bool

The closing token of a group: ) or \right).

Names

def decodeFormula (raw : List Char) : List Char

The name a formula token spells, if it is formula-shaped.

def subOrPlain (sf : Sf) (b : List Char) (r : List Tok) : List Char × List Tok

A single letter or Greek base, then an optional upright name subscript.

def pSeg (sf : Sf) : List Tok → Option (List Char × List Tok)

One name segment.

def pNameLoop (sf : Sf) : Nat → List Char → List Tok → Option (List Char × List Tok)

A name: segments joined by the scope dot.

JSON inside LaTeX escape nodes

Negative literals after an operand

def negTok : Tok → Bool

A negative literal token.

def negate : JNum → JNum

The number with the opposite sign.

def negTail : List Tok → Option (List Tok)

A negative literal right after an operand (k-1, k -1) is a binary minus and a positive number: the tokens with the sign moved off the number, when the number without its sign has a canonical form (the negation of -2^63 has none, so that spelling is refused after an operand).

Expressions

def mkNodeP (o : String) (as : List Json) (fs : List (String × Json)) : Json

Rebuild a node with fields: op and args go back to their sorted places.

inductive CallArg source
inductive CallArg where
  | pos (v : Json)
  | named (k : String) (v : Json)

What pCallArgs read: positional arguments, or named ones.

def sortKV : List (String × Json) → List (String × Json)

Sort bindings by key (insertion).

def unitsOfRaw (raw : String) : Option (List Esm.Units.Factor)

The unit a LaTeX \mathrm{…} after a number spells.

def qty (n : JNum) (l : List Esm.Units.Factor) : Json

A quantity node.

def dNode (x : Json) (w : String) : Json

A derivative node.

def tNode (n : String) (bs : List (String × Json)) : Json

A template application node.

def constNode (v : Json) : Option String → Json

A constant node: the value, and the units when there are some.

def pCells (sep : Kw) : List Tok → Option (List JNum × List Tok)

Numbers separated by sep, up to the first other token.

def pGrid (csep rsep close : Kw) : Nat → List Tok → Option (List (List JNum) × List Tok)

Rows of numbers: cells separated by csep, rows by rsep, closed by close .

def gridJson (rows : List (List JNum)) : Json

A LaTeX bmatrix: one cell per row is a column (rank 1).

def constTail (v : Json) (r : List Tok) : Option (Json × List Tok)

The unit after an array, if any.

def icTailP (sf : Sf) : List Tok → Option (List Tok)

The ic tail (t_{0}) / (t_0).

def jsonStart : List Tok → Bool

A LaTeX field value that is JSON starts with {, [, a string or a word.

def keyOf (c : Ctx) (ts : List Tok) : Option (String × List Tok)

A field key: LaTeX \mathrm{key} =, Typst upright("key") =.

def dCallNode (w : String) (xs : List Json) : Json

A spatial derivative with trailing operands.

def eulerHead (c : Ctx) : List Tok → Bool

The input starts with Euler’s number raised to a power: an upright e, or an italic e when no variable e is declared.

def pNamePrim (c : Ctx) (ts : List Tok) : Option (Json × List Tok)

A name, or an initial condition N(t_{0}).

def pWord (c : Ctx) (w : String) (r : List Tok) : Option (Json × List Tok)

A bare run of letters (RT): the declared name when only it is declared, the product of its letters when only they are, refused otherwise (ambiguous or unknown).

def dName (c : Ctx) (k : Kw) : List Tok → Option (String × List Tok)

d then a name, up to a closing token k: dN} (a run of letters that is not itself a declared name) or d N}.

def pItalicD (c : Ctx) (r : List Tok) : Option (Json × List Tok)

The italic-d derivative variant, when no variable d is declared and neither dN nor dt written as one word is a declared name (a declared whole word wins, as in pWord : \frac{dx}{dt} with dx and dt declared is their quotient): LaTeX \frac{dN}{dt} after \frac{, Typst (d N)/(d t) after the first (.

def isKw (k : Kw) : Tok → Bool

The token is the keyword k.

inductive Row source
inductive Row where
  | rng (k : String) (v : Json)
  | filt (x : Json)

A row of a reduction’s subscript, or an output entry: a range, or an expression (the filter, or a bare output entry).

def pRedHead (sf : Sf) : List Tok → Option (RedK × Bool × List Tok)

The operator of a reduction and whether it is marked (args listed): LaTeX \sum\limits, Typst limits(sum).

def pNames (sf : Sf) : Nat → List Tok → Option (List String × List Tok)

Names separated by commas.

def rowParts : List Row → Option (List (String × Json) × Option Json)

The ranges and the filter of a list of rows: ranges, then at most one filter, last.

def entryParts : List Row → Option (List Json × List (String × Json))

The output entries of a list of entry rows, with their ranges.

def withOut (F : FaqForm) (outs : List Json) (ors : List (String × Json)) : FaqForm

The node an output form reads: the reduction’s parts with the output entries.

def withSup (F : FaqForm) (st : List String) : FaqForm

The superscript set read after the output entries (the reduction inside the bracket carries none, bracketSetOK ).

def bracketSetOK (F : FaqForm) (st : List String) : Bool

The sets of an output form as the printer writes them: none on the reduction inside the bracket (the printer writes the set on the bracket), and none at all when the operator is marked (args lists the body’s arrays).

def pSup (sf : Sf) (r : List Tok) : Option (List String × List Tok)

The superscript set of a reduction or of an output form, if there is one: LaTeX ^{\{f, g\}}, Typst ^({f, g}).

def mkTail (sf : Sf) (rows : List (Json × List (Json × Json))) (r : List Tok) : Option (Json × List Tok)

The makearray node of a table read back, with its subscript of output indices if one follows.

def pOr (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 1: an or chain.

def pAnd (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 2: an and chain.

def pCmp (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 3: one comparison (comparisons do not chain: a \le k \le b is a range row of a reduction, never an expression).

def pAdd (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 4: + chains and binary -, including a negative literal right after an operand (negTail ). x :: acc is the pending + chain.

def pMul (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 5: unary minus, and * chains (juxtaposition, \cdot, \times; LaTeX inline / groups to the left). A negative literal never continues a product by juxtaposition (k -1 is k - 1); after \cdot it is a factor (k \cdot -1).

def pNot (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 6: not.

def pFrac (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 7: the Typst fraction, grouping to the left.

def pPow (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 8: a power.

def pExpo (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

An exponent: LaTeX {…}; Typst (…) or one simple token.

def pPost (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

Level 9: postfix index, _{…} or […].

def pList (c : Ctx) : Nat → Nat → List Tok → Option (List Json × List Tok)

A comma-separated list of expressions at level L (1 or 4), possibly empty when the next token closes it.

def pLv (c : Ctx) : Nat → Nat → List Tok → Option (Json × List Tok)

Parse at level 1 or level 4.

def pCallArgs (c : Ctx) : Nat → List Tok → Option (List CallArg × List Tok)

Call arguments at level 4: each positional, or name = value.

def pFields (c : Ctx) : Nat → List Tok → Option (List (String × Json) × List Tok)

Open-operator fields after ;: key = value, the value as math or as JSON.

def pVal (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

A field value: JSON (LaTeX JSON tokens, Typst #value) or an expression.

def pOpn (c : Ctx) : Nat → String → List Tok → Option (Json × List Tok)

\operatorname{n}(…): a call-form core operator, a deprecated one (rewritten), a template (by the symbol table, positional or named), or an open operator.

def pDCall (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

\operatorname{D}_{x}(u, a, b) after the _.

def pPrim (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

A primary.

def pEulerExp (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

\exp written as a power of e.

def pRange (c : Ctx) : Nat → List Tok → Option (Row × List Tok)

A range row: k \in S, k \in S(i, j), a \le k \le b.

def pRow (c : Ctx) : Nat → (Tok → Bool) → List Tok → Option (Row × List Tok)

A row ending at a token isEnd accepts: an expression read whole first (it never reads a range row whole: a range row has \in or a second \le), else a range.

def pRows (c : Ctx) : Nat → Kw → Kw → List Tok → Option (List Row × List Tok)

Rows separated by sep, up to close (consumed).

def pRedBody (c : Ctx) : Nat → RedK → Bool → List Tok → Option (FaqForm × List Tok)

A reduction after its operator: the subscript rows (LaTeX: one row, a lone dense range k=a with ^{b}, or \substack; Typst: rows in _( … )), then the body.

def pRed (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

A rank-0 reduction, or a reduction wrapped with its output entries.

def pFn1 (c : Ctx) : Nat → String → List Tok → Option (Json × List Tok)

f(x) after the opening parenthesis.

def pFracL (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

After \frac{: a time or spatial derivative, the italic-d variant, or a quotient.

def pDT (c : Ctx) : Nat → Kw → List Tok → Option (Json × List Tok)

(dif N)/(dif t) and (partial u)/(partial x) after the first (dif / (partial.

def pDApply (c : Ctx) : Nat → String → List Tok → Option (Json × List Tok)

A derivative operator applied to a parenthesized operand.

def pCases (c : Ctx) : Nat → List Tok → Option (Json × List Tok)

cases rows: value & if cond then a row break, ending with value & otherwise; a chain re-nests to the right.

def pPair (c : Ctx) : Nat → List Tok → Option ((Json × Json) × List Tok)

A bound pair after its [: a, b].

def pRegTail (c : Ctx) : Nat → List Tok → Option (List (Json × Json) × List Tok)

The further bound pairs of a region, each after \times [.

def pMkRows (c : Ctx) : Nat → Kw → Kw → Kw → List Tok → Option (List (Json × List (Json × Json)) × List Tok)

The rows of a makearray table: a value, the cell separator csep, a region; rows separated by rsep, up to close (consumed).

def pEscArgs (c : Ctx) : Nat → List Tok → Option (List Json × List Tok)

Typst escape arguments: , a repeated, then ).