Version: v0.14.0

The Typst lexer lemma

Lean module Esm.Document.Math.LexTypst, formal/Esm/Document/Math/LexTypst.lean at ffd5aee2f868
Imports Esm.Document.Math.Typst, Esm.Document.Math.LexLatex

Every list of Typst tokens (tokOK ) in which each call parenthesis follows a token after which ( is a call parenthesis (CallOK) reads back from its rendering. The proof goes token by token, as for LaTeX; the one Typst-specific step is the parenthesis: the lexer tells a call ( from a grouping ( by the character before it, and the renderer writes a space before a grouping ( exactly when that character would make it a call.

Tokens

def kwOK : Kw → Bool

Typst keywords: the ones with a Typst spelling, except times, which the printer writes only before a bracket (Adj : after a number, times 10^… would be its scientific tail).

def tokOK : Tok → Prop

The Typst tokens: what the Typst printer writes.

def Sep (t : Tok) (X : List Char) : Prop

What may follow the spelling of t: nothing that extends it.

Characters

Dispatch

theorem lexOne_word proved source
theorem lexOne_word (prev : Option Char) (s : String) (hs : s = "delim" ∨ s = "augment") (X : List Char)
    (hX : ∀ c ∈ X.head?, isIdC c = false) : lexOne prev (s.toList ++ X) = some (.word s, X)

An identifier the lexer reads as one word: the named arguments of the makearray table head.

theorem lexOne_times proved source
theorem lexOne_times (prev : Option Char) (X : List Char) (hX : ∀ c ∈ X.head?, merges (.kw .times) c = false) :
    lexOne prev (tokChars (.kw .times) ++ X) = some (.kw .times, X)

times: read as the keyword (the printer writes it before a bracket).

Numbers

theorem sciExp_head proved source
theorem sciExp_head (c : Char) (r : List Char) (h1 : isWsT c = false) (h2 : c ≠ 't') :
    sciExp (c :: r) = none

No scientific tail where times does not come first.

theorem sciExp_tail proved source
theorem sciExp_tail (e : Int) (X : List Char) (hX : Number.NumStop X) :
    sciExp (timesTen ++ ((if e < 0 then '(' :: Number.printInt e ++ [')'] else Number.printInt e) ++ X)) =
      some (e, X)

The printed scientific tail reads back as its exponent.

Identifier tokens

Upright pieces

One token

Heads

What may follow a number or a unit block

Token lists

def Adj : List Tok → Prop

Neighbouring tokens: a call parenthesis follows a token after which ( is a call parenthesis, and a unit block never follows a unit block.

theorem noGlue_times proved source
theorem noGlue_times (S W : List Char) (hS : S = [] ∨ S = [' ']) :
    sciExp (tokChars (.kw .times) ++ (S ++ '[' :: W)) = none ∧
      (' ' :: (tokChars (.kw .times) ++ (S ++ '[' :: W))).dropPrefix? Esm.Units.thinSep = none

times before a bracket: no scientific tail, no unit separator.

def HC (t : Tok) (S : List Char) (prev : Option Char) : Prop

What the lexer needs to tell the first token’s parenthesis right.

theorem lex_render proved source
theorem lex_render (ts : List Tok) (h : ∀ t ∈ ts, tokOK t ∨ t = .kw .times) (ha : Adj ts)
    (hfirst : ∀ r, ts ≠ .kw .lp :: r) : lex (render ts) = some ts

The Typst lexer lemma: a list of Typst tokens (and times before a bracket) whose call parentheses each follow a name-like token, and whose unit blocks are not adjacent, reads back from its rendering.

This module also proves 39 lemmas without doc comments; they are in the source.