Version: dev

This is the documentation for the development version. For the latest release, see v0.12.1.

The Markdown container

Lean module Esm.Document.Source.Markdown, formal/Esm/Document/Source/Markdown.lean at 184aa13d35e5
Imports Esm.Document.Source.Reaction, Esm.Document.Source.YamlLines

An ESM document in Markdown is CommonMark with $…$ / $$…$$ math (LaTeX), fenced blocks, YAML front matter and directives:

---
esm: "2.1.0"
name: "Decay"
---

```esm:description
First-order radioactive decay.

::model[Decay]

:var[$N$]{default=100 units=“mol”}

$$ \frac{\mathrm{d} N}{\mathrm{d} t} = -\lambda N $$

The constructs ([`Esm.Document.Source.RItem`](/ast/dev/generated/document-formats/esm-document-source-items#Esm.Document.Source.RItem)), and where they may appear:

* front matter: the first line is `---`, then the YAML of the front-matter entries
  ([`Esm.Document.Yaml`](/ast/dev/generated/document-formats/esm-document-yaml)), then a line `---`;
* at the start of a line: a fenced block ```` ```esm:description ```` holding a
  description verbatim, or ```` ```esm:<key> ```` holding a path block's value in YAML;
  a display equation `$$`, its LaTeX on the following lines, then `$$` (optionally
  followed by an equation label `{#eq-x}`, which is discarded); a component directive
  `::model[Name]` or `::reaction-system[Name]` on a line of its own;
* at the start of a line: a reaction, `$$\ce{…}$$ {#id key=value …}` on one line, in
  mhchem ([`Esm.Document.Source.Reaction`](/ast/dev/generated/document-formats/esm-document-source-reaction)) with the rate as LaTeX math over the arrow,
  `->[{$k$}]` (braced, since mhchem ends the label at the first `]`), and the reaction's
  `id` and other fields in the label;
* at the start of a line: a template, a display whose math is
  `\operatorname{f}(a, b) \coloneqq body`, or a rule,
  `\text{rule } \operatorname{r}: \; lhs \Rightarrow rhs`; with other fields the display sits in a
  container `:::template{key=value …}` or `:::rule{key=value …}` closed by `:::`;
* at the start of a line: a subsystem included by reference,
  `::subsystem[Name]{ref="…" key=value …}`;
* at the start of a line: a plot directive `::plot{key=value …}` on one line, an analysis
  with exactly one plot: the analysis's fields, the plot as the attribute `plot`;
* at the start of a line: a container `:::initialization` or `:::constraints` on a line
  of its own, then display equations, then a line `:::`; each equation in it is an
  initialization equation or a constraint;
* anywhere in a line: a declaration `:var[$N$]{…}`, `:param[…]{…}` or `:species[…]{…}`,
  the name in LaTeX math and the attributes as `key=value` pairs, each value a YAML
  scalar or a single-quoted YAML flow collection (`shape='["x", "y"]'`).

Everything else is prose, which the parser skips: [`lex`](#Esm.Document.Source.Md.lex) reads a construct where one
starts and otherwise drops one character. A code span is skipped whole: a run of
backticks (not a ```` ```esm: ```` fence at the start of a line) runs to the next
occurrence of the same run, so `` `:var[x]` `` or a fenced code block holding an example
is prose; a run with no such occurrence after it is literal. So the parse depends only on
the sequence of constructs ([`lex_render`](#Esm.Document.Source.Md.lex_render)): inserting or removing prose and code spans,
and adding or removing equation labels, never changes it.

Restrictions this places on prose outside code spans ([`Inert`](#Esm.Document.Source.Md.Inert)): prose may not contain a
declaration opener (`:var[`, `:param[`, `:species[`) anywhere, nor start a line with
```` ```esm: ````, `$$`, `::model[`, `::reaction-system[`, `| decl | name |`,
`:::initialization` or `:::constraints`, nor start a line with a table row holding a `rate`
cell (a reaction table, which the lexer refuses). In particular display math (`$$`) at the
start of a line is always a model equation, so outside a model it rejects the file;
illustrative math in prose uses inline `$…$` or another delimiter.

The reader first drops a leading byte-order mark and, when the first line ends in CRLF,
reads every CRLF as LF (`decodeText`).

## Spellings

<div class="lean-decl" id="Esm.Document.Source.Md.fenceEsm">
<div class="lean-decl-head"><span class="lean-kind">def</span> <a class="lean-name" href="#Esm.Document.Source.Md.fenceEsm" title="Esm.Document.Source.Md.fenceEsm"><code>fenceEsm</code></a> <a class="lean-src" href="https://github.com/EarthSciML/EarthSciAST/blob/184aa13d35e5b30889e97811c776341973ff4e97/formal/Esm/Document/Source/Markdown.lean#L84">source</a></div>

```lean
def fenceEsm : List Char

```esm:

def fence : List Char

```

def dd : List Char

$$

def dModel : List Char

::model[

def dRs : List Char

::reaction-system[

def dashes : List Char

---

def descInfo : List Char

The info string of the description block.

def dPlot : List Char

::plot{, the start of a plot directive.

def dRxn : List Char

$$\ce{, the start of a reaction line.

def dInit : List Char

The containers of initialization equations and constraints, and their end.

def dRole : Role → List Char

The opening line of the container of a role.

def dkw : DKind → List Char

The directive keyword of a declaration.

def dSub : List Char

::subsystem[, a subsystem included by reference.

Path-block keys

def escKey (c : Char) : List Char

One character of a quoted key: the JSON escape, and the backtick as ``` (the info string of a backtick fence cannot hold a backtick).

def keyMd (k : String) : List Char

A path block’s key in the fence’s info string: bare when it is an identifier other than description (the description block), quoted otherwise.

def parseKeyMd (info : List Char) : Option String

Read a path block’s key from the info string (all of it).

Declarations

def nameLatex (n : String) : List Char

A declared name, as LaTeX math under the empty symbol table (so it reads the same whatever else the document declares).

Attribute values: YAML scalars and flow collections

An attribute value is a YAML scalar (100, "mol", true) or a flow collection, ["x", "y"], {a: 1, "b c": [2]}: elements and entries separated by , , keys bare when they are identifiers that YAML reads as strings, else quoted. A list or object (empty ones too) is written single-quoted, shape='["x", "y"]', since remark-directive refuses an unquoted value holding a bracket or brace; inside the quotes ' is &#39; and & is &amp;, the character references remark-directive decodes. The unquoted form is still read.

def pFlow : Nat → List Char → Option (Json × List Char)

A value: a flow collection at [ or {, else a scalar.

def pItemsF : Nat → List Char → Option (List Json × List Char)

The elements after the first, each after , , up to ].

def pEntryF : Nat → List Char → Option ((String × Json) × List Char)

One entry: key: value.

def pEntriesF : Nat → List Char → Option (List (String × Json) × List Char)

The entries after the first, each after , , up to }.

def isColl : Json → Bool

A value whose YAML starts with a bracket or brace (a list or an object, empty ones included): micromark-extension-directive, and so remark-directive, refuses such a value unquoted.

def sqEsc (cs : List Char) : List Char

The text of a single-quoted attribute value: ' as &#39; and & as &amp;, the character references micromark-extension-directive decodes inside a quoted value.

def sqUnesc : List Char → List Char

Undo sqEsc .

def attrVal (v : Json) : List Char

An attribute value: a scalar as its YAML, a list or object as its YAML flow collection single-quoted (shape='["x", "y"]').

def attrMd (a : String × Json) : List Char

One attribute: key=value.

def attrsMd : List (String × Json) → List Char

The attributes, separated by spaces.

def declMd (d : Decl) : List Char

:var[$N$]{default=100 units="mol"}.

def pQuoted (u : List Char) : Option (Json × List Char)

A single-quoted value after its opening quote: the YAML inside, then the text after the closing quote.

def pAttr (s : List Char) : Option ((String × Json) × List Char)

One attribute.

def pAttrsMore : Nat → List Char → Option (List (String × Json) × List Char)

The attributes after the first, each after a space, up to the closing brace.

def pAttrs (f : Nat) : List Char → Option (List (String × Json) × List Char)

The attributes after the opening brace, up to the closing one.

def declBody (f : Nat) (k : DKind) : List Char → Option (RItem × List Char)

A declaration after its [.

Block constructs

def fenceItem (info body : List Char) : Option RItem

A fenced block’s construct, from its info string and body.

def labelOK (sl : List Char) : Bool

An equation label after the closing $$: nothing, or {#id}, with spaces around.

def fenceAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A fenced block after ```esm:.

Templates and rules

A template is a display whose math starts with its header, \operatorname{f}(a, b) \coloneqq, then its body: $$, \operatorname{f}(a, b) \coloneqq body on one line, $$. A rule is \text{rule } \operatorname{r}: \; lhs \Rightarrow rhs. The header (the name, the parameters as names under the empty symbol table) and the rule’s split at its arrow are read here, by the lexer; only the body and the left-hand side are math. With other fields the display is wrapped in a container directive whose attributes hold them, :::template{description="…"} or :::rule{params='["u"]' priority=1}, closed by a line ::: (attributes after the closing $$ would keep remark-math from closing the display).

def paramsTxt : List String → List Char

The parameters of a template header, separated by , .

def tmRaw (t : Tm String) : List Char

The math text of a template or rule display (one line).

def pName (s : List Char) : Option (String × List Char)

One parameter: a name, up to the next , or ).

def pParams : Nat → List Char → Option (List String × List Char)

The parameters after (, up to ) \coloneqq .

def splitAt (sep : List Char) : List Char → Option (List Char × List Char)

Split at the first occurrence of sep.

def tmLex (raw : List Char) : Option (String × List String × Option (List Char) × List Char)

Read the math text of a display as a template or rule: its name, its parameters, its left-hand side (a rule’s) and its body, as text.

def mathAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A display after its opening $$: a template or rule when its math reads as one, otherwise an equation (optionally labelled).

def tmContAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A template or rule with other fields, after :::template{ or :::rule{: the attributes, the display, and the closing ::: line.

def dirAt (K : Kind) (u : List Char) : Option (RItem × List Char)

A component directive after its [.

Reactions

def latexR : RSurf

The math surface of reaction sides: LaTeX under the empty symbol table.

def arrowL : List Char

The arrow with the rate over it, around the rate’s text (which ends at its $).

def rxnEnd : List Char

After the products: the end of the math and the start of the label.

def rxnBody (x : Rxn String) : List Char

The text of a reaction line after $$\ce{.

def rxnAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A reaction line after its $$\ce{.

def subAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A subsystem directive after its ::subsystem[.

def plotAt (f : Nat) (u : List Char) : Option (RItem × List Char)

A plot directive after its ::plot{.

Containers of initialization equations and constraints

def contBody (ro : Role) : Nat → List Char → Option (List RItem × List Char)

The displays of a container, up to its closing ::: line; blank lines between them are skipped.

def contOpen (s : List Char) : Bool

A container starts here.

def contAt (f : Nat) (s : List Char) : Option (List RItem × List Char)

A container, at its opening line.

The lexer

def blockOpen (s : List Char) : Bool

A block construct starts here (at the start of a line).

def declOpen (s : List Char) : Bool

A declaration starts here.

def blockAt (f : Nat) (s : List Char) : Option (RItem × List Char)

Read the block construct that starts here.

def declAt (f : Nat) (s : List Char) : Option (RItem × List Char)

Read the declaration that starts here.

The declarations table (an accepted input variant)

A per-model declarations table is read as the declarations it lists; the printer never writes it (it writes :var directives):

| decl | name | default | units |
| --- | --- | --- | --- |
| var | $N$ | 100 | "mol" |
| param | $\lambda$ | 0.1 | "1/s" |

The first two columns are the directive and the name; every other column header is an attribute key, and a cell is a YAML scalar or empty (the attribute is absent).

def tableOpen : List Char

| decl | name |, the start of a declarations table.

def trimCell (s : String) : String

A cell without its surrounding spaces.

def headerCells (s : List Char) : List String

The header cells of a table line: the line split at |, each cell trimmed.

def rateTable (s : List Char) : Bool

The start of a reaction table with a rate column: a line starting with | that has a cell rate. It is not a reaction notation (Notation.variants, rejected), and the lexer refuses it rather than skipping it as prose, which would drop its reactions.

def pCols : Nat → List Char → Option (List String × List Char)

The attribute columns of the header, up to its newline.

def pCell : List Char → Option (Option Json × List Char)

One cell: value |, or | when the attribute is absent.

def pCells : List String → List Char → Option (List (String × Json) × List Char)

The cells of a row, one per attribute column.

def kindOf (w : List Char) : Option DKind

The directive keyword of a row.

def pRow (cols : List String) : List Char → Option (Decl × List Char)

One row: | var | $N$ | and the cells, then a newline.

def pRows (cols : List String) : Nat → List Char → Option (List Decl × List Char)

The rows, as long as lines start with | .

def sepOK (l : List Char) : Bool

The separator row: only |, -, : and spaces.

def tableAt (f : Nat) (s : List Char) : Option (List RItem × List Char)

A declarations table, at its | decl | name |.

Code spans

A run of backticks in prose opens a code span (or, at the start of a line, a fenced code block), which runs to the next occurrence of the same run; nothing inside it is a construct. A run with no such occurrence after it is literal.

def afterRun (n : Nat) : List Char → Option (List Char)

The text after the first occurrence of n backticks.

def skipCode (s : List Char) : List Char

The text after the code span that starts here (at a backtick).

def lex : Nat → Bool → List Char → Option (List RItem)

The Markdown lexer: the constructs of a text, skipping prose one character at a time and code spans whole. bol: the text starts at the beginning of a line. A malformed construct is an error (none).

Reading back the pieces

theorem pInline_gen proved source
theorem pInline_gen (v : Json) (hi : Yaml.isInl v = true) (hv : v.WF) (rest : List Char)
    (hr : Number.NumStop rest) : Yaml.pInline (Yaml.inline v ++ rest) = some (v, rest)

A YAML scalar read back whatever follows it (a number needs a non-number character after it).

def FStop (rest : List Char) : Prop

What may follow an attribute value: ,, ], }, a space or |.

theorem pFlow_flow' proved source
theorem pFlow_flow' (v : Json) (hv : v.WF) (rest : List Char) (hr : FStop rest) :
    pFlow (2 * (flow v ++ rest).length + 2) (flow v ++ rest) = some (v, rest)

A value printed as a scalar or flow collection reads back.

def attrOK (a : String × Json) : Bool

An attribute the Markdown attribute syntax can hold: an identifier key and a well-formed value.

def declOK (d : Decl) : Bool

A declaration the Markdown syntax can hold.

def mhchemOK (x : Rxn String) : Bool

What mhchem reads as written: no escaped brace \{ or \} anywhere inside \ce{…} (mhchem matches braces by character and fails on them).

def rxnOK (x : Rxn String) : Bool

A reaction the Markdown syntax can hold: sides the side syntax holds, a rate on one line without $, an id label, and attributes the attribute syntax holds.

Keys

The printer

def subMd (d : Decl) : List Char

::subsystem[Name]{ref="…"}, on a line of its own.

def subOK (d : Decl) : Bool

A subsystem directive the syntax can hold.

def itemMd : RItem → List Char

The text of a construct. A description is written verbatim in its fence; a path block’s value in YAML; a declaration inline (no newline of its own).

def eqnLabeled (raw : String) (label : List Char) : List Char

A display equation with an equation label after its closing $$ ( {#eq-x}).

inductive Seg source
inductive Seg where
  | item (it : RItem)
  | labeled (raw : String) (label : List Char)
  | prose (p : List Char)
  | code (body : List Char)

A piece of a Markdown document: a construct, a labelled equation, prose, or a code span (`…`, whatever its content).

def render (segs : List Seg) : List Char

The text of a document body.

def segItems : List Seg → List RItem

The constructs of a document body: prose dropped, labels dropped.

def isBlock : RItem → Bool

A construct that must start a line (all but declarations).

def itemOK : RItem → Bool

What each construct’s content must satisfy for its text to read back.

def Inert : Bool → List Char → List Char → Prop

Prose that the lexer skips: at no character does a construct start (a block construct only matters at the start of a line), and it holds no backtick (code spans are their own pieces, Seg.code). t is the text after the prose.

def Valid : Bool → List Seg → Prop

A document body whose constructs read back: each construct’s content is spellable, block constructs start a line, labels are labels, and prose is inert.

Lexing, construct by construct

theorem lex_code proved source
theorem lex_code (body t : List Char) (hb : body ≠ []) (hc : '`' ∉ body) (bol : Bool) (g : Nat) :
    lex (g + 1) bol ('`' :: body ++ '`' :: t) = lex g false t

A code span is skipped whole, whatever it holds.

Reactions

Subsystems

Plots

Templates and rules

Containers

def dispTxt (raw : String) : List Char

A display followed by a blank line.

theorem contBody_many proved source
theorem contBody_many (r : Role) (t : List Char) : ∀ (raws : List String),
    (∀ raw ∈ raws, itemOK (.eqn raw) = true) → ∀ g,
    (raws.flatMap dispTxt).length + t.length + 4 < g →
    contBody r g (raws.flatMap dispTxt ++ dEnd ++ '\n' :: t) = some (raws.map (.eqx r), t)

A container holding several displays reads as their equations, in order: each display followed by a blank line, then the closing ::: line (the printer writes one display per container).

theorem lex_render proved source
theorem lex_render : ∀ (segs : List Seg) (bol : Bool), Valid bol segs →
    ∀ f, (render segs).length < f → lex f bol (render segs) = some (segItems segs)

The Markdown lexer reads back every valid document body: exactly its constructs, whatever prose surrounds them and whatever labels the equations carry.

The declarations table reads as its declarations

def colText (k : String) : List Char

A header column.

def cellText : Option Json → List Char

A cell.

def rowText (cols : List String) (d : Decl) : List Char

A row of the table.

def tableText (cols : List String) (ds : List Decl) : List Char

A declarations table over the attribute columns cols.

def TableOK (cols : List String) (d : Decl) : Prop

A declaration a table over cols holds: its name and attributes spellable, and its attributes exactly the columns it has a value in, in column order.

theorem lex_table proved source
theorem lex_table (cols : List String) (hc : ∀ k ∈ cols, isIdent k.toList = true)
    (ds : List Decl) (hd : ∀ d ∈ ds, TableOK cols d) (t : List Char)
    (ht : pre ['|', ' '] t = false) (g : Nat) (hg : (tableText cols ds).length + t.length < g + 1) :
    lex (g + 1) true (tableText cols ds ++ t) = (lex g true t).map (ds.map RItem.decl ++ ·)

A declarations table reads as the declarations it lists (and as the :var/:param/:species directives of the same declarations: the same constructs).

This module also proves 86 lemmas without doc comments; they are in the source.