The Markdown container
An ESM document in Markdown is CommonMark with $…$ / $$…$$ math (LaTeX), fenced
blocks, YAML front matter and directives:
---
esm: "2.1.0"
name: "Decay"
---
```esm:description
First-order radioactive decay.
::model[Decay]
:var[$N$]{default=100 units=“mol”}
$$ \frac{\mathrm{d} N}{\mathrm{d} t} = -\lambda N $$
The constructs ([`Esm.Document.Source.RItem`](/ast/dev/generated/document-formats/esm-document-source-items#Esm.Document.Source.RItem)), and where they may appear:
* front matter: the first line is `---`, then the YAML of the front-matter entries
([`Esm.Document.Yaml`](/ast/dev/generated/document-formats/esm-document-yaml)), then a line `---`;
* at the start of a line: a fenced block ```` ```esm:description ```` holding a
description verbatim, or ```` ```esm:<key> ```` holding a path block's value in YAML;
a display equation `$$`, its LaTeX on the following lines, then `$$` (optionally
followed by an equation label `{#eq-x}`, which is discarded); a component directive
`::model[Name]` or `::reaction-system[Name]` on a line of its own;
* at the start of a line: a reaction, `$$\ce{…}$$ {#id key=value …}` on one line, in
mhchem ([`Esm.Document.Source.Reaction`](/ast/dev/generated/document-formats/esm-document-source-reaction)) with the rate as LaTeX math over the arrow,
`->[{$k$}]` (braced, since mhchem ends the label at the first `]`), and the reaction's
`id` and other fields in the label;
* at the start of a line: a template, a display whose math is
`\operatorname{f}(a, b) \coloneqq body`, or a rule,
`\text{rule } \operatorname{r}: \; lhs \Rightarrow rhs`; with other fields the display sits in a
container `:::template{key=value …}` or `:::rule{key=value …}` closed by `:::`;
* at the start of a line: a subsystem included by reference,
`::subsystem[Name]{ref="…" key=value …}`;
* at the start of a line: a plot directive `::plot{key=value …}` on one line, an analysis
with exactly one plot: the analysis's fields, the plot as the attribute `plot`;
* at the start of a line: a container `:::initialization` or `:::constraints` on a line
of its own, then display equations, then a line `:::`; each equation in it is an
initialization equation or a constraint;
* anywhere in a line: a declaration `:var[$N$]{…}`, `:param[…]{…}` or `:species[…]{…}`,
the name in LaTeX math and the attributes as `key=value` pairs, each value a YAML
scalar or a single-quoted YAML flow collection (`shape='["x", "y"]'`).
Everything else is prose, which the parser skips: [`lex`](#Esm.Document.Source.Md.lex) reads a construct where one
starts and otherwise drops one character. A code span is skipped whole: a run of
backticks (not a ```` ```esm: ```` fence at the start of a line) runs to the next
occurrence of the same run, so `` `:var[x]` `` or a fenced code block holding an example
is prose; a run with no such occurrence after it is literal. So the parse depends only on
the sequence of constructs ([`lex_render`](#Esm.Document.Source.Md.lex_render)): inserting or removing prose and code spans,
and adding or removing equation labels, never changes it.
Restrictions this places on prose outside code spans ([`Inert`](#Esm.Document.Source.Md.Inert)): prose may not contain a
declaration opener (`:var[`, `:param[`, `:species[`) anywhere, nor start a line with
```` ```esm: ````, `$$`, `::model[`, `::reaction-system[`, `| decl | name |`,
`:::initialization` or `:::constraints`, nor start a line with a table row holding a `rate`
cell (a reaction table, which the lexer refuses). In particular display math (`$$`) at the
start of a line is always a model equation, so outside a model it rejects the file;
illustrative math in prose uses inline `$…$` or another delimiter.
The reader first drops a leading byte-order mark and, when the first line ends in CRLF,
reads every CRLF as LF (`decodeText`).
## Spellings
<div class="lean-decl" id="Esm.Document.Source.Md.fenceEsm">
<div class="lean-decl-head"><span class="lean-kind">def</span> <a class="lean-name" href="#Esm.Document.Source.Md.fenceEsm" title="Esm.Document.Source.Md.fenceEsm"><code>fenceEsm</code></a> <a class="lean-src" href="https://github.com/EarthSciML/EarthSciAST/blob/184aa13d35e5b30889e97811c776341973ff4e97/formal/Esm/Document/Source/Markdown.lean#L84">source</a></div>
```lean
def fenceEsm : List Char
```esm:
def dInit : List Char
The containers of initialization equations and constraints, and their end.
Path-block keys
def escKey (c : Char) : List Char
One character of a quoted key: the JSON escape, and the backtick as ``` (the info string of a backtick fence cannot hold a backtick).
def keyMd (k : String) : List Char
A path block’s key in the fence’s info string: bare when it is an identifier other
than description (the description block), quoted otherwise.
def parseKeyMd (info : List Char) : Option String
Read a path block’s key from the info string (all of it).
Declarations
def nameLatex (n : String) : List Char
A declared name, as LaTeX math under the empty symbol table (so it reads the same whatever else the document declares).
Attribute values: YAML scalars and flow collections
An attribute value is a YAML scalar (100, "mol", true) or a flow collection,
["x", "y"], {a: 1, "b c": [2]}: elements and entries separated by , , keys bare when
they are identifiers that YAML reads as strings, else quoted. A list or object (empty ones
too) is written single-quoted, shape='["x", "y"]', since remark-directive refuses an
unquoted value holding a bracket or brace; inside the quotes ' is ' and & is
&, the character references remark-directive decodes. The unquoted form is still
read.
def pFlow : Nat → List Char → Option (Json × List Char)
A value: a flow collection at [ or {, else a scalar.
def pItemsF : Nat → List Char → Option (List Json × List Char)
The elements after the first, each after , , up to ].
def pEntryF : Nat → List Char → Option ((String × Json) × List Char)
One entry: key: value.
def pEntriesF : Nat → List Char → Option (List (String × Json) × List Char)
The entries after the first, each after , , up to }.
def isColl : Json → Bool
A value whose YAML starts with a bracket or brace (a list or an object, empty ones included): micromark-extension-directive, and so remark-directive, refuses such a value unquoted.
def sqEsc (cs : List Char) : List Char
The text of a single-quoted attribute value: ' as ' and & as &, the
character references micromark-extension-directive decodes inside a quoted value.
def attrVal (v : Json) : List Char
An attribute value: a scalar as its YAML, a list or object as its YAML flow collection
single-quoted (shape='["x", "y"]').
def attrsMd : List (String × Json) → List Char
The attributes, separated by spaces.
def pQuoted (u : List Char) : Option (Json × List Char)
A single-quoted value after its opening quote: the YAML inside, then the text after the closing quote.
def pAttrsMore : Nat → List Char → Option (List (String × Json) × List Char)
The attributes after the first, each after a space, up to the closing brace.
def pAttrs (f : Nat) : List Char → Option (List (String × Json) × List Char)
The attributes after the opening brace, up to the closing one.
def declBody (f : Nat) (k : DKind) : List Char → Option (RItem × List Char)
A declaration after its [.
Block constructs
def fenceItem (info body : List Char) : Option RItem
A fenced block’s construct, from its info string and body.
def labelOK (sl : List Char) : Bool
An equation label after the closing $$: nothing, or {#id}, with spaces around.
def fenceAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A fenced block after ```esm:.
Templates and rules
A template is a display whose math starts with its header, \operatorname{f}(a, b) \coloneqq,
then its body: $$, \operatorname{f}(a, b) \coloneqq body on one line, $$. A rule is
\text{rule } \operatorname{r}: \; lhs \Rightarrow rhs. The header (the name, the parameters
as names under the empty symbol table) and the rule’s split at its arrow are read here, by
the lexer; only the body and the left-hand side are math. With other fields the display is
wrapped in a container directive whose attributes hold them, :::template{description="…"}
or :::rule{params='["u"]' priority=1}, closed by a line ::: (attributes after the
closing $$ would keep remark-math from closing the display).
def paramsTxt : List String → List Char
The parameters of a template header, separated by , .
def tmRaw (t : Tm String) : List Char
The math text of a template or rule display (one line).
def pName (s : List Char) : Option (String × List Char)
One parameter: a name, up to the next , or ).
def pParams : Nat → List Char → Option (List String × List Char)
The parameters after (, up to ) \coloneqq .
def splitAt (sep : List Char) : List Char → Option (List Char × List Char)
Split at the first occurrence of sep.
def tmLex (raw : List Char) : Option (String × List String × Option (List Char) × List Char)
Read the math text of a display as a template or rule: its name, its parameters, its left-hand side (a rule’s) and its body, as text.
def mathAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A display after its opening $$: a template or rule when its math reads as one,
otherwise an equation (optionally labelled).
def tmContAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A template or rule with other fields, after :::template{ or :::rule{: the
attributes, the display, and the closing ::: line.
def dirAt (K : Kind) (u : List Char) : Option (RItem × List Char)
A component directive after its [.
Reactions
def latexR : RSurf
The math surface of reaction sides: LaTeX under the empty symbol table.
def arrowL : List Char
The arrow with the rate over it, around the rate’s text (which ends at its $).
def rxnEnd : List Char
After the products: the end of the math and the start of the label.
def rxnBody (x : Rxn String) : List Char
The text of a reaction line after $$\ce{.
def rxnAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A reaction line after its $$\ce{.
def subAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A subsystem directive after its ::subsystem[.
def plotAt (f : Nat) (u : List Char) : Option (RItem × List Char)
A plot directive after its ::plot{.
Containers of initialization equations and constraints
def contBody (ro : Role) : Nat → List Char → Option (List RItem × List Char)
The displays of a container, up to its closing ::: line; blank lines between them
are skipped.
def contAt (f : Nat) (s : List Char) : Option (List RItem × List Char)
A container, at its opening line.
The lexer
def blockOpen (s : List Char) : Bool
A block construct starts here (at the start of a line).
def blockAt (f : Nat) (s : List Char) : Option (RItem × List Char)
Read the block construct that starts here.
def declAt (f : Nat) (s : List Char) : Option (RItem × List Char)
Read the declaration that starts here.
The declarations table (an accepted input variant)
A per-model declarations table is read as the declarations it lists; the printer never
writes it (it writes :var directives):
| decl | name | default | units |
| --- | --- | --- | --- |
| var | $N$ | 100 | "mol" |
| param | $\lambda$ | 0.1 | "1/s" |
The first two columns are the directive and the name; every other column header is an attribute key, and a cell is a YAML scalar or empty (the attribute is absent).
def headerCells (s : List Char) : List String
The header cells of a table line: the line split at |, each cell trimmed.
def rateTable (s : List Char) : Bool
The start of a reaction table with a rate column: a line starting with | that has a
cell rate. It is not a reaction notation (Notation.variants, rejected), and the lexer
refuses it rather than skipping it as prose, which would drop its reactions.
def pCols : Nat → List Char → Option (List String × List Char)
The attribute columns of the header, up to its newline.
def pCell : List Char → Option (Option Json × List Char)
One cell: value |, or | when the attribute is absent.
def pCells : List String → List Char → Option (List (String × Json) × List Char)
The cells of a row, one per attribute column.
def pRow (cols : List String) : List Char → Option (Decl × List Char)
One row: | var | $N$ | and the cells, then a newline.
def pRows (cols : List String) : Nat → List Char → Option (List Decl × List Char)
The rows, as long as lines start with | .
def tableAt (f : Nat) (s : List Char) : Option (List RItem × List Char)
A declarations table, at its | decl | name |.
Code spans
A run of backticks in prose opens a code span (or, at the start of a line, a fenced code block), which runs to the next occurrence of the same run; nothing inside it is a construct. A run with no such occurrence after it is literal.
def afterRun (n : Nat) : List Char → Option (List Char)
The text after the first occurrence of n backticks.
def skipCode (s : List Char) : List Char
The text after the code span that starts here (at a backtick).
def lex : Nat → Bool → List Char → Option (List RItem)
The Markdown lexer: the constructs of a text, skipping prose one character at a
time and code spans whole. bol: the text starts at the beginning of a line. A malformed
construct is an error (none).
Reading back the pieces
theorem pInline_gen (v : Json) (hi : Yaml.isInl v = true) (hv : v.WF) (rest : List Char)
(hr : Number.NumStop rest) : Yaml.pInline (Yaml.inline v ++ rest) = some (v, rest)
A YAML scalar read back whatever follows it (a number needs a non-number character after it).
def FStop (rest : List Char) : Prop
What may follow an attribute value: ,, ], }, a space or |.
theorem pFlow_flow' (v : Json) (hv : v.WF) (rest : List Char) (hr : FStop rest) :
pFlow (2 * (flow v ++ rest).length + 2) (flow v ++ rest) = some (v, rest)
A value printed as a scalar or flow collection reads back.
def attrOK (a : String × Json) : Bool
An attribute the Markdown attribute syntax can hold: an identifier key and a well-formed value.
def mhchemOK (x : Rxn String) : Bool
What mhchem reads as written: no escaped brace \{ or \} anywhere inside \ce{…}
(mhchem matches braces by character and fails on them).
def rxnOK (x : Rxn String) : Bool
A reaction the Markdown syntax can hold: sides the side syntax holds, a rate on one
line without $, an id label, and attributes the attribute syntax holds.
Keys
The printer
def itemMd : RItem → List Char
The text of a construct. A description is written verbatim in its fence; a path block’s value in YAML; a declaration inline (no newline of its own).
def eqnLabeled (raw : String) (label : List Char) : List Char
A display equation with an equation label after its closing $$ ( {#eq-x}).
inductive Seg where
| item (it : RItem)
| labeled (raw : String) (label : List Char)
| prose (p : List Char)
| code (body : List Char)
A piece of a Markdown document: a construct, a labelled equation, prose, or a code
span (`…`, whatever its content).
def segItems : List Seg → List RItem
The constructs of a document body: prose dropped, labels dropped.
def isBlock : RItem → Bool
A construct that must start a line (all but declarations).
def itemOK : RItem → Bool
What each construct’s content must satisfy for its text to read back.
def Inert : Bool → List Char → List Char → Prop
Prose that the lexer skips: at no character does a construct start (a block
construct only matters at the start of a line), and it holds no backtick (code spans are
their own pieces, Seg.code). t is the text after the prose.
def Valid : Bool → List Seg → Prop
A document body whose constructs read back: each construct’s content is spellable, block constructs start a line, labels are labels, and prose is inert.
Lexing, construct by construct
theorem lex_code (body t : List Char) (hb : body ≠ []) (hc : '`' ∉ body) (bol : Bool) (g : Nat) :
lex (g + 1) bol ('`' :: body ++ '`' :: t) = lex g false t
A code span is skipped whole, whatever it holds.
Reactions
Subsystems
Plots
Templates and rules
Containers
theorem contBody_many (r : Role) (t : List Char) : ∀ (raws : List String),
(∀ raw ∈ raws, itemOK (.eqn raw) = true) → ∀ g,
(raws.flatMap dispTxt).length + t.length + 4 < g →
contBody r g (raws.flatMap dispTxt ++ dEnd ++ '\n' :: t) = some (raws.map (.eqx r), t)
A container holding several displays reads as their equations, in order: each
display followed by a blank line, then the closing ::: line (the printer writes one
display per container).
theorem lex_render : ∀ (segs : List Seg) (bol : Bool), Valid bol segs →
∀ f, (render segs).length < f → lex f bol (render segs) = some (segItems segs)
The Markdown lexer reads back every valid document body: exactly its constructs, whatever prose surrounds them and whatever labels the equations carry.
The declarations table reads as its declarations
def tableText (cols : List String) (ds : List Decl) : List Char
A declarations table over the attribute columns cols.
def TableOK (cols : List String) (d : Decl) : Prop
A declaration a table over cols holds: its name and attributes spellable, and its
attributes exactly the columns it has a value in, in column order.
theorem lex_table (cols : List String) (hc : ∀ k ∈ cols, isIdent k.toList = true)
(ds : List Decl) (hd : ∀ d ∈ ds, TableOK cols d) (t : List Char)
(ht : pre ['|', ' '] t = false) (g : Nat) (hg : (tableText cols ds).length + t.length < g + 1) :
lex (g + 1) true (tableText cols ds ++ t) = (lex g true t).map (ds.map RItem.decl ++ ·)
A declarations table reads as the declarations it lists (and as the
:var/:param/:species directives of the same declarations: the same constructs).