Unified Markdown surface — research and design

Status: built. The research and design below were written first (2026-09-20); every phase of §10.3 has since landed in code, and the old surfaces are gone (phase 5, 2026-09-24). What is still owed is almost all verification on a device, listed in §10.6. The sections that say what will be built are kept as written; each phase’s own status notes say what was. Branch: feat/unified-markdown-surface (cut from remote main @ 16691f7). Date: 2026-09-20 (design); 2026-09-24 (status). Scope: replace the source editor, the WYSIWYG editor and the preview with one widget, written from scratch in pure Flutter/Dart, uniform in functionality, performance and visual rendering, supporting CommonMark, GitHub Flavored Markdown, wikilinks, inline math and block math, and fast on Geometria 1.md (934 769 bytes, 10 331 lines, 13 845 math spans — 841 display blocks and 13 004 inline — measured, §5.4).

The verdict in six sentences

1. The design that satisfies all the constraints is one widget with three modes (source, live, read) over one pipeline, where the note’s Markdown text is the only document, the parse tree is derived and disposable, and the three modes differ only in which characters are rendered, whether there is a caret, and whether prose is monospaced (§8). This is the only model that satisfies the repo’s “disk is source of truth” rule without a lossy serialization codec.
2. The performance answer is not a new trick — it is the app’s own existing tricks, generalized: an O(change) line-state tokenizer whose keystroke chain is already flat at 0.30 / 0.28 / 0.37 ms across 200 KB / 500 KB / 934 KB, a block/inline parse split that already measures 14 ms vs 378 ms on the 934 KB note, and a frozen-extent height map that already renders only the visible blocks. The rewrite’s job is to make all three modes run on them, which is what removes the WYSIWYG’s 200 KB cap (below which, measured, it does not even open any of this repo’s fixtures), the preview’s 137–182 ms wait for first content, and its per-block math cost on the geometry note.
3. The hardest engineering problem is not the renderer, it is the editable text surface: a custom TextInputClient that reports visual text while the document is source text, with a per-block offset map, an Android IME that has to be verified on a real device, and no silent data loss. This is where the schedule risk lives, and it should be spiked before the plan is committed to.
4. The parser is measured, not written (D2) — and the measurement is in. The pure-Dart markdown package scores 645/652 on CommonMark 0.31.2 and 662/677 on GFM (§4.9), with 22 examples triaged: 8 to fix, 14 pinned because they are serializer details or spec quirks that never reach the user. highlight stays as the code grammar engine with its rendering path removed. That takes 23–42 days out of the estimate — from ~140–245 to ~121–210 — and two new requirements (the export seam, and removing the split preview) take 5–9 of it back.
5. The rewrite also deletes real duplication: today the same find bar, list tally, context menu, formatting commands, active-format computation and link handling are implemented once per surface, with slightly different behaviour. One widget collapses each of those rows to one implementation — and the cross-engine scroll sync the split preview needed is deleted outright rather than kept in step (D4).
6. It is a 6–10 month project for one engineer, and it can be stopped at any phase boundary. read mode alone (Phase 2) fixes the preview’s wait for first content, the math cost and the size cap, and is worth shipping on its own.

How to read this document

Statements are labelled by provenance throughout: measured (a command was run, the command is given), read (from the repo, a spec or a source, with a link), or proposed (the author’s design judgement). §12.5 explains the method.

Table of contents

• The verdict in six sentences
• How to read this document
• 1. The mandate
  • 1.1 What was asked
  • 1.2 What that decomposes into
  • 1.3 The repo’s own rules that constrain the answer
  • 1.4 What keeping the two pure-Dart engines changes
  • 1.5 The decisions, and what each one changes
• 2. Where we are today
  • 2.1 The three surfaces, and the code that exists for them
  • 2.2 What the code already measured (and what it says to keep)
  • 2.3 The existing assets worth porting, not discarding
• 3. The feature contract: what must not regress
  • 3.1 The drop-in contract: what the one widget must answer
  • 3.2 The overlap: what the rewrite actually deletes
• 4. The specification: CommonMark and GFM
  • 4.1 Versions and where they live
  • 4.2 Block-level element inventory
  • 4.3 Inline-level element inventory
  • 4.4 The emphasis algorithm in executable detail
  • 4.5 Spec test suite
  • 4.6 Extension zone — what CommonMark/GFM deliberately do not cover
  • 4.7 Round-trip fidelity requirements
  • 4.8 Recommended conformance strategy
  • Appendix A — Quick reference constants for the Dart implementation
  • Appendix B — Sources
  • 4.9 The markdown package measured
• 5. The corpus reality
  • 5.1 Global stats
  • 5.2 Block-level census
  • 5.3 Inline-level census
  • 5.4 Math deep-dive
  • 5.5 Structural worst cases
  • 5.6 Scroll/layout sizing model
  • 5.7 A synthetic “worst note” spec and fixture
  • 5.8 Summary of the numbers that should drive the design
• 6. The primitives available
  • 6.0 Executive summary of the cost model
  • 6.1 Text layout & painting cost model
  • 6.2 Building a custom render/viewport
  • 6.3 Editable text from scratch
  • 6.4 Making text look native-quality without HTML
  • 6.5 Math rendering without a package
  • 6.6 Syntax highlighting without a package
  • 6.7 Isolates and parsing off the UI thread
  • 6.8 A concrete performance budget
  • 6.9 Traps and known Flutter pitfalls
  • 6.10 Suggested module layout for the new widget
  • 6.11 Source index (for follow-up reading)
• 7. Prior art: what to steal
  • 7.1 The three candidate architectures
  • 7.2 CodeMirror 6, mechanism by mechanism
  • 7.3 Incremental parsing: Lezer, tree-sitter, and the minimum for Markdown
  • 7.4 ProseMirror and Lexical: the position-mapping idea
  • 7.5 Live preview in the wild
  • 7.6 Virtualized document rendering in other ecosystems
  • 7.7 Math rendering: KaTeX and MathJax
  • 7.8 Syntax highlighting prior art
  • 7.9 What Niman already does that must be carried over
  • 7.10 Ideas worth stealing
• 8. The proposal: one surface, three modes
  • 8.1 The shape of the answer
  • 8.2 The document model: the source text is the document
  • 8.3 One typography, one block metric set
  • 8.4 Layout: only the blocks the viewport shows
  • 8.5 The parser: incremental, two-phase, line-oriented
  • 8.6 Inline rendering: change styles, not characters
  • 8.7 Editing mechanics
  • 8.8 The two specialised renderers
  • 8.9 What replaces what
  • 8.10 How each requirement is met, concretely
  • 8.11 A sketch of the public API
• 9. The performance budget
  • 9.1 How to read a number in this project
  • 9.2 The budget, by operation
  • 9.3 The legacy baseline to beat (measured)
  • 9.4 The benchmark harness the implementation must ship with
  • 9.5 Memory budget
  • 9.6 Regression gates
• 10. Effort and phasing
  • 10.1 The rule that shapes the plan
  • 10.2 What the decisions do to the effort
  • 10.3 The stages
  • 10.4 What to spike first, before committing to the plan
  • 10.5 How this lands in the repo’s workflow
  • 10.6 What is still owed
• 11. Risks, alternatives and open questions
  • 11.1 The risks, ranked by what they would cost
  • 11.2 Alternatives considered, and why they lose
  • 11.3 The open questions, and what was answered
  • 11.4 What would make this project fail, in one sentence each
• 12. Appendix
  • 12.1 Source inventory of the surfaces being replaced
  • 12.2 The dependency ledger
  • 12.3 Canonical references
  • 12.4 Glossary
  • 12.5 How this document was produced, and how to reproduce it
  • 12.6 First-day checklist
• 13. Annex: the incumbent baseline, measured in full
  • 13.1 Environment and method
  • 13.2 Results
  • 13.3 Complexity analysis (from the numbers)
  • 13.4 Headline table the new implementation must beat
  • 13.5 Reproducibility
  • 13.6 Caveats — why these numbers are not device reality

1. The mandate

1.1 What was asked

Replace the three components that today act as source editor, WYSIWYG editor and preview with one widget that is uniform in functionality, performance and visual rendering. It must keep supporting everything the app supports today — CommonMark, GitHub Flavored Markdown, wikilinks, inline math and block math. No existing package may be adopted: the implementation is to be written from scratch and kept as pure Flutter/Dart as possible. It must stay fast on adversarial real documents, with Geometria 1.md (934 769 bytes, 10 331 lines, 13 845 math spans — 841 display blocks and 13 004 inline) as the named worst case. Everything learned and thought through goes into one Markdown file. The work happens on an ad-hoc branch cut from remote main.

1.2 What that decomposes into

1.3 The repo’s own rules that constrain the answer

From AGENTS.md and docs/dev/conventions.md, all of which the proposal must satisfy rather than argue with:

• Disk is source of truth. One note = one .md. Nothing may be stored that cannot be rebuilt from the file. This is the single most important constraint on the document model: a Delta-like or tree-like in-memory document that is serialized on save is out; the source text must be the document.
• No O(n) full scans on hot paths. Target: 1 M notes and novel-length files.
• No disk I/O on the UI isolate.
• No god classes: one class per file, split at ~300 lines.
• Every platform ships every feature — Android, Linux, Windows in the same round. A surface that is fast on desktop and unusable on a phone is not acceptable.
• The keep-its-place interface rule: a control that shows only in some states keeps its place.
• English everywhere, docs updated in the same PR, one logical change per commit.

1.4 What keeping the two pure-Dart engines changes

Measured on this checkout with flutter pub deps --json: the seven surface packages (flutter_quill, re_editor, flutter_markdown_plus, katex, katex_dart, flutter_highlight, flutter_smooth_markdown) pull in a dependency closure of 122 packages. Of the 235 packages in the graph, 55 (23 %) are reachable only through the surface stack — including quill_native_bridge_* (native code compiled into every one of the seven platforms), sqflite (a second SQLite, on top of the sqlite3 the index already bundles), flutter_svg, vector_graphics*, cached_network_image, flutter_cache_manager, provider, flutter_colorpicker, html, csslib, isolate_manager, isolate_contactor, rxdart, diff_match_patch, flutter_math_fork, file_selector_* and flutter_keyboard_visibility_*.

Building the surface from scratch removes those 55 packages and the seven direct dependencies, plus ~10 500 lines of Niman code that exists to bridge those packages’ models to Markdown (wysiwyg/markdown_document_codec.dart, wysiwyg/quill_*, preview/block_parse.dart’s glue, the three separate find controllers, three separate toolbars’ dispatch).

Decided (D2, 2026-09-20): markdown and highlight stay. Neither pulls native code, so neither compromises R7, and keeping them removes the single largest work item in this document. What changes is not whether the surface is written from scratch — it still is — but where the parser comes from:

• markdown 7.3.1 becomes the CommonMark/GFM engine, and Phase 1 stops being “write a conformant parser” and becomes measure and close the gap: run the spec suite against it, enumerate and pin every failure, and fix only what the app actually needs (§4.8). That is days, not weeks — and it is the biggest single lever in this document (§10.2).
• What it does not remove. The package has no incremental API, so the line-state block scanner and the lazy per-block inline phase (§8.5) are still Niman’s — the package is called per block, on already-bounded input, never over the document. The single-token-source invariant for links (§2.3) is still Niman’s, and the extensions the package does not know (wikilinks, math delimiters, frontmatter) are still Niman’s.
• highlight stays as the code grammar engine, not as the renderer. Its measured hazards are unchanged and must be contained (§8.8.2, K13): never call Result.toHtml() (super-quadratic: 100 KB → 6 869 ms), never lex more than the visible or changed lines, and guard the input, because Dart has no regex timeout and the cpp grammar is O(n²) on adversarial input. The line-state driver and the line cache are Niman’s either way; the package supplies the language rules.

1.5 The decisions, and what each one changes

Recorded 2026-09-20, from the user’s answers. Everything below is settled unless it says otherwise, and the rest of this document assumes it.

2. Where we are today

2.1 The three surfaces, and the code that exists for them

Totals: 11 814 lines across lib/src/{editor,preview,links,frontmatter}, of which roughly 4 600 are source-editor, 2 500 WYSIWYG, 3 400 preview, and the rest shared (links/, frontmatter/). 38 test files / 8 651 lines exercise those surfaces (of 48 970 test lines in the repo).

The three are wired together in exactly one place, lib/src/ui/note_view.dart (_NoteViewState, 2 225 lines): _editorPane() chooses WysiwygEditor or NoteEditor from widget.showWysiwyg, and _buildPreview() mounts MarkdownPreview beside them. Everything else — toolbar, format menu, find, tools, spellcheck, links, tabs — is dispatched from that State into whichever surface is up, and each surface answers through a different interface. That dispatch layer is where most of the “uniformity” problem actually lives.

2.2 What the code already measured (and what it says to keep)

These numbers are in the source comments and in docs/records/editor-alternatives.md. They are the baseline; they are also, in two places, the design that the rewrite should keep rather than reinvent.

On the 0.507 ms. That is the figure of record in editor-alternatives.md, and it was measured through HighlightDocument.replace — the offset-edit API, not the app’s keystroke path. The re-measurement for this document (§9.3) splits it: the real keystroke chain (a controller edit plus the synchronous EditorHighlightSync listener) is 0.296 ms best / 0.319 ms median at 200 KB and stays flat at 0.277/0.357 (500 KB) and 0.372/0.458 (934 KB), while the offset-edit API alone measures 0.462/1.026. Two consequences: the bar is a flat ~0.3–0.5 ms across a 4.5× size range rather than a single number, and the offset-edit path is the O(lines) one (it rewrites the whole _lineStarts index) — a reminder not to reintroduce an offset-based edit model in the replacement without doing the same accounting.

2.3 The existing assets worth porting, not discarding

The rewrite is not a green field. Four pieces of Niman code are already the right shape and should be carried over nearly intact:

1. The incremental tokenizer (editor/highlighting.dart). It is a line-oriented state machine: each line gets a _State (fence, inMath, inFrontmatter), tokens are cached per line, and HighlightDocument.replaceLines re-tokenizes only what moved. This is precisely CodeMirror 6’s StreamLanguage/Lezer idea, already in Dart, already O(change), already measured at 0.507 ms. It already knows about fences, inline math, display math, frontmatter, headings, lists, task boxes, blockquotes, rules, links, images, wikilinks, tags, code spans, emphasis and strikethrough — i.e. it is ~80 % of a block and inline scanner. Port it, generalize it, and make it the single token source.

2. The single-parse-rule invariant for links (links/parser.dart). It explicitly reads over the shared tokenizer so the indexer, the editor’s Ctrl+click and the preview agree on what a link is, and so links inside code fences, math or frontmatter are never links. That invariant is the reason the app’s behaviour is coherent today, and the new surface must inherit it — one tokenizer feeding highlighter, links, indexer, outline, tags, spellcheck and rendering.

3. The height map and two-way line↔pixel mapping (preview/scroll_map.dart). ScrollMap holds blockStartLines, measured blockHeights, _extents frozen on first use, and binary searches for blockForLine, previewOffsetForLine, lineForPreviewOffset. It feeds SliverVariedExtentList.itemExtentBuilder so a jump lays out only the blocks it lands on. Its two hard-won lessons are recorded in its own comments and must not be re-learned: (a) an unmeasured block’s extent must be frozen on first use, because letting a shared average re-estimate an already-placed block makes SliverVariedExtentList assert as offsets move under the scroll position; (b) without a real total extent the lazy list guesses low and every long jump is clamped, which is what once pulled the preview away from the editor. Generalize it into the surface’s height map, and add what it lacks: a Fenwick/prefix-sum so lookups are O(log n) rather than O(blocks).

4. The off-thread parse split (preview/preview_work.dart). A top-level isolate entry, message-in/message-out of plain strings, deliberately not Isolate.run (its closure carries the State’s element and the isolate rejects it), with a typed failure object so a failed read is never mistaken for text. The block phase runs off-thread, the inline phase per block on demand. Keep the shape; upgrade the payload — the message today is the whole source string, so every keystroke ships 934 KB across the port boundary. The replacement should hold a mirror of the buffer in the worker and send only the splice.

3. The feature contract: what must not regress

3.1 The drop-in contract: what the one widget must answer

This is the checklist that decides whether the replacement is actually a drop-in. Every item below is a real call site or a real test today.

3.1.1 NoteEditor’s constructor (source mode)

controller: CodeLineEditingController · focusNode · showLineNumbers · autofocus · fontSize · scrollController: CodeScrollController · findController: CodeFindController · findBuilder: CodeFindBuilder · shortcutsActivators · onIndicator: ValueChanged<CodeIndicatorValueNotifier> (the laid-out line rows the scroll sync reads) · spellCheck: EditorSpellCheck · column: NoteColumn · formatMenu: FormatMenuBuilder · caretWidth · typewriter.

Two of these are package types that disappear with re_editor (CodeLineEditingController, CodeIndicatorValueNotifier); the new widget owns its own equivalents. onIndicator matters: the comment explains that the editor’s maxScrollExtent counts unwrapped lines as one row, so a fraction of it cannot say which line is on screen — the surface must expose its own line-level indicator, not a scroll fraction.

3.1.2 WysiwygEditorState’s imperative API (live mode)

requestEditorFocus() · openFind({replace}) · replaceDocumentRange(line, start, end, replacement) (the spell panel’s fix) · plainTextLines · hasFocus · scrollOffset · scrollController · controller (handed to QuillEditorCommands for the toolbar) · activeItems: ValueNotifier<Set<ToolbarItem>> (what is on at the caret, for the toolbar’s pressed state) · autoFocus · typewriter.

Note activeItems and the toolbar’s command dispatch are the uniformity surface: they must work identically in source and read modes, and today they are reimplemented twice (quill_editor_commands.dart, md_editing.dart + highlight_sync.dart).

3.1.3 MarkdownPreview’s constructor (read mode)

data · styleSheet · syntaxHighlighter · imageBuilder · checkboxBuilder · bulletBuilder · builders · padding · column · controller: ScrollController · onTapLink(text, href, title) · onWikiLink(WikiRef, display) · embedResolver(String) → Future<String?> · mathStyle · mathCache · scrollMap · imageDirectory.

3.1.4 Behaviour the docs promise and tests pin

• Tabs and per-tab state: caret, selection and scroll come back; undo history survives a tab moving between panes (“keeping everything, undo included”); a very long note keeps its editor only while it is on screen — which means the surface’s state must be cheap to tear down and rebuild, and undo must not live inside a scroll-dependent object.
• Scroll sync between an editor pane and a preview pane (scroll_map.dart + scroll_sync.dart + editor_lines.dart, ~970 lines) — today a two-way estimator over two different layout engines.
• Zen mode: hides rail/tree/tabs/side panel/toolbar row/status row/line numbers, thickens the caret, keeps the preview switchable, keeps the split panes alive behind it.
• Typewriter mode: the caret’s row (not line) stays centred, with half a screen of room below the last line; wheel/scrollbar scrolling is unaffected until the caret moves again.
• The split preview goes (D4). Today a pane can show editor and preview side by side (splitPreview, previewEnabled, a 0.2–0.8 split ratio, and the auto behaviour above 600 dp), kept in step by a two-way scroll sync. With live mode as an interactive preview (D9) that whole axis is redundant, so it is removed rather than generalized: the preview becomes the read mode of the same surface, switched like any other mode. This is a deliberate, user-visible change and it must land with docs/user/editing.md updated in the same commit — including the loss of previewEnabled and the split ratio, and the fact that the two-pane split (two notes side by side, Ctrl+\) is a different feature and stays.
• Readable line length: a centred column, on by default, 480–1400 px (700 default), shared by both editors and the preview, with the toolbar, find bar and status row keeping to the same column while their backgrounds span the pane; the scrollbar spans the pane and the wheel works from the margins.
• Preview-only pane: when the preview is the only pane the IME must be dismissed (_dismissKeyboardForPreview).
• Ctrl+click on a link in the editor (mouse only; touch clicks stay edit-only) opens it; the caret-landed-token read is retried across a couple of frames because the package’s tap handler lands late.
• Clipboard carries Markdown in both editors, so the same text copies identically from either; pasting Markdown into the WYSIWYG restores structure; pasting HTML uses the HTML.
• Tidy the Markdown: never touches tables, math, frontmatter or HTML, nor the code inside a fence (only its two fence lines, for their language and closing fence, #72); idempotent; saves first.
• Enter inside a list carries the list on in both editors; inside a fence, math block or frontmatter it does nothing — and _isPlainLine answers that from the highlighter the editor already keeps, explicitly so the two cannot disagree about what a list is. The new surface must keep that property: one source of truth for “what is this line”.
• Tools (EditorTool.countList): both editors offer the same tools, and an unavailable tool is listed greyed with its reason rather than hidden — and list_tally + quill_tally are today two implementations of one tool.
• Context menu: the formatting actions with the same icons, names and grouping as the toolbar, with the format at the caret reading as on, applied to the selection; right-click (long-press on a phone). Today editor_context_menu.dart and Quill’s own menu are two code paths.
• Absolute paths and offsets everywhere: the caret’s offset is a source offset; initialCaretOffset from a template {{cursor}} places the caret and takes focus; NoteEditor/WysiwygEditor both accept it.

3.1.5 The feature inventory (nothing here may regress)

Formatting: bold, italic, strikethrough, superscript, underline, link, code, image, heading, bullet list, ordered list, quote, outdent, indent, tools (14 toolbar items). Find/replace with case sensitivity and replace-one/replace-all through the undoable edit path. Folding by heading section. Outline. Word count. List tally. Spellcheck: hunspell on desktop, system IME on Android, a per-library personal dictionary, right-click Add to dictionary, a background-scanned issues panel capped at 200 words with a progress bar. Links: Markdown links, wikilinks with #heading and |alias, ![[embed]] resolution (library-root, then note-relative, then unique-name via the index), dead-link note creation. Images: copied into the library, never base64. Frontmatter: YAML with known fields (title tags date pinned aliases). Math: $…$ inline, $$…$$ display, including display math inside lists and blockquotes. Tags: inline #tag shared with the index.

3.2 The overlap: what the rewrite actually deletes

The reason a single widget is attractive is measurable in duplication:

Every row with three entries is a place where the same user-visible feature is implemented once per surface and behaves slightly differently. A single widget collapses each row to one implementation. That — not line count — is the real argument for R2.

4. The specification: CommonMark and GFM

Research reference for a hand-written CommonMark + GFM + extensions parser/renderer/editor in pure Dart. Compiled from primary sources (spec text, spec.json, cmark-gfm sources) fetched 2025; every claim below is traceable to a URL in the section it appears in.

Scope of the target surface: CommonMark 0.31.2 ⊂ GFM 0.29-gfm ⊂ Niman extensions (wikilinks, inline math $...$, display math $$...$$, YAML frontmatter). Disk is authoritative; the writer never rewrites user bytes.

4.1 Versions and where they live

4.1.1 Version table

4.1.2 Relationship: GFM = CommonMark + 5 extensions (+ a few silent deltas)

The GFM spec states verbatim: “GFM is a strict superset of CommonMark. All the features which are supported in GitHub user content and that are not specified on the original CommonMark Spec are hence known as extensions.” (gfm/spec.txt §1.1)

The five extensions, and the only five registered in cmark-gfm’s extensions/core-extensions.c (source):

smart (smart punctuation) exists in cmark-gfm’s tree as an optional extension but is not part of GFM and is not enabled by GitHub; do not implement it. Renderer-level options that are not syntax: full-info-string (emits data-meta="…" for the rest of a fence info string) and table-prefer-style-attributes (style="text-align: left" instead of align="left") — see test/extensions-full-info-string.txt and test/extensions-table-prefer-style-attributes.txt.

4.1.3 GFM deviations from CommonMark — measured, not asserted

I diffed all 677 published GFM examples against all 652 CommonMark 0.31.2 examples, section by section, after normalizing the → tab glyph. Result: shared sections are example-for-example identical except for the following. These are the complete deviations an implementer must know.

Two operational consequences:

1. The published GFM HTML is a partial CommonMark-0.30 sync; cmark-gfm’s test/spec.txt is not. They differ in 12 examples (HTML-only) vs 7 (spec.txt-only) — e.g. HTML has [ẞ] and <!-- not a comment -- two hyphens -->, test/spec.txt has [Толпой] and <!-- this is a --\ncomment - with hyphens -->. Pick one as the golden source and pin it by commit SHA. Recommendation: use the published HTML’s 677 examples (that is what “0.29-gfm” means to users).
2. D3 (HTML comments) is the only deviation that changes parsing of common input. CM 0.31.2 accepts <!--> foo --> as raw HTML; GFM escapes it. Since Niman wants both, see §4.8 for how to pin this.

Prose-level deltas that are not per-example diffs: GFM’s spec text is CM 0.29’s text for §2–§3, §5.1–5.2, §5.4, §6.1–6.4, §6.6–6.8, §6.10, §6.12–6.14 — including CM 0.29’s <!-- comment rule (D3) and CM 0.29’s “1–7 digits” wording is already present in both (the digit constraint matched; only the test example differs).

4.2 Block-level element inventory

Two-phase model is mandatory: phase 1 parses block structure line by line; phase 2 parses the raw text of each leaf block (paragraph, heading, table cell, link title/destination, info string) for inlines. Phase 2 needs the link-reference-definition map, so it runs only after phase 1 completes. (CM 0.31.2 §Precedence, Appendix: A parsing strategy)

“Indent” below always means up to 3 spaces unless stated otherwise. Tabs advance to the next multiple-of-4 column when measuring indentation, but are preserved literally inside content.

4.2.1 List-marker grammar (feeds B18/B19)

4.2.2 Tight vs loose — exact operationalization

loose(list) = ∃ itemᵢ, itemᵢ₊₁ separated by ≥1 blank line
           OR ∃ item with ≥2 direct child blocks separated by ≥1 blank line
tight(list) = ¬loose(list)

Blank lines inside a fenced code block, an indented code block, or a type-1–5 HTML block do not count. Blank lines inside a sublist make the sublist loose; whether they make the outer list loose depends on whether they separate the outer items or the outer item’s direct children. The HTML consequence is <p> wrapping only.

4.3 Inline-level element inventory

4.3.1 Precedence (the “which wins” table)

CommonMark resolves conflicts by four stated principles plus a fixed scan order. Implementation order:

Sources: CM §Precedence, CM §6.4 principle 17.

4.3.2 The inline table

Sources: §6.1, §6.2, §6.3, §6.4, §6.6, §6.7, §6.8, §6.10, §6.12, GFM §6.5, GFM §6.9, GFM §6.11.

4.3.3 Character classes (needed for flanking)

Dart note: RegExp(r'[\p{P}\p{S}]', unicode: true) covers Unicode punctuation in Dart 3. U+0000 in input must be replaced by U+FFFD before parsing (CM §2.3 “Insecure characters”).

4.4 The emphasis algorithm in executable detail

This is the CommonMark Appendix procedure plus the openers_bottom optimization as actually implemented in commonmark.js (lib/inlines.js, processEmphasis, scanDelims). Follow it literally.

4.4.1 Data structures

class Delim {
  final int cc;          // 0x2A '*' | 0x5F '_' | 0x7E '~' | 0x5B '[' | 0x21 '!' (for '[', cc is '[' and image is a flag)
  final Inline node;     // the text node holding the literal delimiters
  int numdelims;         // delimiters not yet consumed
  final int origdelims;  // length of the run as originally scanned — NEVER mutated
  final bool canOpen, canClose;  // from scanDelims
  Delim? previous, next;
}

The delimiter stack is a doubly linked list held on the inline parser. numdelims mutates; origdelims does not, and the rule-of-3 test uses origdelims.

4.4.2 scanDelims(cc) — compute canOpen / canClose

numdelims ← count of consecutive cc starting at pos       // for '!'+ '[' handle separately
if numdelims == 0: return null
char_before  ← the char before the run, or '\n' if at start of subject
char_after   ← the char after the run, or '\n' if at end of subject
after_ws  = isUnicodeWhitespace(char_after)
after_pn  = isUnicodePunctuation(char_after)
before_ws = isUnicodeWhitespace(char_before)
before_pn = isUnicodePunctuation(char_before)

left_flanking  = !after_ws  && (!after_pn  || before_ws || before_pn)
right_flanking = !before_ws && (!before_pn || after_ws  || after_pn)

if cc == '_':
    canOpen  = left_flanking  && (!right_flanking || before_pn)
    canClose = right_flanking && (!left_flanking  || after_pn)
else:                       // '*' and '~'
    canOpen  = left_flanking
    canClose = right_flanking

pos ← startpos              // do not consume here; the caller re-reads
return (numdelims, canOpen, canClose)

“the beginning and the end of the line count as Unicode whitespace” — hence the '\n' defaults.

Delimiter-run definition: a run of * is a maximal sequence not preceded or followed by a non-backslash-escaped *. An escaped \* therefore breaks a run.

Checked examples from the spec:

| Input | left-flanking? | right-flanking? | |—|—|—| | ***abc | yes | no | | ` abc | yes | no | | “abc” | yes | no | | abc* | no | yes | | “abc” | no | yes | | abc**def | yes | yes | | “abc”_“def” | yes | yes | | abc ** def | no | no | | a _ b` | no | no |

4.4.3 processEmphasis(stackBottom)

stack_bottom is either null (whole document) or the [/![ delimiter that opened the link/image currently being closed. It is an exclusive lower bound.

openers_bottom ← array of 14 slots, all initialized to stackBottom

closer ← first delimiter above stackBottom
while closer != null:
    if !closer.canClose:
        closer ← closer.next; continue

    // index: 0 = '\'' , 1 = '"' (smart-punct only; unused without it)
    //        '_' → 2 + (closer.canOpen ? 3 : 0) + (closer.origdelims % 3)   → 2..7
    //        '*' → 8 + (closer.canOpen ? 3 : 0) + (closer.origdelims % 3)   → 8..13
    idx ← indexFor(closer)

    opener ← closer.previous
    openerFound ← false
    while opener != null
          && opener != stackBottom
          && opener != openers_bottom[idx]:
        oddMatch = (closer.canOpen || opener.canClose)
                && closer.origdelims % 3 != 0
                && (opener.origdelims + closer.origdelims) % 3 == 0
        if opener.cc == closer.cc && opener.canOpen && !oddMatch:
            openerFound ← true; break
        opener ← opener.previous

    oldCloser ← closer

    if closer.cc in {'*','_','~'}:                    // '~' only when the GFM extension is on
        if !openerFound:
            closer ← closer.next
        else:
            useDelims ← (closer.numdelims >= 2 && opener.numdelims >= 2) ? 2 : 1
            opener.numdelims -= useDelims
            closer.numdelims -= useDelims
            // truncate the literal text nodes by useDelims chars from the END of both
            emph ← new Node(useDelims == 1 ? emph : strong)   // for '~' → del
            move all inline siblings strictly between opener.node and closer.node into emph
            opener.node.insertAfter(emph)
            removeDelimitersBetween(opener, closer)
            if opener.numdelims == 0: unlink(opener.node); removeDelimiter(opener)
            if closer.numdelims == 0:
                temp ← closer.next
                unlink(closer.node); removeDelimiter(closer)
                closer ← temp
            // else: do NOT advance — the same closer may pair again
    else:                                             // smart punctuation (omit)
        ...

    if !openerFound:
        openers_bottom[idx] ← oldCloser.previous       // lower bound for future searches
        if !oldCloser.canOpen:
            removeDelimiter(oldCloser)                 // it can never be a closer-for-this-type again

// finally drop every delimiter above stackBottom
while delimiters != null && delimiters != stackBottom: removeDelimiter(delimiters)

4.4.4 look for link or image and how it invokes processEmphasis

on ']':
  walk back the delimiter stack for an opener '[' or '!['
  if none:                    return literal ']'
  if found but !active:       remove that delimiter; return literal ']'
  if found and active:
      if not (inline link | full ref | collapsed ref | shortcut ref): 
          remove opener delimiter; return literal ']'
      else:
          node ← Link or Image whose children are the inlines after opener.node
          processEmphasis(opener)             // <-- stack_bottom = the '[' delimiter
          removeDelimiter(opener)
          if node is a Link (not Image):
              for every '[' delimiter *before* opener: active ← false   // forbids nested links

Note the ordering: processEmphasis runs before the opener is removed and before earlier [s are deactivated. Also note that at the very end of inline parsing the parser calls processEmphasis(null) once.

4.4.5 The rule of three (spec rules 9/10) in one sentence

If one of the two delimiters can both open and close emphasis, then (len(openerRun) + len(closerRun)) must not be a multiple of 3, unless both lengths are multiples of 3.

The implementation form is exactly the oddMatch expression above, and it uses origdelims, not numdelims. Getting this wrong is the single most common source of spec failures.

4.4.6 The four tie-breaking principles (spec rules 13–17)

4.4.7 Why openers_bottom exists, and how to get it right

Naïvely, for each closer you rescan the whole stack: O(n²). openers_bottom caches, per (delimiter char, closer.canOpen, closer.origdelims % 3) bucket, the deepest stack position at which a matching opener is known not to exist. Because a failure to find an opener for a closer of run-length L also proves there is no opener for any other closer with the same L % 3 and the same canOpen, the bound is sound.

Two easy-to-miss details:

1. The bucket key uses the closer’s original run length mod 3, and whether the closer can also open — six buckets per delimiter char.
2. openers_bottom is local to one processEmphasis call. The array is re-created (all slots = stack_bottom) on every invocation, including nested invocations from link closers.

4.5 Spec test suite

4.5.1 Counts

4.5.2 CommonMark 0.31.2 section-by-section example counts

4.5.3 GFM 0.29-gfm section-by-section example counts (677 total)

4.5.4 JSON shape

spec.json (CommonMark 0.31.2) is an array of 652 objects, each:

{
  "markdown":  "Foo\nBar\n---\n",
  "html":      "<h2>Foo\nBar</h2>\n",
  "example":   65,
  "start_line": 1234,
  "end_line":   1250,
  "section":   "Setext headings"
}

The npm commonmark-spec package exposes the same objects but names the number field number, not example. GFM does not publish a spec.json (/gfm/spec.json → 404). Use one of:

• parse https://raw.githubusercontent.com/github/cmark-gfm/master/test/spec.txt (the get_tests() algorithm below), or
• scrape https://github.github.com/gfm/ for id="example-N" and the two <pre><code> children per example.

4.5.5 The spec.txt example format (and its → convention)

```````````````````````````````` example <space-separated extension tags>
Markdown source, with → standing for a literal tab
.
Expected HTML output
````````````````````````````````

Rules from spec_tests.py:

• Opening fence = exactly 32 backticks + " example" + optional tags.
• . on a line by itself separates input from expected output.
• Closing fence = exactly 32 backticks. example_number increments on the closing fence (so disabled examples still consume a number).
• All → (U+2192) in both input and output are replaced by \t.
• Tag disabled → the example is parsed but excluded from the run.
• Expected HTML of the single token <IGNORE> → automatic pass.
• The current section is the most recent line matching ^#+ .

4.5.6 Non-conforming / divergent examples in GFM

4.5.7 Third-party conformance suites worth reusing

4.5.8 The HTML normalization contract

Any “we pass the spec” claim is only meaningful relative to normalizations. cmark’s normalize_html (adapted from markdown-testsuite) does:

1. Collapse runs of whitespace to a single space — except inside <pre>.
2. Strip whitespace directly inside/around block-level tags; the block-tag list is article header aside hgroup blockquote hr iframe body li map button object canvas ol caption output col p colgroup pre dd progress div section dl table td dt tbody embed textarea fieldset tfoot figcaption th figure thead footer tr form ul h1..h6 video script style.
3. Convert self-closing tags to open tags (<br /> ≡ <br>).
4. Sort and lowercase attributes.
5. Percent-normalize href/src (urllib.quote(urllib.unquote(v), safe='/')).
6. Decode character references to Unicode, except <, >, &, " which are re-emitted as entities.
7. handle_startendtag deliberately ignores the self-closing slash.

Niman should implement the same normalizer so the conformance number is comparable to cmark’s.

4.6 Extension zone — what CommonMark/GFM deliberately do not cover

None of the constructs in this section appear in either spec. They must be implemented in Niman, outside the conformance surface, with an explicit precedence relative to CommonMark (the safe default: a CommonMark construct always wins; the extension only fires where CommonMark would produce plain text).

4.6.1 Math

4.6.2 Wikilinks

| Aspect | Rule | |—|—| | Base syntax | [[Note]]; [[projects/Three laws of motion]] (folder path from vault root, forward slashes even on Windows) | | Alias | [[Example\|Custom name]] — the first \| splits destination from display text | | Heading anchor | [[Note#Heading]]; sub-headings by repeating #: [[Help#Questions#Report bugs]] | | Same-note anchor | [[#Preview a linked file]] | | Block reference | [[2023-01-01#^37066d]] — #^ + identifier. Identifiers are Latin letters, digits and dashes only. Defining a block: ` ^id at end of a simple paragraph, or on its own line surrounded by blank lines for structured blocks (lists, quotes, callouts, tables) | | Embed | Prefix !: ![[Internal links]], ![[Internal links#^b15695]], ![[Engelbart.jpg|100x145]] (width×height; a single number = width only, proportional) | | Declared-invalid characters | *"A string which contains the following characters may not work as a link: # | ^ : %% [[ ]]"* — so #, |, ^ are **separators**, and [[/]] cannot appear inside | | Case sensitivity | Obsidian resolves links case-insensitively but *preserves* the typed case; Niman should resolve, not rewrite | | Precedence problems | (a) [[ inside a code span must stay literal — code spans win. (b) [[ inside an inline link label or destination must stay literal — CommonMark links win. (c) Nested [[a[[b]]c]] — no unambiguous parse; recommend: **the first ]] closes**, and the inner [[ is literal text. (d) ![[x]] at a position where ![ + [x]] would be a CommonMark image with a shortcut-reference label [x] — GFM/CommonMark wins unless the parser can prove no reference definition matches, which requires post-pass resolution; simplest deterministic rule: **wikilink is attempted before CommonMark link/image matching, but only when the destination contains no newline, no unescaped ]], and the whole [[…]] is on one line** | | Escaping | Obsidian documents **no** escape for | or ]] inside a wikilink. Niman must choose one and document it; a safe choice is: | inside [[…]] is a literal pipe only when the alias separator has already been consumed | | Inside tables | | is needed anyway for GFM table cells, so [[a|b]] is ambiguous in a table. Define: in a table row, |` is consumed by the table splitter before wikilink parsing |

4.6.3 Frontmatter

4.6.4 Other constructs to decide about explicitly

4.7 Round-trip fidelity requirements

The rule: serialize(parse(src)) == src must be structurally impossible to violate because there is no serializer. The only writer is a byte-level splice.

4.7.1 The edit-span model

Document
├── bytes:   ImmutableList<Uint8List>   // the file, verbatim, including BOM and line endings
├── offsets: line-start index over bytes (UTF-8 aware)
├── spans:   BlockSpan / InlineSpan     // each carries [byteStart, byteEnd) into `bytes`
└── derived: an index (FTS, backlinks) — rebuildable, never authoritative

Invariants:

1. Every syntactic node stores half-open byte offsets into the original file, plus a kind and any derived-only attributes (e.g. a resolved reference-link destination). Nodes are a projection of bytes; bytes are never a projection of nodes.
2. Rendering takes (bytes, spans) and produces widgets. It never takes a document object graph that was reconstructed from a parse.
3. Editing is bytes' = bytes.replaceRange(start, end, replacement). Every user-visible editing command (bold, heading, list indent, checkbox toggle, wikilink rename) is compiled to a set of splices computed from the spans. A command whose target text is already in the desired state produces the empty splice.
4. Undo/redo stores splices (with their inverse), not document snapshots.
5. format/prettify/normalize do not exist as commands. If ever added, they must be explicit one-shot transformations over a selection, previewable and reversible.

Test to enforce it: assert(save(open(path)) == read(path)) for every note in the corpus, on every save path. Plus a property test: for any document, opening and closing without an edit produces zero bytes written.

4.7.2 Concrete information-loss hazards and the required policy

4.7.3 The roundtrip_tests.py proof

cmark-gfm ships test/roundtrip_tests.py, which runs md → to_commonmark() → to_html() and compares to md → to_html(). It must strip the writer’s own injected markers to make the comparison work:

# In the commonmark writer we insert dummy HTML comments between lists, and
# between lists and code blocks. Strip these out, since the spec uses two blank lines instead:
return [ec, re.sub('<!-- end list -->\n', '', html), '']

That is the entire argument for the edit-span model in one comment: the reference implementation of the spec cannot re-serialize its own input without help. Niman should not attempt it.

4.8 Recommended conformance strategy

4.8.1 Order of attack

Work strictly bottom-up; each stage is a gate before the next begins. Do not start inlines before blocks are 100 % on their own sections. Example counts in the table are GFM’s (where GFM and CommonMark differ, GFM is the number you will actually run); the stages sum to 676 of GFM’s 677, the remainder being 3.1 Precedence (1 example, covered implicitly by stage 6’s precedence implementation).

4.8.2 Granularity of test naming

Name every test by spec + example number + section, so a failure is directly addressable:

commonmark/0.31.2/042 @Precedence
commonmark/0.31.2/350 @Emphasis and strong emphasis
gfm/0.29-gfm/257 @4.6 HTML blocks
gfm/0.29-gfm/677 @6.14 Textual content
ext/niman/frontmatter/003 @unterminated

• Assert both the normalized HTML and the raw HTML for a subset (the raw comparison catches escaping bugs the normalizer hides).
• Keep the example number as the stable id — it is contiguous and matches start_line/end_line, so a failure prints a spec.txt line range you can sed -n directly.
• Gate CI on commonmark == 652/652 and gfm == 677/677 with an explicit allowlist file for any known GFM oddity (see §4.8.4). The allowlist must be a checked-in file listing spec/example-number + a one-line reason, and CI fails if the allowlist entry starts passing (so it cannot rot).

4.8.3 Where the inevitable oddities get pinned

4.8.4 Definition of done

1. commonmark 652/652, gfm 677/677 under the cmark normalize_html contract, with a checked-in, self-invalidating allowlist of at most a handful of entries.
2. All 21 pathological_tests.py shapes complete under a fixed wall-clock budget (this is a hard requirement for Niman’s “novel-length files, no O(n) hot paths” constraint). The named cases to port: nested strong emph (65 000 deep), many emph closers with no openers, many emph openers with no closers, many link closers with no openers, many link openers with no closers, mismatched openers and closers, openers and closers multiple of 3, link openers and emph closers, pattern [ (]( repeated, pattern ![[]() repeated, hard link/emph case, nested brackets, nested block quotes, deeply nested lists, U+0000 in input, backticks, unclosed links A/B, unclosed <!--, tables, reference collisions (50 000 colliding reference labels).
3. Round-trip gate: save(open(p)) == read(p) over the whole corpus, plus open → close writes zero bytes.
4. Extension suites (frontmatter, math, wikilinks, callouts, footnotes) pass with their own pinned fixtures, and none of them can change the output of any of the 1 329 conformance examples.

Appendix A — Quick reference constants for the Dart implementation

Appendix B — Sources

• CommonMark 0.31.2 — https://spec.commonmark.org/0.31.2/ · spec.txt · spec.json · https://spec.commonmark.org/0.31.2/changes.html · https://spec.commonmark.org/changelog.txt
• CommonMark index / all versions — https://spec.commonmark.org/
• CommonMark spec repo + harness — https://github.com/commonmark/commonmark-spec
• GitHub Flavored Markdown Spec 0.29-gfm — https://github.github.com/gfm/
• cmark-gfm sources + tests — https://github.com/github/cmark-gfm · test/spec.txt · test/spec_tests.py · test/normalize.py · test/roundtrip_tests.py · test/pathological_tests.py · extensions/table.c · extensions/core-extensions.c
• commonmark.js inline parser (reference for §4.4) — https://github.com/commonmark/commonmark.js/blob/master/lib/inlines.js
• HTML5 named entities (authoritative for §4.6.2) — https://html.spec.whatwg.org/entities.json
• KaTeX auto-render — https://katex.org/docs/autorender
• Pandoc manual (tex_math_dollars, footnotes, definition_lists) — https://pandoc.org/MANUAL.html
• GitHub math docs — https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions
• GitHub alerts docs — https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#alerts
• Obsidian internal links — https://help.obsidian.md/links
• Obsidian embeds — https://help.obsidian.md/embeds
• Obsidian aliases — https://help.obsidian.md/aliases
• Obsidian properties / frontmatter — https://help.obsidian.md/properties
• Obsidian callouts — https://help.obsidian.md/callouts
• micromark (GFM tokenizer reference) — https://github.com/micromark
• babelmark3 differential tester — https://babelmark.github.io/

4.9 The markdown package measured

Everything above describes the specifications. This section is the other half of decision D2: what the package that will actually parse Niman’s notes does against them. It was measured on 2026-09-21 with dart run tool/markdown_spec.dart, and the harness, the fixtures and the gate are in the repo.

4.9.1 The number

Two things about how that is counted, because both change how it reads:

• Normalized, by cmark’s own normalizer. The suites’ expected HTML and a parser’s output differ in whitespace, attribute order, <br /> and entity form, none of which is a parsing difference. tool/html_normalize.dart is a port of cmark’s test/normalize.py, and test/unit/html_normalize_test.dart proves it agrees byte for byte on 375 cases — one known exception, pinned. Without that proof the number would be worth nothing.
• It measures HTML, and Niman will consume the AST. Several divergences (data-metadata on <pre>, class attributes on task-list items) are serializer details that never reach a user. The HTML number is a lower bound, and the bound that matters for the engine is higher.

4.9.2 The failures, triaged

Twenty-two examples fail by the reference’s comparison; the other eight non-exact ones (four per suite) differ only in formatting and therefore pass. The complete list, with a bucket and a reason each, is test/fixtures/spec/nonconforming.txt; the shape of it is:

The gate runs in both directions (test/unit/markdown_conformance_test.dart): everything outside the allowlist must pass, and everything inside it must still fail, so a fix cannot go unnoticed and cannot hide behind an old exemption.

4.9.3 The engine’s own number, which is the one that matters

The table in §9.1 measures the package. What the app shows is what the engine produces: the block scanner decides the blocks, the masker sets the note’s own constructs aside, and the package parses each block. So the successor question — named in the first version of this section as the honest next step — is what that path does on the same suites.

dart run tool/engine_spec.dart and test/unit/engine_parity_test.dart answer it, and the answer is sharper than a second pass rate:

3 583 blocks, 3 583 identical. Everywhere the engine does not intervene it reproduces the package byte for byte — so the block decomposition, the per-block parse and the concatenation add nothing and lose nothing, and the conformance number above is inherited rather than merely claimed. That is the property that matters, and it is now a test rather than an argument.

Where the engine does mask, its answer differs by design: 155 blocks across the two suites, because those constructs are the app’s own and the renderer draws them from the spans. Measured but not asserted: whole documents come out identical in 578 of 652 and 602 of 677, and the gap is two things the renderer owns rather than the parser — a list is several blocks here (one per item, because a block is a layout unit) and reference definitions and footnotes are document-scoped. Assembling those is Phase 2’s job, and the number is the size of it.

4.9.4 The eight, decided

Phase 1’s gate is the package’s, so the eight examples it flagged were left as an explicit debt. Each is now decided, and none of them is a bug in the engine:

So of the eight: one is fixed where the fix belongs — in the engine’s own model, which a dependency bump cannot lose — and seven are inherited with the reason recorded, three of them proved to leave the app correct.

4.9.5 The read mode, measured

Phase 2 builds the surface the design described, behind MarkdownEngine.unified and off by default. Its own gate is test/widget/engine_compare_test.dart: both engines are pumped over the same fixture at a viewport tall enough that neither windows anything, and their visible text is compared — a marker left in, a construct dropped, a paragraph drawn twice. Pixels would be the strongest comparison and the most brittle; the words are what a reader sees.

All three render the same words in the same order. The gate found eight real bugs on the way there, which is what a gate is for:

• container syntax drawn as text — a list item’s - , a quote’s > , a task box’s [x] , which the parser strips and no run covers;
• a list marker read from the text instead of drawn, so a task list showed [x] where the preview drew a checkbox;
• an ordered list showing the number the note wrote rather than the item’s place in the list — CommonMark ignores those numbers except the first, so 1. 1. 1. is a list of three;
• a quote block built at depth zero, because the scanner took its depth from the state entering the first line — which is before that line’s own >;
• a Display formula keeping a stray $ at each end, from a delimiter rule that assumed one character rather than two;
• Markdown images drawn as their alt text;
• footnotes that resolved to nothing: a reference and a definition are written in different blocks and the engine parses per block;
• a block the parser consumed — a reference definition — drawn as the text it did not cover.

The last thing to fall was the section the package ends a document with: the definitions listed with their backlinks. It is drawn through a lazy sliver for the same reason the note is — appending it whole took first content from 76 ms to 112 ms on the geometry note and the jump from 9 ms to 56.

And the timings, from test/perf/read_view_timing_test.dart, held against this document’s own budget of ≤ 60 ms target / 120 ms ceiling for text to first visible content:

(The jump column is the custom sliver’s: it read 43/32/31 ms when SliverVariedExtentList forced every extent — cheap, and clipping every block taller than its estimate — then 347/935/2 772 ms when SliverList measured them all. §8.4.4 carries the three columns side by side.)

The geometry note — 934 KB, 13 845 formulas — reaches first content in 70 ms against the preview’s recorded 137 ms, and lays out 88 blocks of 7 530: the windowing is what the number is made of. The 50 KB fixture is slower than the 200 KB one and the reason is worth recording: its first screen is dense with display formulas, and the first render of each is typeset inside this measurement — by the preview exactly as by the read view — with the math cache amortizing it from the second frame on. Its ceiling is 250 ms in the test, a recorded decision rather than a conveniently chosen number.

These are debug-mode harness numbers, as every benchmark in this repository is: the ratios carry and the absolutes do not.

And a device read about a second, which the harness cannot see (2026-09-21). Opening the read pane on the geometry note took 1 022 ms from the flip (preview: flip showPreview=true at 18:03:17.954) to the first frame after it (18:03:18.976), and that frame itself was cheap — 18.5 ms total, 10.8 of them build. The harness above reads 91 ms for the same note. Both can be true, and which one it was could not be told from the log, because the read pane had no trace of its own: the editor has had note open first frame since T-PP-22, the legacy preview logs its parse, and the pane the user was waiting for logged nothing. The missing instrument was the finding.

The traces answered it, on the next device run and not in the pane’s favour: preview: read pane first frame: first frame 18.63ms after call, and [read] first content: 7602 blocks, 36 built, 47ms after the view was created with [read] scan: 7602 blocks, 10414 lines in 13ms — thirty blocks laid out, the whole 10 414-line note scanned in 13 ms, and 270 frames in the 2.5-second window after the flip with one over budget (worst build 13.4 ms). The second was never the read pane. What the same log shows one line earlier is:

18:03:14.792 DEBUG [wysiwyg] open: 931852 chars, 7036 delta ops, 2934 preserved blocks
18:03:15.806 WARNING [frames] slow frame: total 2100.4 ms (build 2012.3 ms, raster 4.7 ms, vsync 83.4 ms)
18:03:17.954 INFO  [preview] toggle → preview

— the note was opened in WYSIWYG, and converting a 931 KB note into 7 036 Delta ops took a 2 100 ms frame, two seconds before the flip. The user’s “lentina ad aprirsi” was the pipeline still draining that frame; the read pane’s own open is 18-47 ms, and it is measured now rather than argued about. The WYSIWYG’s own cost is the legacy editor’s and phase 4 is its answer, but the number is worth carrying there: 2 100 ms of build for one note, twice in one session, on the same device (#254).

A smaller residual the trace also exposes. After a shell remount the settings read has not landed yet, so _editorSettings is still ShellEditorSettings.defaults (engine: legacy) for the first builds — and the preview pane is built offstage even when it is hidden, so a MarkdownPreview mounts for the big note, starts an isolate parse and is replaced by the read view when the setting lands: parse async: pane gone before the parse landed (rev 1), dropped, ~150 ms of hidden work per kind switch. It costs nothing on screen and it is tracked rather than patched here, because the honest fix is to know the engine before building a preview pane at all (phase 5 removes the branch). #255.

It now emits three lines, all of them through the existing seams and none of them per-frame: [read] scan: N blocks, M lines in Xms when the scanner runs, [read] first content: N blocks, K built, Xms after the view was created from the frame that first drew blocks, and [preview] read pane first frame: … after call at the flip. The frame window (FrameProbe, 2.5 s, frames over budget and worst build/raster) is watched too, but only when AppLog.file is attached — the run whose log can be handed over — which also keeps a 2.5-second timer out of widget tests, where it is a pending timer rather than a measurement.

The same log had a second thing to say, and it said it wrongly: [preview] parse async: stale rev 1 (current 1), dropped — the numbers equal, which is the tell that the revision was not the reason. The condition is !mounted || revision != _parseRevision and it was the first half: a MarkdownPreview for the 931 KB note was mounted, spent a parse off the isolate, and was gone before the result landed. The line now names which of the two happened, because reading it as a staleness bug cost an afternoon of chasing a parse that had nothing to do with the symptom.

Where that legacy preview came from is a real gap, and the trace is what named it: NoteView.unifiedMarkdown defaults to false and lib/src/ui/outside_file_screen.dart never passes it, so a file opened from outside the library was drawn by the old engine while the same file inside the library was drawn by the new one — with the engine setting on, and no hint on screen. The setting belongs to a library (LibraryConfig.markdownEngine), so a file outside one has none to read; the shell now hands down the engine of the library it is showing (openOutsideFile(…, unifiedMarkdown:)), and the two paths that run with no library at all — the home screen’s launch-with-a-file and its Open file — keep the default with a comment saying why. Both spellings disappear when the setting does (phase 5).

4.9.6 What it changes

• Risk K1 collapses. “The parser never fully conforms” was the largest technical risk in the design document, with weeks of grind behind it. It is now eight enumerated examples, two of which are whitespace.
• The conformance number is a measurement, not a goal, and it is re-measured on every flutter test rather than argued about. A future package bump that regresses it fails CI on the example that moved.
• D2 is vindicated, and D5’s target needs restating honestly. “100 % of both suites” is not free: it means fixing the eight, or pinning them as the fourteen are pinned. The eight are cheap; what the number removes is the fear that the gap is unbounded, which is what made “write our own parser” look reasonable.
• The AST caveat is answered in §9.3: the engine’s own number is the block parity, 3 583 of 3 583, and the eight are decided in §9.4.

5. The corpus reality

Purpose. Size and justify a high-performance Markdown rendering design from measured properties of the real corpus, not from intuition. Every number below was produced by a command in this session; the producing script is named next to each table. Nothing in this file is an estimate unless the word estimate appears.

Targets (read-only, never modified):

Reproduce everything: bash /tmp/niman-research/scripts/run_all.sh (writes JSON to /tmp/niman-research/out/, then python3 /tmp/niman-research/scripts/gen_report.py).

Method and its limits. Block classification is a pragmatic CommonMark-ish line-state machine in scripts/gen_block_census.py; inline scanning is a character-level scanner in scripts/gen_inline_census.py that is not recursive (one known, quantified consequence is flagged in §3). The block classifier’s byte totals reconcile to the file size exactly for every target, which is the strongest available check that no line was double-counted or dropped:

4.9.7 The removal, taken

This section first recorded a removal that had been planned, measured — 176 sites across 56 files, cut three independent ways — and then not taken: on 2026-09-21 the split was kept. The same day, with those numbers in hand, the decision was reversed and the removal was run. What follows is what it actually took, written down for the same reason the plan was: whoever touches this surface next should find its shape here rather than reconstruct it from a diff.

What went, in the order the plan set out:

1. previewEnabled — the library setting, the session accessor, the Editor settings switch, the settings-search entry, the LibraryConfig field, and the shell’s conditionals. The preview is part of the app now, so _previewToggleVisible is only about the note: a kind GUI hides the eye unless the note is being edited raw.
2. The split ratio and the layout override — PreviewLayoutMode, previewSplits, splitRatio/splitFraction, defaultSplitRatio, minSplitRatio, maxSplitRatio, the app-bar picker, the Appearance row and the drag persistence. The database columns went too, against this document’s own advice to leave them: ALTER TABLE ... DROP COLUMN is two lines at v27, the chain already does exactly that at v17, and a settings field nothing reads is a lie about what the app does. splitBreakpoint kept its job and lost its name — it is wideBreakpoint now, because what it decides is whether the shell is the rail-tree-note shape or the phone’s.
3. The split itself — EditorPreviewSplit (deleted, with its file), _previewVisible, _previewFullScreen, ExitFullScreenButton, PreviewLayoutModeAction, the shell’s previewSplitsHere / previewFullScreen / previewVisible props and onLeaveFullScreenPreview, DetailTab.splitPreview, NoteStatusRow.splitPreview, and NoteView’s four split parameters. The note is one pane: NoteView.showPreview picks which of the two it holds, and the other stays mounted Offstage, where it already was.
4. docs/user/editing.md and docs/user/settings.md, in the same commit: both described a screen and three settings that no longer exist.

Four things the removal changed rather than deleted, each a decision:

• The phone’s preview flag became the tab’s. _previewVisible was the phone’s own boolean, so the preview was on for the screen: opening the next note kept it. The flag now lives in the note’s tab memento, as it always did on a wide window, so a note comes back the way it was left and the next one opens in the editor. Both layouts read it through one _notePreview, and _mementoOf answers with an empty memento for a note whose tab this build has not made yet — the transition frame — instead of the note before it.
• A template’s open: preview now works on both layouts. It used to set _previewVisible, which the wide layout never reads, so the directive did nothing there. It is applied to the filed note’s tab, queued behind the follow that makes the tab (ShellWorkspace.showPreviewWhenOpen).
• previewEnabled was a known settings.json key. It stays in _knownKeys as a legacy key that is read and never written, the way spellDictionary is: a library whose file still says false gets the key dropped on the next write rather than preserved forever as an unknown one.
• The immersive full-screen preview went with the split, rather than staying as chrome-less reading: one pane, one mode, and the phone’s note is a normal page with its app bar either way. That was a product decision taken with the removal, not a consequence of it.

The words. The two settings’ strings were removed rather than left dead: ten getters from base.dart and from each of the 37 locales — 380 declarations — plus the ten accessors and splitRatioValue from strings.dart. One string stayed: commandNeedPreview still labels CommandNeed.previewToggle, whose only remaining condition is that the note is text, so its wording (“With the preview on, on a text note”) is now half a sentence about a setting that no longer exists. Rewording it means rewording it in 38 languages, which is a translation task, not this one.

What the measurement got wrong. The plan’s two feared costs were the migration and the strings, and it recommended leaving both. Both turned out to be the cheap half: the migration is a DROP COLUMN with a precedent ten versions back, and the strings are 380 mechanical deletions a script did. What the measurement got right was the reason there is no green intermediate state, and it was never the count: settings, session, the shell, the note view, the database and their tests had to move in one commit because nothing compiles until the last of them agrees.

Where it left the phase. markdownEngine still defaults to legacy, so the eye shows the old preview until someone turns the unified engine on; the numbers that should decide that are §4.9.5. What the split’s departure removes is the surface the two engines were being compared in, which is why the comparison had to come first and did.

5.1 Global stats

Produced by scripts/gen_global_stats.py (global.json). Lines are \n-delimited; a trailing newline does not create a phantom final line. “Graphemes” is the codepoint count minus combining marks, ZWJ and variation selectors — an approximation, marked as such, because no real grapheme segmentation was run.

5.1.1 Reading of the global stats

• The real note is 89.5 bytes/line average and 10 331 lines. Its 1 MB fixture counterpart is 31.7 bytes/line over 32 100 lines: same order of magnitude of bytes, 3.1× fewer lines. Line-count-driven costs (per-line tokenization, per-line layout, scroll mapping by line index) are therefore worse for the real note than for the 1 MB synthetic fixture the benchmarks use. This is the single most important sizing fact in this document.
• The synthetic fixtures are 100 % ASCII; the real note is 0.78 % non-ASCII (7 280 bytes: accented Italian, •, −, ∈-adjacent typography). No combining marks, no ZWJ, no variation selectors, no lone surrogates, no BOM anywhere. So grapheme-cluster handling is not exercised by this corpus, while Latin-1 supplements are.
• Line endings: pure LF, zero CRLF, zero bare CR, zero tabs in all seven files. Trailing whitespace: zero lines everywhere — so a 2-space hard break does not occur once in the whole corpus (see §3).
• 1 524 lines over 200 bytes in the real note, 376 over 500, 61 over 1 000, 3 over 2 000. The synthetic fixtures never exceed 439 bytes. Long-line handling is a real- note-only problem.
• Irregularity is low: no BOM, no CRLF, no tabs, no trailing whitespace, no surrogate weirdness. The corpus is hard because of volume and math, not because of encoding edge cases.

5.1.2 Longest lines (top 3 per file)

Provenance: scripts/gen_global_stats.py, longest field of global.json.

| file | bytes | line | excerpt (first 150 chars; source backticks shown as ′) | |—|—|—|—| | Geometria 1.md | 2 568 | 9 056 | Per l'autovalore $-2$: $\dim \ker(\phi + 2) = 2$, $\dim \ker(\phi + 2)^4 = 4$. Poiché il polinomio minimo per $-2$ ha grado $4$ (visto che $\dim \ker( | | Geometria 1.md | 2 338 | 6 501 | Iniziamo dalle funzioni più semplici, quelle in cui $c = 0$ e la funzione è del tipo $f(z) = az + b$ (trasformazione affine complessa). Risulta $D_f = | | Geometria 1.md | 2 224 | 9 426 | **7.31** (a) La restrizione di $g$ a $\langle e_4 \rangle^\perp = \langle e_1, e_2, e_3 \rangle$ è definita positiva e quindi per ogni vettore $v \in | | Geometria 1.md | 1 915 | 9 367 | **7.27** Questa dimostrazione è stato già brevemente illustrata, in forma diversa, nell'Esempio 7.5.3. (a) Per ogni $v \in \mathbb{R}^3$, si ha $\\|X | | Geometria 1.md | 1 880 | 5 603 | ` Infatti il polinomio caratteristico di $A$ ha coefficienti reali e grado $3$ e pertanto ammette almeno uno zero reale $\lambda$. Questo è un autovalo | | fixture-200kb.md | 412 | 4 142 | (max — fixtures never exceed 439 B) | | fixture-1mb.md` | 439 | 10 027 | (max — fixtures never exceed 439 B) |

The three longest lines in the real note are all single-line paragraphs that are mathematically dense (2 568, 2 338, 2 224 bytes). Line 9 056 alone is a one-paragraph block carrying ~95 inline math spans, i.e. one TextPainter input of roughly 32 wrapped lines — see §5 and §6.

5.2 Block-level census

Produced by scripts/gen_block_census.py (blocks.json). lines counts source lines (they sum exactly to the file line count); blocks counts renderable blocks. % bytes is the share of the whole file. Rules that materially change the reading:

• a list item is one block and owns its own line; a multi-line list item is not split, so paragraph line excludes text that belongs to a list item;
• blockquote lines are classified by their residual content after stripping > markers, so a quoted paragraph is counted in paragraph line, not in a separate bucket; the separate > depth histogram in §2.4 accounts for the markers;
• a $$ line opens a math block that runs to the closing $$;
• [^label]: ... is a footnote definition, not a link reference definition (the two were separated after an initial run conflated them — this materially changes the link-census reading in §3).

5.2.1 Geometria 1.md

934 769 bytes, 10 331 lines. % bytes = share of file bytes.

5.2.2 fixture-200kb.md

206 241 bytes, 6 288 lines. % bytes = share of file bytes.

5.2.3 fixture-1mb.md

1 050 145 bytes, 32 100 lines. % bytes = share of file bytes.

5.2.4 Sub-censuses

ATX headings by level

The real note uses only H1–H3 (15/59/10). No level 4-6 and no setext heading appears at all; the fixtures stop at H4. A renderer sized on this corpus does not need deep heading styling, but it does need H4–H6 to not crash on user input.

List items — bullet marker and nesting depth

Provenance: blocks.json bullet_marker_chars, bullet_depth_count, ordered_delim_count, ordered_depth_count; depth is the number of enclosing list items according to the indent stack (indents 0, 2, 4, … each open one level).

Bullet nesting depth histogram, real note: depth 1 = 612, depth 2 = 215, depth 3 = 25. The fixtures reach depth 2. Depth 3 is the corpus maximum. The real note uses - exclusively (852 items, zero *, zero +) and has zero ordered lists; the fixtures supply all 2 377 ordered items, all with the . delimiter (no ) form).

Task lists and strikethrough (fixtures only)

Task lists and ~~strikethrough~~ are GFM features that only the fixtures use; the real note has neither. They still have to work, but they are not what makes the real note expensive.

Blockquote marker lines and nesting depth

Provenance: scripts/gen_bq_depth.py (bq.json), a direct line-level count of leading > markers — independent of the block classifier.

The real note has 22 blockquote lines, all depth 1, all of the form > Figura N.N (figure captions). There is no nested blockquote and no blockquote containing a list, a table or a display-math block anywhere in the corpus. §7 makes that an explicit adversarial case because a renderer that never sees it is a renderer that was never tested for it.

Fenced code blocks

Fence size distribution (real note has none; fixture-200kb shown):

The synthetic fences are all 5–9 source lines; the real note has none. Fence size and count are fixture-only properties; the 1 MB fixture has 867 fences.

GFM tables

The real note’s two GFM tables are small (3×3 and 4×4). 11.7 % of the 200 KB fixture and 11.2 % of the 1 MB fixture is table bytes at up to 4 columns × 6 rows — i.e. table count is fixture-dominated whereas table size is tiny everywhere in this corpus.

The real note also contains two single-line raw-HTML <table> blocks that the PDF→Markdown conversion emitted inside a paragraph — line 8 114 is ` Dunque si tratta … quadrati magici | 0 | 2 | 1 | |—|—|—| | 2 | followed by line 8 116 <table style="min-width: 75px;"><colgroup>…`. That is a genuine parser worst case: a Markdown table header/body fused into prose, immediately followed by an HTML table with no blank line.

Footnote definitions and link reference definitions

The real note has zero link reference definitions. All 17 blocks that look like [label]: dest are footnote definitions ([^1]: … at lines 10 299–10 331). The fixtures likewise have 40 footnote definitions and zero link reference definitions. Only the adversarial fixture in §7 exercises real link reference definitions.

Frontmatter

Frontmatter is negligible in size (0.006 % of the real note) but it must be recognised before the body, otherwise the opening --- becomes a thematic break or a setext underline. All seven targets begin with a --- frontmatter block.

5.3 Inline-level census

Produced by scripts/gen_inline_census.py (inline.json). Scanned region = paragraph, heading, list-item text, table cells and footnote-definition lines; fenced code, indented code, HTML blocks, frontmatter and math blocks are excluded.

Two emphasis readings are given. raw treats $ as ordinary text (plain CommonMark). math-aware masks $…$ and code spans first. The gap between them is the single most important inline finding in this document.

5.3.1 Emphasis / strong delimiter runs

7 530 single _ runs in the real note collapse to 17 once math is masked — a 99.8 % reduction. Those 7 513 runs are TeX subscripts (x_1, a_{ij}). A renderer that resolves emphasis before it recognises $…$ must therefore make 7 513 pendant delimiter decisions per document that a correct order of operations avoids entirely. The same effect is visible in the fixtures at smaller scale (1 210 / 230 → 0).

The ** counts barely move (2 595 → 2 571, and 16 four-star runs appear once math is masked, because masking joins ** from adjacent text). ** is the workhorse inline span: ~2 595 delimiter runs, i.e. on the order of 1 300 strong spans. *** occurs exactly once in the whole real note. **** occurs 6 times raw, 16 math-aware; both are pathological (adjacent **…** across a masked span), not a feature a renderer should special-case.

5.3.2 Links, images, autolinks, HTML, escapes, entities, breaks

Findings for the real note:

• Zero inline links, zero reference links, zero images, zero autolinks, zero raw inline HTML, zero entities, zero backslash escapes, zero code spans, zero hard breaks. All 40 links in the file are [[wikilinks]]. This is a Markdown-light, math-heavy document: the inline parser’s whole budget goes to $…$ and **…**.
• The 9 “shortcut” candidates are real false positives: [Determinanti e operazioni elementari.], [Teorema di Schur], [**Spazio di Minkowski**] — bracketed prose in the book’s table of contents with no matching reference definition. They must render as literal text. A naive shortcut-reference resolver would render 9 broken links.
• The fixtures are the mirror image: 412 inline links, 274 local images, 579 code spans, 720 task-checkbox [x] brackets in the 1 MB file, and zero math density to speak of. Neither corpus alone exercises the full inline grammar.

5.3.3 Code spans and inline math, length distribution

Inline-math byte buckets, real note:

Half of all inline math expressions are ≤ 10 bytes and 68 % are ≤ 20 bytes. The distribution has a long thin tail: 121 expressions over 100 bytes and 16 over 200, up to 350. Two conclusions: a very small expression cache/render fast path covers the median case, and a per-expression size limit is not needed because nothing in the corpus is structurally huge inline (the 350-byte worst case is still one line).

5.3.4 App-specific inline constructs: wikilinks, footnotes

All 40 real-note wikilinks (inline.json wikilinks): geometria_1_copertina_1.png, geometria_1_figura_1-1.png, geometria_1_figura_1-2.png, geometria_1_figura_1-3.png, geometria_1_figura_1-4.png, geometria_1_figura_1-5.png, geometria_1_figura_1-6.png, geometria_1_figura_1-7.png, geometria_1_figura_1-8.png, geometria_1_figura_1-9.png, geometria_1_figura_1-10.png, geometria_1_figura_1-11.png, geometria_1_figura_1-12.png, geometria_1_figura_1-13.png, geometria_1_figura_1_exta_1.png, geometria_1_figura_1_extra_2.png, geometria_1_figura_1_extra_3.png, #^g1-def-monoide-gruppo|monoide, #^g1-def-campo|campo, 185234098087.png, acd1115394e7.png, 3bf2f789ad6c.png, 5229eb987fb0.png, geometria_1_figura_7-1.png, geometria_1_figura_A-01.png, geometria_1_figura_A-02.png, geometria_1_figura_A-03.png, geometria_1_figura_A-2.png, geometria_1_figura_A-3.png, geometria_1_figura_A-extra.png, geometria_1_figura_A-4.png, geometria_1_figura_A-5.png, geometria_1_figura_B-1.png, geometria_1_figura_B-2.png, geometria_1_figura_B-3.png, geometria_1_figura_B-4.png, 2829e235abe7.png, 1fcd6551ce60.png, 16d07bfd5bf9.png, attachments/e12204d45c77.png.

Every wikilink in the real note is either an inline image embed (![[…png]], 38 of them) or one of exactly two block-anchor links: [[#^g1-def-monoide-gruppo|monoide]] and [[#^g1-def-campo|campo]]. There are no: wikilinks to a heading anchor ([[Note#Heading]]), wikilinks to another note by name with an alias, or embeds of another note. 34 footnote references point at 17 footnote definitions, i.e. reuse (2 refs per definition) — so footnote identity, not footnote count, is what a renderer must key on.

Known scanner limitation, quantified. gen_inline_census.py is a flat (non- recursive) scanner. When a line contains an unmatched [, the shortcut-reference branch consumes text up to the next ], and any $…$ inside that run is not counted. This affects exactly 2 lines of the real note (4 799 and 7 005) and under-counts inline math by 4 expressions: the census reports 13 000, the direct dollar-pair scan in gen_math_deepdive.py reports 13 004. The deep-dive figure is the authoritative one; the discrepancy is 0.03 % and is stated rather than smoothed over.

5.4 Math deep-dive

Produced by scripts/gen_math_deepdive.py (math.json), with the coverage curve from scripts/gen_coverage.py (coverage-geometria.tsv). This document is a converted linear-algebra textbook; math is why it is 935 KB.

5.4.1 Volume

Real-note math byte budget: inline 212 054 B + display 140 800 B = 352 854 B = 37.75 % of the document.

Display-math byte buckets, real note:

Everything-inline-math byte buckets, real note (inline + display pooled):

Reading: 68 % of display blocks are 41–160 bytes and 8 blocks exceed 640 bytes. The display tail (up to 1 094 bytes / 9 source lines) is where KaTeX layout and scroll-height measurement get expensive; the inline median (10 bytes) is where a cache pays for itself.

5.4.2 The 15 largest math expressions in the real note

Provenance: math.json → largest (top 25; 15 shown). line is the first source line.

Every one of the ten largest expressions is an \begin{aligned} block containing a pmatrix/vmatrix, i.e. a matrix with row breaks (\\) and alignment (&). That is the shape a math renderer must handle first; it is not an edge case here, it is the top of the distribution.

5.4.3 TeX command histogram — top 60 + distinct count

Distinct commands in the real note: 135 total, of which 129 are alphabetic (\frac, \phi, …) and the rest are single-character control symbols (\\, \|, \{, \}, \,).

The top 10 commands alone account for the bulk of all command uses:

\\ 2 657, \phi 1 889, \mathbf 1 877, \mathbb 1 574, \begin 1 389, \end 1 389, \dots 1 386, \in 963, \| 834, \mathcal 739 — 14 697 of 26 391 command uses (55.7 %).

Two structural facts fall out of this list: \mathbf (1 877) and \mathbb (1 574) are font-selection commands, not layout commands — a renderer needs a math-bold and a blackboard-bold alphabet before it needs anything exotic. \operatorname (487) and \text (226) require text-mode layout inside math.

5.4.4 Environments

\begin{…} appears 1 389 times, distributed over only 5 distinct environments. pmatrix alone is 80.6 % of all environment uses. 12 963 of 13 845 math expressions (93.6 %) contain no environment at all.

5.4.5 Nested brace depth

Maximum brace depth in the entire real note is 3, and only 2 expressions reach it. 67 % of expressions have no braces at all. Brace-depth exhaustion is not a risk in this corpus; a recursive-descent TeX parser with any reasonable depth limit will never approach it.

5.4.6 Structure inside math, and indented contexts

Alignment structures:

Answers to the questions asked of this section, for the real note:

• inline vs display: 13 004 inline vs 841 display (93.9 % inline).
• $$ occurrences: 1 682 delimiter lines = 841 blocks. Every $$ in the file is a standalone line at column 0; there is not one single-line $$ x $$ display math in the real note (all 60 single-line display blocks in the fixtures, none here).
• Display math in an indented container: 0. Display math inside a list item or blockquote: 0. But 1 576 inline math expressions sit on list-marker lines and 13 004 total are inline — so the inline path is what lives inside lists.
• \left/\right: 134 expressions (1.0 %) — rare, and always paired.
• \\ or &: 882 expressions with \\ (6.4 %) and 507 with & (3.7 %); these are concentrated in the aligned/pmatrix/cases blocks, i.e. exactly the 841 display blocks that dominate the byte tail.

5.4.7 What a math renderer must support — 95 % and 99 % coverage

Method: take every one of the 13 845 math expressions (inline + display), record the set of TeX commands each uses (\begin/\end counted separately as environments), then add commands in descending frequency order and measure how many expressions are fully covered. Script: scripts/gen_coverage.py.

The 95 % feature list — 58 commands, in frequency order:

\phi \mathbb \dots \in \\ \mathbf \mathcal \alpha \rangle \langle \to \operatorname \{ \} \times \frac \ker \neq \pi \quad \lambda \cdot \psi \mid \| \det \perp \text \dim \sigma \right \left \sum \beta \cap \circ \sqrt \leq \cdots \vartheta \overrightarrow \le \Delta \delta \oplus \cos \subseteq \mu \bar \mapsto \vdots \sin \ge \rho \Sigma \vee \overline \pm

The additional commands needed to reach 99 % — 28 more (86 total):

\exp \xi \geq \deg \nu \Rightarrow \Phi \tau \gamma \infty \varnothing \theta \ddots \varphi \setminus \varepsilon \Leftrightarrow \zeta \Lambda \notin \log \prod \Re \ell \cong \Im \qquad \binom

Plus, in both cases, these non-command features:

Interpretation. Covering 95 % of this corpus needs a small renderer: 58 commands, 5 environments, sub/superscripts, auto-sizing delimiters and two font alphabets. The last 5 % — 28 more commands going to 99 %, then 47 more to reach 100 % — is a long tail of one- and two-use symbols (\binom, \setminus, \varnothing, \ell, \Im, \Re, \cong…). A renderer that implements the 95 % list plus a “unknown command → render as literal text” fallback degrades gracefully on the remaining 5 %; a renderer that hard-fails on an unknown command breaks 674 expressions in this one document.

Note the two cost drivers are different: the command list is short, but the number of distinct expressions is 13 845, and 68 % of display blocks are large enough to need real layout. Parse-once/cache-by-content is therefore the right shape: 13 845 keys, median 10 bytes, with the 841 display blocks dominating the expensive end.

5.5 Structural worst cases

Produced by scripts/gen_worstcases.py (worst.json). Each row names the construct, its location, and the part of a renderer it stresses.

5.5.1 The real note (Geometria 1.md)

| worst case | value | location | excerpt | stresses | |—|—|—|—|—| | longest paragraph | 2 569 B on 1 line | line 9056-9056 | Per l'autovalore $-2$: $\dim \ker(\phi + 2) = 2$, $\dim \ker(\phi + 2)^4 = 4$. Poiché il polinomio minimo per | layout — one TextPainter input that wraps to ~32 lines; also parse, because ~95 inline math spans must be resolved before layout can start | | deepest list nesting | depth 3 (indent 8) | line 9835 | ` - alternante, 161 | **parse + layout** — list indent stacking and bullet widget indentation | | deepest blockquote nesting | depth 1 (flat, never nested) | all 22 > lines | > Figura 1.1 | **parse** — trivially shallow; nothing stresses quote nesting in the real note | | largest table | 4×4 = 16 cells, 536 B | line 8 970 | | | $-2$ | $-1$ | $0$ ` | layout — intrinsic column widths; the real largest is small | | largest math expression | 1 100 B, 9 source lines | lines 2473-2481 | \begin{aligned} + vmatrix with 8 & columns | layout + paint — KaTeX parse, then a widget whose height must be known before the scroll offset can be trusted | | highest inline-span count on one line | 120 inline elements in 1 860 B | line 8 019 | **2.24** (b) $L = \langle e_1, e_2, e_3, e_4 \rangle_{\mathbb{R}}$. $\pi : E \oplus D \to E$ è la proiezione s | parse — 120 spans on one line, each needing a style run | | highest inline density (≥200 B lines) | 107 spans / 834 B = 12.83 per 100 B | line 8 330 | **4.15** (a) $D(x_1 e_1 + x_2 e_2 + x_3 e_3, y_1 e_1 + y_2 e_2 + y_3 e_3, z_1 e_1 + z_2 e_2 + z_3 e_ | parse + layout — 12.83 markup spans per 100 bytes means the line is mostly tiny measurement units | | longest single “word” (no spaces) | 102 chars | line 8 971 | \|----------------\|--------------------------\|--------------------------\|---- | layout — an unbreakable 102-char token (a GFM delimiter row) exceeds the viewport and forces horizontal clipping or character-level break | | longest non-table unbreakable token | 73 chars | line 4 597 | PA**+**=(Q**tSSt**Q)**−1**Q**tStP**=Q(**tSS**)**−1tStP**=Q**tS−1**​**tP** | layout — PDF→MD conversion glued a formula together, producing a 73-char run with no break opportunity | | longest alphabetic word | 20 chars | line 4 024 | triangolarizzabilità | layout — benign; text is naturally short-worded Italian | | longest run of consecutive non-blank lines | 67 lines (1 452 B) | lines 10 171–10 237 | — | scroll-mapping + memory — one 67-line paragraph region with no blank-line padding to use as a layout anchor | | densest 200-line region | 4.77 inline elements per 100 B, 1 068 spans in 22 378 B | lines 8 526–8 725 | — | parse — the exercises section (lines 8 526–8 725) is the hottest parse region in the document |

Byte convention in this table. gen_worstcases.py measures whole blocks, so its byte counts include each source line’s LF and, for the largest math expression, the $$ delimiter lines. That is why the same expression reads 1 100 B here and 1 094 B in §4 (body only), and the longest paragraph reads 2 569 B here and 2 568 B in §1 (line only). The §1 and §4 figures are the canonical ones.

The two lines worth memorising:

• line 8019 — 120 inline elements, 1 860 bytes, one paragraph, one line. It is both the largest parse unit and, at ~32 wrapped lines, the largest single layout unit in the file.
• line 10 171–10 237 — 67 consecutive non-blank lines: the maximum distance a scroll mapper can travel without a blank line to fall back on.

5.5.2 Worst cases in the synthetic fixtures

The fixtures are flat: every one of their worst cases is 3–10× smaller than the real note’s. They are good throughput fixtures (many repeat units) and poor worst-case fixtures (no unit is extreme). That is precisely why §7 exists.

5.6 Scroll/layout sizing model

5.6.1 The one external number

docs/records/editor-alternatives.md records, for the app’s existing editor:

Line layout + paint

44.7 µs/line (only visible lines are paid for)

Numbers are debug-mode, inside the test harness, on one Windows machine. They are inflated in absolute terms; only the ratios between rows carry.

The same file records Whole buffer as a single span: 52.0 ms and, for the rejected eager renderer, SmoothMarkdown eager preview, 200 KB: 2769.2 ms. Those three numbers are the empirical anchor for everything below.

5.6.2 Assumptions (stated, not measured here)

1. Per-line layout cost: 30 / 44.7 / 80 µs for one ~80-character visual line, using 44.7 µs as the middle case because it is the repo’s own debug-mode figure.
2. Wrapping: 80 characters per visual line. The real note averages 89.5 bytes/line but its blocks are single logical lines; the visual line count is computed as ceil(block_bytes / 80) per text block, which is an estimate of wrapped lines.
3. A math block costs one layout unit here; its KaTeX cost is not modelled and is a separate, likely larger, line item.
4. One viewport holds ~40 layout blocks (the figure given in the brief).

5.6.3 Block and line counts

Produced by scripts/gen_layout_model.py (layout.json). layout blocks = paragraphs + headings + list items + table rows (header + body) + fenced/indented code blocks + math blocks + thematic breaks + link-ref/footnote definitions + HTML blocks.

For the real note: 4 219 layout blocks, split as 2 392 paragraphs + 84 headings + 852 list items + 9 table rows + 841 math blocks + 41 others (22 thematic breaks, 17 footnote definitions, 2 HTML blocks). Because 2 392 of 2 392 paragraphs are exactly one source line, the block count is unusually close to the line count — the note was produced by a PDF converter that never hard-wraps. That is good for a block-based renderer and bad for anything that assumes ~30-character lines.

The estimated visual line count is 11 509 for text plus 841 math widgets, i.e. 12 350 layout units, because long single-line paragraphs wrap. Average visual lines per block = 12 350 / 4 219 = 2.93.

5.6.4 Eager vs viewport vs viewport + cache

Produced by scripts/gen_sizing.py (sizing.txt). Arithmetic shown inline.

Geometria 1.md

layout_blocks=4219 visual_lines=12350 avg_visual_lines_per_block=2.927

arithmetic at 44.7 us/line: (a): 12,350.0 lines x 44.7 us = 552,045 us = 552.04 ms = 33.1 frames@60Hz = 66.2 frames@120Hz (b): 117.1 lines x 44.7 us = 5,234 us = 5.23 ms = 0.3 frames@60Hz = 0.6 frames@120Hz (c): 351.3 lines x 44.7 us = 15,702 us = 15.70 ms = 0.9 frames@60Hz = 1.9 frames@120Hz

/tmp/niman-research/fixtures/worst-note.md (adversarial fixture)

layout_blocks=1282 visual_lines=18589 avg_visual_lines_per_block=14.500

arithmetic at 44.7 us/line: (a): 18,589.0 lines x 44.7 us = 830,928 us = 830.93 ms = 49.9 frames@60Hz = 99.7 frames@120Hz (b): 580.0 lines x 44.7 us = 25,926 us = 25.93 ms = 1.6 frames@60Hz = 3.1 frames@120Hz (c): 1,740.0 lines x 44.7 us = 77,778 us = 77.78 ms = 4.7 frames@60Hz = 9.3 frames@120Hz

(Arithmetic for test/fixtures/markdown/fixture-200kb.md, test/fixtures/markdown/fixture-1mb.md is in out/sizing.txt; the shape is identical and those files are not the sizing target.)

5.6.5 Plain answer

The adversarial fixture shows 14.5 visual lines per block, an artefact of its deliberate 2 000-line mega-paragraph and 2 000-line code fence: a “40-block viewport” there is not a realistic viewport, so its viewport row is a worst case rather than an expectation.

Eager layout is not viable. At the repo’s own debug-mode 44.7 µs/line, laying out every block of the real note costs 552 ms = 33 frames at 60 Hz; at the pessimistic 80 µs it costs 988 ms = 59 frames; at the optimistic 30 µs it still costs 371 ms, 22 frames. Even the most generous plausible figure puts eager layout more than an order of magnitude over a frame budget. The existing app already knows this: the rejected eager renderer measured 2 769.2 ms on a 200 KB fixture, and the accepted editor pays only for visible lines. Only visible (plus a bounded cache of) blocks may be laid out. The 120 Hz budget (8.33 ms) means the cache extent must be filled incrementally across frames, not synchronously: one viewport of cache costs 15.7 ms at 44.7 µs/line, i.e. ~1.9 frames at 120 Hz.

Two further consequences of the measured numbers:

• 2.93 visual lines per block means a “~40 blocks” viewport is really ~117 visual lines, not 40. Any cost model that multiplies viewport blocks by a per-line cost under-estimates by 2.9×.
• At 841 display-math blocks in the real note and 902 in the adversarial fixture, math layout — not prose layout — is the largest single widget class in the document, and it is excluded from the 44.7 µs figure above.

5.7 A synthetic “worst note” spec and fixture

Generator: in the repo, at tool/make_worst_note.dart, re-runnable and byte-deterministic (no randomness, no timestamps). It writes test/fixtures/spec/worst-note.md, and dart run tool/make_worst_note.dart --check prints what it produced next to the spec below. test/unit/worst_note_test.dart asserts both halves of the contract: that the fixture still has the shape this table asks for, and that the engine — the block scanner and the extension masker — survives it, which is Phase 1’s “the parser is run against it”. Every construct below is justified by a measured worst case from §1–§6; each section of the fixture is labelled in-file with the corpus value it exceeds.

5.7.1 Specification

5.7.2 Measured stats of the generated fixture

Measured 2026-09-21 on the committed fixture, which the generator reproduces byte for byte:

The engine’s own numbers on it, from the same run: 1 427 blocks, every one of the twelve block kinds present, and a keystroke in prose re-scanning fewer than 30 lines. A keystroke inside the 2 000-line paragraph re-scans about 2 000 — which is O(block), not O(document), and is the case a caller has to plan for.

5.7.3 The stats the first version had

Provenance: scripts/gen_global_stats.py, gen_block_census.py, gen_inline_census.py, gen_math_deepdive.py, gen_worstcases.py, gen_layout_model.py, all run on /tmp/niman-research/fixtures/worst-note.md by run_all.sh.

5.7.3 What the fixture is for

The real note and the existing fixtures are complementary and each misses half the problem:

• the real note has the volume, the math and the long lines but has no fences, no ordered lists, no nested quotes, no display math in a container, no link reference definitions, no CRLF and no setext headings;
• the synthetic fixtures have the full GFM block grammar but are ~30 bytes/line, flat (depth 2), 100 % ASCII, and never exceed 439 bytes on a line.

worst-note.md is the union: it contains every construct either corpus lacks, at 1.48 MB and 11 773 lines — above both — with 19 700 inline and 902 display math expressions, i.e. more math volume than the real note. A renderer that opens it, scrolls it, jumps to the end, and returns the exact byte count must have correct block parsing, bounded layout, a math cache and a scroll map that survives a 2 000-line paragraph and a 2 000-line code fence.

Suggested assertions for a future benchmark against this fixture (all bounds below are derived from the measurements in §6, not invented):

5.8 Summary of the numbers that should drive the design

6. The primitives available

Target: Niman, Flutter 3.47.2 stable / Dart SDK ^3.13.2, Android + Linux + Windows. Constraint: one new widget, pure Flutter/Dart, no third-party markdown/editor/math/highlight package. Hard case: one 934 KB, 10 331-line Markdown note, 13 845 math spans (841 display blocks and 13 004 inline — the brief’s “1 683” counted the $$ lines, and the correction is recorded in §5.3 and §9.3). Disk is source of truth; never lose or normalise the user’s bytes.

Provenance of every claim below. Flutter SDK checkout on this machine: /home/alessandro/develop/flutter (framework revision d3b14c8769, engine a804b26164, Dart 3.13.2). dart:ui text source is not in the dart-sdk snapshot; I read it from the engine checkout at /home/alessandro/develop/flutter/engine/src/flutter/lib/ui/text.dart and from /home/alessandro/develop/flutter/bin/cache/flutter_web_sdk/lib/ui/text.dart. Line numbers quoted are from those files. Tags used throughout: [src] = read from SDK source on disk, [doc] = official docs / API reference, [measured] = measured on this machine, [inference] = my reasoning, not verified.

Current locked versions replaced by this work (from pubspec.lock, read-only): re_editor 0.10.0, flutter_quill 11.5.1, flutter_markdown_plus 1.0.12, markdown 7.3.1, katex 1.0.0, katex_dart 0.1.1, highlight 0.7.0, flutter_math_fork 0.7.4 (transitive).

6.0 Executive summary of the cost model

The whole design follows from six facts, each verified below:

1. Shaping is the cost; painting is nearly free. ui.Paragraph is immutable and its layout() performs line breaking and shaping in C++ (Skia skparagraph). TextPainter.paint() is one canvas.drawParagraph call. [src]
2. A Paragraph is never incrementally updated. TextPainter.markNeedsLayout() calls _layoutCache?.paragraph.dispose() and drops the cache; layout() then rebuilds the whole ui.Paragraph from the InlineSpan tree via ParagraphBuilder. There is no “setText” on a built paragraph. [src]
3. Text layout cannot leave the root isolate, and Flutter has no feature or roadmap item to change that. The engine’s UIDartState::ThrowIfUIOperationsProhibited() throws “UI actions are only available on root isolate.” for PictureRecorder, Canvas and ParagraphBuilder/Paragraph; flutter#41707 is closed as not planned. [src, engine] Everything else — parsing, highlighting, math parsing, disk I/O — can and should move off-thread.
4. Line breaking is cheaper than shaping, but not free. TextPainter.layout() re-runs paragraph.layout(ParagraphConstraints(width: ...)) on the existing paragraph when only the width changed, and short-circuits entirely if the paragraph is already wider than its maxIntrinsicWidth (_TextPainterLayoutCacheWithOffset._resizeToFit, text_painter.dart:471-516). [src]
5. Crossing an isolate is now cheap, and the return path is the one that matters. [measured] A String is shared, not copied (flat 12–24 µs from 1 KB to 32 MB); typed data is copied at ~5.4 GB/s; TransferableTypedData is not faster end-to-end because fromList copies; but Isolate.exit is genuinely zero-copy — 32 MB returned from inside a worker costs 498 µs versus 5 563 µs when the buffer is captured by the closure (11×). Build the result inside the worker and Isolate.exit it.
6. The current preview stack’s real cost is not lexing, it is materialisation. highlight 0.7.0 parses 200 KB of Dart in 57.9 ms [measured], but its Result.toHtml() path is super-quadratic (100 KB → 6 868.8 ms) because of str += plus three replaceAll(RegExp) passes per node, and flutter_highlight runs it inside build() on the UI isolate with no cache. Separately, the cpp grammar is clean O(n²) on 'a'*n + '!' (8 KB → 1 854 ms) and Dart has no regex timeout — a user-pasteable hang. [measured]

Therefore: one Paragraph per block, a layout cache keyed by a stable hash, blocks outside the viewport never laid out, parse/highlight/math-parse work in isolates returning flat typed buffers, and the incremental re-lex stopping at the first line whose entering state matches the cached one.

6.1 Text layout & painting cost model

6.1.1 The object graph

TextSpan / WidgetSpan tree  (Dart objects, immutable, @immutable)
        |  TextSpan.build(ui.ParagraphBuilder)      [src text_span.dart]
        v
ui.ParagraphBuilder          (native, engine-side)
        |  pushStyle/addText/pop/addPlaceholder
        v
ui.Paragraph                 (native, IMMUTABLE once built)
        |  .layout(ui.ParagraphConstraints(width: w))   <-- THE expensive call
        |  .getBoxesForRange / getPositionForOffset / computeLineMetrics  (cheap reads)
        v
Canvas.drawParagraph(paragraph, offset)                <-- cheap

TextPainter (painting/text_painter.dart:589) is a Dart-side wrapper that owns exactly one ui.Paragraph inside a _TextPainterLayoutCacheWithOffset (:426), plus cached List<ui.LineMetrics> and List<TextBox> placeholder boxes (:520-525). It is not itself a cache — it caches one layout of one text+style combination.

6.1.2 Exact API surface (Flutter 3.47.2)

// painting/text_painter.dart
TextPainter({
  InlineSpan? text, TextAlign textAlign = TextAlign.start, TextDirection? textDirection,
  @Deprecated double textScaleFactor = 1.0, TextScaler textScaler = const _UnspecifiedTextScaler(),
  int? maxLines, String? ellipsis, Locale? locale, StrutStyle? strutStyle,
  TextWidthBasis textWidthBasis = TextWidthBasis.parent, TextHeightBehavior? textHeightBehavior,
});

void  markNeedsLayout();                                    // :781  disposes the Paragraph
void  layout({double minWidth = 0.0, double maxWidth = double.infinity});  // :1221
void  paint(Canvas canvas, Offset offset);                  // :1322
void  dispose();                                            // :1827
void  setPlaceholderDimensions(List<PlaceholderDimensions>? value);  // :1064

double get width;            // :1151  asserts laid out AND !_debugNeedsRelayout
double get height;           // :1160
Size   get size;             // :1168
double get minIntrinsicWidth;      // :1135   requires layout
double get maxIntrinsicWidth;      // :1143   requires layout
double get preferredLineHeight;    // :1129   DOES NOT require layout (own 1-char template paragraph)
bool   get didExceedMaxLines;      // :1194
double computeDistanceToActualBaseline(TextBaseline baseline);  // :1178

Offset     getOffsetForCaret(TextPosition position, Rect caretPrototype);  // :1447
double     getFullHeightForCaret(TextPosition position, Rect caretPrototype); // :1496
List<TextBox> getBoxesForSelection(TextSelection selection,
    {ui.BoxHeightStyle boxHeightStyle = ui.BoxHeightStyle.tight,
     ui.BoxWidthStyle  boxWidthStyle  = ui.BoxWidthStyle.tight});            // :1666
ui.GlyphInfo? getClosestGlyphForOffset(Offset offset);                        // :1696
TextPosition  getPositionForOffset(Offset offset);                           // :1714
TextRange     getWordBoundary(TextPosition position);                        // :1730
WordBoundary  get wordBoundaries;                                            // :1745
TextRange     getLineBoundary(TextPosition position);                        // :1750
List<ui.LineMetrics> computeLineMetrics();                                   // :1794
List<TextBox>? get inlinePlaceholderBoxes;                                    // :1039
List<PlaceholderDimensions>? _placeholderDimensions;                          // :1082

static double computeWidth({...});              // :642  creates + lays out + disposes a throwaway painter
static double computeMaxIntrinsicWidth({...});  // :696  same

dart:ui side (ui/text.dart):

factory ParagraphBuilder(ParagraphStyle style);
  void pushStyle(TextStyle style); void pop();
  void addText(String text);
  void addPlaceholder(PlaceholderDimensions dimensions, {double baselineOffset = 0.0, double scale = 1.0});
  Paragraph build();
  int get placeholderCount; List<double> get placeholderScales;

abstract class Paragraph {
  double get width, height, longestLine, minIntrinsicWidth, maxIntrinsicWidth,
         alphabeticBaseline, ideographicBaseline;
  bool get didExceedMaxLines; int get numberOfLines;
  void layout(ParagraphConstraints constraints);          // ParagraphConstraints({required double width})
  List<TextBox> getBoxesForRange(int start, int end, {BoxHeightStyle, BoxWidthStyle});
  List<TextBox> getBoxesForPlaceholders();
  TextPosition getPositionForOffset(Offset offset);
  GlyphInfo? getGlyphInfoAt(int codeUnitOffset);
  GlyphInfo? getClosestGlyphInfoForOffset(Offset offset);
  TextRange getWordBoundary(TextPosition position);
  TextRange getLineBoundary(TextPosition position);
  List<LineMetrics> computeLineMetrics();
  LineMetrics? getLineMetricsAt(int lineNumber);
  int? getLineNumberAt(int codeUnitOffset);
  void dispose(); bool get debugDisposed;
}

Note what does not exist in dart:ui: there is no per-glyph shaping API, no “shape this string and give me advances” entry point, no glyph-ID or glyph-advance query, and no way to mutate a built Paragraph. (GlyphInfo exists — ui/text.dart:192 in the web SDK — but it only exposes graphemeClusterLayoutBounds, graphemeClusterCodeUnitRange, writingDirection for a range; it is not a shaping oracle.) [src] This is why custom math rendering must go through TextPainter for glyph runs and Canvas for rules.

6.1.3 Cheap vs. expensive, precisely