Block scanner: the indentation model (design for the remaining repros)
Superseded, kept for its evidence. The model was built — phase 3 of the plan in
docs/dev/block-scanner-containers.md— but not as this document designs it. Its diagnosis held (each container must measure the line where its parent left it); its model did not: the target parser,package:markdown, keeps no container stack, and hands a lazy line on undedented, which is what this document’s cases 10-13 are. The walk that emulates it (ContainerWalk) reaches 0 of 40 000, and cases 1 and 4 turned out to be the harness’s, not the scanner’s. Read the plan document for what was done; the tables below are the evidence it started from. The package itself is gone since (docs/dev/block-tree.md, phase 8): the reading iscmark-gfm’s.
docs/dev/block-scanner-containers.md records the container rework up to
309 of 40 000 documents differing (138 minimal) on
fix/block-scanner-containers. Everything left needs one change the
incremental fixes could not make: a single, per-line notion of how far
the line is indented past the containers it is already in. This
document is the plan for that change, with the evidence it is built on.
Do not start from the current code’s _contentColumn/_indentOf/
_listMarker; start from the model below and rewrite those three to
follow it.
The names below are the ones the code had when this was written. Since
phase 2 of the plan (docs/dev/block-scanner-containers.md) the readers
live in line_containers.dart (LineContainers.contentColumn, …),
line_syntax.dart (LineSyntax.listMarker, LineSyntax.indentOf, …)
and line_rules.dart (LineRules.listAfter, …); that document has the
full map. The rewrite is LineContainers, and LineState’s two stacks.
Evidence
Every minimal repro left wants containers measured where their content
actually begins, and the two things that are measured today both start
at the wrong place. The reference (package:markdown, the parser the
read view uses) for the shapes that matter:
A quote’s content is measured past the >, and a > past it nests
Input (· = space) |
Reference | Scanner now |
|---|---|---|
>·-·w then ····>·w |
blockquote ul li blockquote p (quote 2) |
one quote, depth 1 |
>·-·w then ··>·w |
blockquote ul li (quote 1, the item’s text) |
one quote, depth 1 |
>·w then ····>·w |
blockquote p (one, the > is text) |
one quote, depth 1 |
>·w then >·>·w |
blockquote p blockquote p (quote 2) |
one quote, depth 1 |
··>·w then ····w |
blockquote p (the last line in the quote) |
quote then code |
The first two are the same two lines with different indent, and the
reference tells them apart only by where the second > stands
relative to the item’s content column (4):
- ` > w
— the>` at 4 is inside the item, so it opens a quote in the item: two levels. - ` > w
— the>` at 2 is short of the item’s content column, so it is the item’s text in the one open quote: one level.
A quote block is the quote, not the items in it
| Input | Reference | Scanner now |
|---|---|---|
>·-·w |
blockquote ul li (the quote is at list −1) |
quote at list 0 |
*·w / ··>·w / >·w |
ul li blockquote p blockquote p |
li, quote0, quote−1 |
>·-·w (alone) |
blockquote ul li |
quote0 |
The scanner counts the item written after the > as the quote’s own
list depth. It is not: the quote sits in the items outside it, and the
items inside it are read when the quote’s own lines are scanned again
the way BlockView._quoteContent already does.
Indented code inside a quote is measured from the quote’s content
| Input | Reference | Scanner now |
|---|---|---|
>·w / ····w / ····w |
blockquote p pre code |
quote, quote, code |
··>·w / ····w |
blockquote p |
quote, code |
Four spaces past the quote’s content column is code; four spaces past the margin is not, when the quote is indented. The first row shows the boundary: a quote takes one indented paragraph continuation, and the second indented line opens code that ends the quote.
An item’s contents are measured from the item’s marker
| Input | Reference | Scanner now |
|---|---|---|
··1.·w / ······--- / ····w |
ol li h2 pre code (code in the item) |
heading, code at −1 |
··2)·w / ······--- / ····w |
ol li h2 pre code |
heading, code at −1 |
··1.·w / ······-·w / ····*·w |
ol li ul li ul li (three levels) |
li0 li1 li0 |
··1.·w / ······`` ` / ····w |
ol li pre code |
item, fence, text |
` 1. w has its marker at 2 and its content at 5. w at 4 is less
indented than the content column but still **inside the item** — one
space past the marker — and is code there (four spaces from the
*marker's own line start*, i.e. the container's content, not the
margin). * w` at 4 likewise opens a list level inside the item,
and the reference has it three deep, not at zero.
Fences and items close together
| Input | Reference | Scanner now |
|---|---|---|
1.·w / ···` / / w` |
ol li pre code pre code |
item, fence, paragraph in item |
*·w / ··`` ` / *·w / ··*·w |
ul li pre code li ul li |
item, fence, item0, item1 |
The line that closes an item’s fence leaves the item with it, and is
then read as if no fence were open (the w after the close is code in
the reference — a quirk of package:markdown, but the parser is the
target). The earlier fix got the second row’s depths right but not the
first row’s kinds.
The model
One value per line, computed once, before any container decision:
effectiveIndent(line) = the column of the line's first content character,
after consuming the indent of every open container
(each open item's content column, each open quote's
content column)
Concretely, start at column 0 and walk the line:
- skip up to three spaces, then a
>and one space → that is a quote level; the column now past it is the new base; - if both an item is open and the column reaches the item’s content column, consume it (the base becomes the item’s content column).
What is left when neither applies is effectiveIndent: how far the
line is indented past everything it is already in. Every decision
reads that value, not _indentOf from the margin:
| Decision | Reads |
|---|---|
| a marker may interrupt a paragraph | the marker at effectiveIndent, at most effectiveIndent + 3 |
| an ATX heading | # within three of effectiveIndent |
| a thematic break | --- within three of effectiveIndent |
| an indented code block | four or more past effectiveIndent |
| a fence opens | three at most past effectiveIndent |
| an item opens | a marker at effectiveIndent, content at marker + width + padding |
| an item continues | the line at or past the item’s content column |
| a quote opens | a > at effectiveIndent |
listDepth and quoteDepth then fall out of the walk itself: they are
how many containers were consumed, not two counts kept apart. A
block that is a quote reports the depth of the items it was consumed
into before the > — depth of the walk minus the quotes it opened —
which is what _quoteOuterItems started to do.
Where it touches
Three functions compute the model; the rest read it:
_afterQuotes(text)→ fold into the walk (_consumeContainers)._indentOf(text)→ replace by_effectiveIndent(text, state)._contentColumn(text, state)→ the base the walk started each decision from; keep for the item comparisons only._listMarker(text, reach)→ take the walk’s start offset, markers then compared to the item columns the walk produced.
LineState keeps quoteDepth and listStack as today, plus:
quoteContentColumn— where the innermost quote’s content starts (what_isQuoteIndentedand the “one continuation” rule need);quoteMarkerColumn— where its>stands (what decides nested vs same level);quoteIndented— as today.
Both columns are per-container, not derived on the fly from the line,
because a lazy line carries no > of its own and must keep what the
open quote had.
Test cases to pin, one at a time
Write these as unit cases in test/unit/block_scanner_test.dart before
the rewrite, so each step of it is checked against a known answer. They
are the minimal repros from the harness, with the reference’s answer.
Measured against the scanner on fix/block-scanner-containers
(01963d99): 7 are red and 8 pass. The 8 passing are regression
cases the rewrite must not break; the 7 red are what it is for.
| # | Input (· = space) |
Line | Want | Today |
|---|---|---|---|---|
| 1 | >·-·w / ····>·w |
1 | quote L−1 Q2 | quote L−1 Q1 (red) |
| 2 | >·-·w / ··>·w |
1 | quote L−1 Q1 | pass |
| 3 | >·w / ····>·w |
1 | quote L−1 Q1 | pass |
| 4 | >·w / >·>·w |
1 | quote L−1 Q2 | quote L−1 Q1 (red) |
| 5 | ··>·w / ····w |
1 | quote L−1 Q1 | pass |
| 6 | >·-·w |
0 | quote L−1 Q1 | pass |
| 7 | *·w / ··>·w / >·w |
2 | quote L−1 Q1 | pass |
| 8 | >·w / ····w / ····w |
2 | code L−1 Q0 | pass |
| 9 | >·w / ····w |
1 | quote L−1 Q1 | pass |
| 10 | ··1.·w / ······--- / ····w |
2 | code L0 Q0 | code L−1 (red) |
| 11 | ··2)·w / ······--- / ····w |
2 | code L0 Q0 | code L−1 (red) |
| 12 | ··1.·w / ······-·w / ····*·w |
2 | item L2 Q0 | item L0 (red) |
| 13 | ··1.·w / ······`` ` / ····w |
2 | code L0 Q0 | code L−1 (red) |
| 14 | 1.·w / ···` / / w` |
3 | code L−1 Q0 | paragraph L0 (red) |
| 15 | *·w / ··`` ` / *·w / ··*·w |
3 | item L1 Q0 | pass |
The red classes, in the harness’s own counts: 1 and 4 are the
quoted → quoted Q1 (nested quote) class, 10/11/13/14 the
quoted → code L−1 and code → code L−1 classes, 12 the
text → text L0 deep-item class.
Method
- Write cases 1-15 as unit tests (red against the current scanner).
- Introduce
_consumeContainersand_effectiveIndent, and rewrite_listMarker/_contentColumn/_isIndented/_isRule/_headingLevelcall sites to read them together — the two failed attempts in this session moved the harness by thousands when only some call sites were changed. - Keep the unit suite and, at each step, run
NIMAN_SCANNER_DIFF=1 flutter test test/unit/block_scanner_commonmark_test.dart; the minimal count must fall at every step. A step that raises it is reverted whole. - Keep
block_scanner_test.dart’s “a rescan that stops short” green at every step: phase 1 of the plan indocs/dev/block-scanner-containers.mdunskips it before the rewrite starts. - Only then touch the downstream
BlockParser.contentTextindent stripdocs/dev/block-scanner-containers.mdlists.
What the last two attempts proved (do not repeat)
- Passing an offset past the quotes into
_listAfteralone: 309 → 6 393. The marker/indent readers must all move together. - Adding
quoteColumn/quoteIndentas state and threading them: 309 → 4 343, withquoted → codeandcode → quotedswapping places. The columns alone are not enough; the walk is, because it makes the indent one value instead of two half-measured ones. - Measuring the indent past the quotes (
_indentAfterQuotes) and making the marker columns relative to it: 309 → 6 346. The stack’s item columns are absolute and a relative marker does not compare to them — the walk has to carry how many quotes were consumed and compare items only at the same quote level. - Making indented code in an item measure from the item’s marker alone (not its content column): 309 → 352, 59 new minimal repros. A marker four in is a sublist, not code in the item; the rule needs the walk’s effective indent to tell them apart.
Each partial change fixes the shape it aimed at and moves others by the hundreds, which is what makes this a rewrite: the four numbers above are the same lesson four times.
A fifth attempt, this session, went further: it made _listMarker
measure past the quotes, _contentColumn read the content position past
them, _quoteDepthAfter sum a nested > past an item’s content,
_quoteOuterItems give a quote block the items outside it, and added a
quoteIndent for the indented-code boundary. It passed fourteen of
the fifteen cases (all but the quote-at-the-margin one) and still sat
at 4 476 of 40 000 differing / 577 minimal, against 309/138 before it —
145 of the new repros are quoted → code shapes like 1. w /
` ``` ` / ` > w`, where an item and a quote interleave.
The reason is the one the four numbers already carry: _listMarker
returned absolute columns, _contentColumn compared a position past the
quotes, and every other reader sat between the two. A rewrite that
measures once, in the walk, removes the gap; patching the three readers
keeps it and moves the errors around. The fifteen cases are a good
checkpoint for that rewrite — they are green in the fifth attempt — but
the harness count is the gate, and it did not fall.
A sixth attempt made the marker and item columns relative to the
quote prefix (so both halves measure from the same point) and gave a
quote block the items before its >: 309 → 4 229 / 413, two of the
fifteen cases red again. Same signature.
Why patching cannot work (six attempts, six times)
| Attempt | Harness (diff / minimal) |
|---|---|
| starting point | 309 / 138 |
offset past quotes in _listAfter only |
6 393 / — |
quoteColumn/quoteIndent as state |
4 343 / — |
| indent past quotes, marker columns relative | 6 346 / 197 |
| indented code from the item’s marker | 352 / 175 |
| five readers moved together (14/15 cases) | 4 476 / 577 |
marker and item columns relative + _quoteOuterItems |
4 229 / 413 |
Every attempt is a different subset of {_listMarker, _contentColumn,
_indentOf/_indentPastQuotes, _quoteDepthAfter, _quoteOuterItems,
quoteIndent} moved to measure past the quotes, and every one leaves the
same defect: an item’s column is measured on the line that opened it,
a line’s position is measured on the line being read, and the two are
compared as if they were one space. The walk has to produce both in
the same coordinates — the item’s column when it opened is itself the
result of a walk over its own line — and no patch to the readers gives
that, because the item columns are already in the stack from earlier
lines.
So the rewrite is: replace _contentColumn, _indentOf/_afterQuotes
and _listMarker with one _consume(text, state) that walks the line
and returns the content position, the depth and the markers in the
line’s own coordinates, and make the stack hold each item’s column
as that walk produced it (so opening and reading use one function).
The fifteen cases and the harness are the gate.
The fifteen cases are in block_scanner_test.dart’s “the indentation
model” group: eight run, seven are skip: pending with this document’s
name. The rewrite unskips them.