Web capture (#531)
Saving a web page — or a selection of it — as a clean Markdown note, its pictures in the attachments folder. Written before the code (2026-10-07), as the plan the work follows; the screens are in docs/design/web-capture.
Why the plan changed from the issue
The issue chose Readability.js running in “the same headless JS engine the diagrams bring” (#530). #530 shipped a Mermaid engine written in Dart instead, so there is no JS engine to share, and adding one — QuickJS or flutter_js, Readability.js and a light DOM such as linkedom — would put a native runtime in every build for this one feature.
The constraint that shapes the rest: no new pub dependency.
- The article is found by a Dart port of Readability.js (Mozilla,
Apache-2.0) over
package:html, already a dependency through the EPUB reader. The port keeps upstream’s algorithm and names, so a diff against a new upstream release stays readable. - The conversion to Markdown is
XhtmlMarkdown(lib/src/epub/xhtml_markdown.dart), moved tolib/src/markdown/from_html/(markdown/html/is the writer the other way) and grown for the web. - HTTP is
dart:io’sHttpClient, as the sync and the updates use it. - The clipboard’s HTML is read by Niman’s own native code on each
platform, not by
super_clipboard, which would bring a Rust toolchain. - Notifications with buttons are already in flutter_local_notifications
22 on all three platforms (
AndroidNotificationAction,LinuxNotificationAction,WindowsAction).
The port
lib/src/capture/readability/, one concern per file and each under 300
lines, every file headed “Ported from mozilla/readability, Apache-2.0”:
patterns and options, the DOM walk, text metrics, candidate scores,
preparing the document, grabbing the article, the top candidate and its
siblings, cleaning (plain and conditional), post-processing (absolute
URLs), metadata (meta tags, OpenGraph, JSON-LD), isProbablyReaderable, and
the entry point with upstream’s retry loop over the flags.
What package:html 0.15.7 lacks, and how it is filled:
| Missing | Filled by |
|---|---|
nextSibling / previousSibling on plain nodes |
helpers over parentNode.nodes |
renaming a tag (localName is final) |
a new Element.tag, attributes copied, children reparented, score moved |
data on a node (node.readability) |
an Expando |
style |
a small parser of the inline display and visibility |
baseURI |
the page’s Uri passed in, then the first base[href] |
innerHTML cached between retries |
document.clone(true) per attempt |
package:html decodes only ASCII and UTF-8 bytes itself, and throws on
Windows-1252, so the page is always decoded to a String before it is parsed.
The licence ships with the app: assets/licenses/readability.txt, added to
Flutter’s LicenseRegistry, shown from an “Open-source licences” row in
Settings › About.
The oracle
Mozilla’s own test pages — source.html, expected.html,
expected-metadata.json — for a subset kept in
test/fixtures/readability/, pinned to an upstream commit, with its README
and licence. Synthetic pages come first; real pages only from permissive or
institutional sites (Wikipedia, Mozilla, IETF).
The article is compared as a tree, not as bytes: a pre-order walk that skips
whitespace-only text and comments and compares the tag, the attribute set
and the collapsed text, reporting the path of the first difference.
Metadata is compared field by field. html5lib and upstream’s JSDOMParser do
not always build the same tree; each such page is listed in
known-diffs.txt with its reason, and the list only shrinks.
Niman’s own fixtures (test/fixtures/capture/) end in a golden .md: a
figure with srcset and lazy pictures under a <base href>, a page that
builds its text with a script, a page with nothing but OpenGraph, a
Windows-1252 page, a Shift_JIS page, a byline from JSON-LD.
A perf test runs the Wikipedia page and holds its absolute bar behind
NIMAN_PERF, as every perf test does.
The pipeline
- Download (
lib/src/capture/fetch/), inIsolate.run: http and https only, up to 5 redirects, 15 s to connect and 30 s in all, at most 10 MB, HTML or XHTML only. The user agent says it is Niman. - Charset: BOM, then the
Content-Typeheader, then a<meta>in the first 1024 bytes, else UTF-8. UTF-8, ASCII, Latin-1 and Windows-1252 are decoded withdecodeNoteText, UTF-16 by hand; any other charset counts as too little text, so the browser — which decodes everything — reads it. - Extract with the port. Too little text means no article, fewer than 500 characters, or a charset step 2 could not decode.
- Only then, the browser, and step 3 again:
- desktop: the engine the PDF export already finds (
findPdfEngine: Edge on Windows, a Chromium onPATHon Linux) with--headless=new --dump-dom, a throwaway profile deleted afterwards, a time budget for the page’s scripts, 45 s and 20 MB at most; - Android: a WebView of its own (
PageReaderBridge.kt, separate from the PDF one), JavaScript on, network pictures off, no file access and no JS interface, itsouterHTMLread once the page has settled.
- desktop: the engine the PDF export already finds (
- Convert to Markdown and download the pictures into the attachments folder, named by their SHA-256 as pasted pictures already are: 15 MB each, 60 at most, 4 at a time.
- Save the note: frontmatter
source,captured,author,tags. When nothing was found even after the browser — or there is no browser, on a Linux without Chromium — the note keeps the title, the description and the picture, under a quoted notice to open the link.
What was removed (scripts, the menu, a cookie banner, the footer) is counted outside the port, so the port stays what the oracle checks.
The platforms
- Desktop: Capture web page… in the command palette, a link pasted
outside the editor, or a link dropped on the window. Dropping a link needs
the runners: on Linux, http(s) entries of
text/uri-listare forwarded; on Windows, an OLEIDropTargetreplacesDragAcceptFiles, for files and URLs alike. - Android: Share → Niman. The share bridge also sends
EXTRA_SUBJECT; a lone URL or “title, URL” is a page, text with a URL carrying#:~:text=(or quoted text before a URL) is a quote, anything else goes to the quick note as today. On Save the sheet closes and the task goes back to the browser; the capture runs in the app’s own engine under the notification plugin’s foreground service (shortService, so it must end within its three minutes: Niman stops at 150 s). No second Flutter engine, no WorkManager. -
Paste as Markdown (Ctrl+Shift+V, the editor’s menu on a phone): Windows reads
HTML Formatthroughdart:ffi(its offsets are bytes, and it carriesSourceURL); Linux asks GTK fortext/htmland the browsers’ source-URL targets; Android readsClipData.getHtmlText(), which has no source URL. Undo puts the plain text in its place.Settled while building it (PR 4): CF_HTML is converted from
StartHTMLtoEndHTML— the context, so list items keep their list — and the fragment alone only when there is none. The page is linked by a last paragraph— [label](<url>), as a captured quote is, the label being a<title>the copy carries, else the host. Pictures stay remote links: a paste downloads nothing, and one with no web address (data:,blob:, a relative address with no page) is left out. On Linux the page is Chromium’schromium/x-source-urlor Firefox’stext/x-moz-url-priv, and the bytes are decoded in Dart, UTF-16 when Firefox wrote them so.
Delivery
One PR at a time, each branched from main once the one before is merged:
- The engine, no UI: the moves (
HtmlMarkdown, the process runner, the attachment store), the port and its oracle, the download, the charsets, the pictures, the notes. - The browser fallback, desktop and Android.
- The capture UI, on all three platforms at once: the dialog, the share sheet, the notifications, the drops, and the user guide.
- Paste as Markdown, on all three platforms at once.
Risks
- The two parsers’ trees differ in places: the known-diffs list absorbs it, and says why.
textis recomputed on every call; on a huge page on a phone that may cost. The perf test watches it; caching text per pass is the fix.- Edge headless: whether
--dump-domreaches a pipe on Windows, and whether a managed policy forbids headless runs, is the first thing PR 2 checks. - Android: the foreground service’s limit, vendors that kill background work, a WebView throttled off-screen. A bot wall ends in the unreadable note, by design.
- The clipboard: X11 and Wayland offer different targets.
- The fixtures add a few megabytes to the repository.