commit b6048de04f6996226fbba070b9919edb1c37c0db Author: phg Date: Thu Jul 23 00:20:12 2026 +0200 docs: research Linux FUSE implementation stack diff --git a/docs/research/linux-fuse-message-stack.md b/docs/research/linux-fuse-message-stack.md new file mode 100644 index 0000000..a5eba3b --- /dev/null +++ b/docs/research/linux-fuse-message-stack.md @@ -0,0 +1,139 @@ +# Linux FUSE and message-parsing implementation stack + +Research date: 2026-07-23 + +## Decision + +Implement the daemon in stable Rust and use this stack: + +| Concern | Choice | Baseline examined | +| --- | --- | --- | +| Linux FUSE protocol | [`fuser`](https://docs.rs/fuser/0.17.0/fuser/) with default features (no `libfuse` feature) | 0.17.0 | +| RFC message headers and address lists | [`mailparse`](https://docs.rs/mailparse/0.16.1/mailparse/) | 0.16.1 | +| Filesystem change notifications | [`inotify`](https://docs.rs/inotify/0.11.4/inotify/) directly, on a dedicated blocking thread | 0.11.x | +| Persistent index | [`rusqlite`](https://github.com/rusqlite/rusqlite) linked to the Nix-provided SQLite library | current compatible release | +| Concurrency | `fuser` worker threads plus one bounded, single-consumer state coordinator; immutable in-memory read snapshots | Rust standard library primitives | +| Build and deployment | Cargo lockfile; Nixpkgs `rustPlatform.buildRustPackage`; Linux-only package and NixOS module | stable Rust supported by the selected `fuser` release | + +Pin exact crate versions and checksums in `Cargo.lock`, but accept compatible patch updates after the integration suite passes. Keep each third-party API behind a small internal adapter so that parser, watcher, FUSE, or database bindings can be replaced independently. + +## Why this fits + +### Rust and `fuser` + +`fuser` exposes the inode-oriented FUSE operations needed here, including `lookup`, `read`, `readdir`, `rename`, `setattr`, and `unlink`; its `Filesystem` implementation must be `Send + Sync`, matching a daemon that serves concurrent kernel requests ([`Filesystem` trait](https://docs.rs/fuser/0.17.0/fuser/trait.Filesystem.html)). On Linux it can replace the userspace portion of libfuse and can be built without the optional `libfuse` feature, reducing the runtime/build surface to the kernel FUSE device and mount tooling ([upstream README](https://github.com/cberner/fuser/blob/v0.17.0/README.md)). + +Version 0.17 has explicit worker-count and cloned-FUSE-fd configuration. Multiple event-loop threads are supported on Linux, and cloned fds require Linux 4.5 or later ([`Config`](https://docs.rs/fuser/0.17.0/fuser/struct.Config.html)). This allows blocking local file reads to proceed concurrently without adding an async runtime. Start with a small fixed worker count (for example four), make it configurable, and measure before increasing it. + +External deliveries and flag renames change entries behind the mounted view. `fuser::Notifier` supports invalidating a directory entry or inode and reporting deletion to the kernel, which is required in addition to publishing a new userspace snapshot ([`Notifier`](https://docs.rs/fuser/0.17.0/fuser/struct.Notifier.html)). Use short, conservative entry/attribute TTLs as a fallback; notifications are the primary cache-coherency mechanism. + +Rust is a good fit because the design combines concurrent FUSE callbacks, borrowed header buffers, path and inode identity, and serialized mutation. Its ownership and `Send`/`Sync` checking make accidental unsynchronized sharing a compile-time problem, while the standard library supplies multi-producer/single-consumer channels and read/write locks ([Rust shared-state concurrency](https://doc.rust-lang.org/book/ch16-03-shared-state.html), [`std::sync`](https://doc.rust-lang.org/std/sync/)). This does not eliminate filesystem logic errors or deadlocks, so integration testing remains mandatory. + +### `mailparse` for a header-only scan + +The projection needs only the RFC message header block, not MIME bodies. `mailparse::parse_headers` consumes raw bytes through the header/body separator, returns every header plus the body offset, and explicitly permits callers that only care about headers to ignore the body offset ([`parse_headers`](https://docs.rs/mailparse/0.16.1/mailparse/fn.parse_headers.html)). Its header-map API performs case-insensitive lookup and can return every repeated value in source order ([`MailHeaderMap`](https://docs.rs/mailparse/0.16.1/mailparse/trait.MailHeaderMap.html)). + +For every `Delivered-To`, `X-Original-To`, `To`, and `Cc` header selected by the specification's precedence rule, call `addrparse_header` on the original `MailHeader`; upstream documents that this avoids correctness problems caused by decoding encoded words before splitting an address list ([`addrparse_header` source and documentation](https://docs.rs/mailparse/0.16.1/mailparse/fn.addrparse_header.html), [`addrparse` caveat](https://docs.rs/mailparse/0.16.1/mailparse/fn.addrparse.html)). Flatten both individual and group addresses, then apply the project's local-part rule: everything after the first `+`, with an empty suffix rejected. + +`mailparse` is intentionally tolerant of real Maildir data and states that it may accept non-strict input such as LF-only messages ([upstream README](https://github.com/staktrace/mailparse#api)). That matches the requirement to omit and log malformed messages rather than fail the mount, but it means tests—not a claim of strict rejection—must define the accepted edge cases. + +Read only until the first blank line and impose a configurable maximum header size before parsing. This prevents a malformed or hostile message with no terminator from causing unbounded allocation. Never parse or retain message bodies during indexing; reads from the mounted file should stream from the source file. + +### Direct `inotify` + +The source is a Linux Maildir with a small, known directory set (`cur` and `new`; `tmp` is ignored), so the cross-platform normalization of a higher-level watcher is unnecessary. The Rust `inotify` crate is an idiomatic wrapper that offers blocking reads and exposes the kernel event mask, filename, and rename cookie ([crate documentation](https://docs.rs/inotify/0.11.4/inotify/), [`Event`](https://docs.rs/inotify/0.11.4/inotify/struct.Event.html)). Run it on one dedicated thread and submit compact change commands to the state coordinator. + +Watch both `cur` and `new` for create, close-write, delete, and move events. The Linux API gives matching `IN_MOVED_FROM`/`IN_MOVED_TO` events a shared cookie, including moves between watched directories ([inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html#DESCRIPTION)). Do not assume the pair is adjacent or atomically enqueued; the kernel documentation explicitly warns that matching rename events is racy and one side may be absent ([inotify rename caveats](https://man7.org/linux/man-pages/man7/inotify.7.html#Dealing_with_rename()_events)). Treat unmatched or timed-out rename halves as removal/addition and reconcile the affected directories. + +`IN_Q_OVERFLOW` means events were dropped, and the Linux documentation recommends rebuilding the cache when this happens. Inotify is non-recursive and can miss changes on remote/network filesystems ([inotify limitations](https://man7.org/linux/man-pages/man7/inotify.7.html#NOTES)). Therefore: + +- reject or clearly document unsupported network-backed source Maildirs in version 1; +- on overflow, watch invalidation, an unmatched event, or any stat/index contradiction, keep serving the last complete snapshot, rebuild from `cur` and `new`, then publish the new snapshot atomically; +- periodically run a low-frequency reconciliation as defense in depth, because the kernel manual recommends consistency checking even for carefully written monitors ([inotify cache guidance](https://man7.org/linux/man-pages/man7/inotify.7.html#DESCRIPTION)). + +### SQLite as a disposable, persistent index + +The source Maildir is authoritative. Store only derived metadata: source identity and current relative path, stat fingerprint, parsed tag values, reversible projected names, stable projected inode assignments, and the parser/schema version. Do not store message bodies. A normalized `messages`, `tags`, and `message_tags` schema supports 100,000 messages and 10,000 tags without loading bodies and permits indexed consistency checks. + +Use one `rusqlite::Connection`, owned exclusively by the state coordinator. FUSE read callbacks consult an immutable in-memory snapshot rather than querying SQLite. This makes request latency independent of database locks, while the database preserves parser results and stable inode assignments between restarts. `rusqlite` normally finds a system SQLite through `pkg-config`; its `bundled` feature instead compiles an embedded SQLite copy ([upstream build notes](https://github.com/rusqlite/rusqlite#notes-on-building-rusqlite-and-libsqlite3-sys)). For Nix, prefer the system library so Nixpkgs controls security fixes and there is only one SQLite version in the closure. + +Use SQLite's rollback-journal mode (`journal_mode=DELETE`), `synchronous=FULL`, `foreign_keys=ON`, and explicit transactions. SQLite documents that transactions appear atomic even across an operating-system crash or power failure ([SQLite atomic commit](https://sqlite.org/atomiccommit.html#_introduction)). WAL's extra reader/writer concurrency is unnecessary because only the coordinator touches the database; WAL also adds checkpoint management and extra `-wal`/`-shm` files ([SQLite WAL overview](https://sqlite.org/wal.html#_overview)). If a later design introduces multiple SQLite connections and WAL, require a SQLite release containing the 2026 WAL-reset fix (3.51.3 or the documented backports) and test busy/checkpoint behavior ([SQLite WAL-reset advisory](https://sqlite.org/wal.html#walreset)). + +At startup: + +1. Open/migrate the database. +2. Scan `cur` and `new` completely. Reuse cached parser results only when the stored fingerprint still matches; otherwise re-read and parse the bounded header block. +3. Reconcile all rows inside a transaction and construct the complete immutable in-memory snapshot. +4. Only after commit and snapshot construction succeeds, mount FUSE and signal systemd readiness. + +For an incremental event or FUSE mutation, the state coordinator performs the authoritative Maildir rename/unlink first, commits the corresponding index change second, publishes one replacement snapshot third, and finally sends kernel invalidations. A crash can occur between the source operation and the index commit, so cross-filesystem atomicity is not claimed; the next reconciliation repairs the disposable index from the authoritative Maildir. + +## Concurrency and component boundary + +Use this ownership model: + +```text +fuser worker callbacks ----\ + > bounded command queue -> state coordinator -> source Maildir +inotify watcher thread ----/ | \ + | -> SQLite connection + v + immutable view snapshot + | + fuser readers + Notifier +``` + +- FUSE lookup/read/readdir/getattr operations take a short-lived read handle to the current immutable snapshot and access source files through validated file handles. +- FUSE rename/unlink/setattr operations send a command and wait for the coordinator's result before replying to the kernel. This serializes global flag and deletion semantics across every projected tag path. +- The watcher uses the same command queue, so external Dovecot/Postfix changes and writes through the projection have one ordering point. +- Use a bounded channel for backpressure; Rust's `sync_channel` is explicitly bounded and supports cloned producers with one receiver ([`sync_channel`](https://doc.rust-lang.org/std/sync/mpsc/fn.sync_channel.html)). If it fills or the watcher cannot submit without jeopardizing liveness, mark the state dirty and schedule reconciliation rather than silently dropping semantic work. +- Do not hold the snapshot lock or a FUSE inode lock while waiting for the coordinator or sending a kernel notification. This rule must be covered by concurrency tests. + +The immutable snapshot should contain compact identifiers, paths, flags, file size/timestamps, tag-to-message membership, and directory enumeration order. Memory is therefore proportional to messages plus tag memberships, not message bytes. Verify the 100,000-message/10,000-tag target with a generated corpus and measure startup time, steady-state resident memory, concurrent `readdir`, flag-renames, and deletion. + +## Nix and systemd implications + +Package with a checked-in `Cargo.lock` and Nixpkgs `rustPlatform.buildRustPackage`; link `rusqlite` to the Nix-provided SQLite and leave `fuser`'s `libfuse` feature disabled. The runtime still needs Linux FUSE support and appropriate mount setup. Current NixOS releases make the `programs.fuse` module opt-in, so the NixOS module must enable/provide the required `fusermount3` tooling or mount with the service's explicitly granted privileges ([NixOS release note](https://nixos.org/manual/nixos/unstable/release-notes#sec-release-26.11)). + +The service should use a per-instance systemd state directory for the SQLite file, start only after source and mountpoint paths exist, remain in the foreground, log to stderr/journald, unmount on orderly shutdown, and report ready only after startup reconciliation and successful mount. systemd/NixOS already models ordering between services and mount units, and custom NixOS services can declare `after`, `requires`, `before`, and `wantedBy` relationships ([NixOS service manual](https://nixos.org/manual/nixos/stable/#sec-systemd)). + +## Viable alternatives considered + +### `fuse3` (Rust async FUSE) + +`fuse3` 0.9.0 is actively released and supports unprivileged `fusermount3`, `readdirplus`, POSIX locks, and async direct I/O; upstream still lists `ioctl` and fuseblk as unsupported and `poll` as unstable ([upstream README](https://github.com/Sherlock-Holo/fuse3)). It is viable, but its async-first API and runtime are unnecessary for a local Maildir/SQLite workload whose mutations are deliberately serialized. At the examined release, docs.rs also failed to build its 0.9.0 API documentation ([docs.rs release page](https://docs.rs/crate/fuse3/0.9.0)), increasing adoption risk. Reconsider it if measurements show that blocking FUSE workers are a bottleneck or the backend later becomes remote. + +### `go-fuse/v2` (Go) + +`go-fuse/v2` is the strongest language-level alternative. It has raw and higher-level inode APIs, normally handles each request in its own goroutine, supports kernel invalidation, and its higher-level API explicitly supports hard links and immutable node IDs ([raw package](https://pkg.go.dev/github.com/hanwen/go-fuse/v2/fuse), [higher-level `fs` package](https://pkg.go.dev/github.com/hanwen/go-fuse/v2/fs)). Choose it instead if the maintainers are substantially stronger in Go. It is not selected because the Rust stack already supplies the exact header-only parser and direct inotify APIs required, avoids a garbage-collected runtime in a long-lived metadata-heavy daemon, and allows one language-level ownership model across borrowed input, inode state, and FUSE callbacks. This is a preference, not a claim that `go-fuse` is incapable. + +### C/C++ with libfuse + +libfuse is the reference userspace implementation and offers a synchronous high-level API plus an asynchronous low-level API ([libfuse API](https://libfuse.github.io/doxygen/)). It is technically complete for this task. It is rejected because implementing the same parser, concurrent inode state, and crash-safe ownership model in a memory-unsafe language adds avoidable risk, while `fuser` exposes the required low-level operations and cache notifications. libfuse's own README also says it currently has no active regular contributors beyond a maintainer applying pull requests and making releases, so using the C reference does not remove upstream-maintenance risk ([libfuse README](https://github.com/libfuse/libfuse#development-status)). + +### `mail-parser` instead of `mailparse` + +Stalwart's `mail-parser` is a strong maintained parser: upstream describes RFC 5322/MIME coverage, zero-copy strings, fuzzing, and use against millions of messages ([crate documentation](https://docs.rs/mail-parser/0.11.5/mail_parser/)). It is preferable for applications that need a rich parsed MIME model. This daemon needs repeated arbitrary header lookup and header-only address parsing; `mailparse::parse_headers`, `get_all_headers`, and `addrparse_header` directly match that narrower job. Keep corpus tests parser-neutral so this choice can be revisited if `mailparse` fails on deployed messages. + +### `notify` instead of direct `inotify` + +`notify` is maintained and its Linux backend translates queue overflow into a `Rescan` event ([backend source](https://docs.rs/notify/latest/src/notify/inotify.rs.html)). It is the right default for cross-platform applications. This service is explicitly Linux-only and benefits from direct access to rename cookies and exact overflow/watch-invalidated masks, so the lower-level `inotify` wrapper is simpler at the semantic boundary despite requiring more careful code. + +### No persistence, flat files, or an embedded key-value store + +An in-memory-only index would force reparsing every message on every boot. A custom JSON/CBOR snapshot can be atomically replaced, but schema migration, referential checks, partial-update handling, and indexed lookups become application code. RocksDB is far heavier than the dataset and introduces a C++ dependency. SQLite supplies transactions, integrity checks, compact indexes, and mature recovery in one system package; treating it as a rebuildable cache limits the consequence of any database failure. + +## Risks and acceptance work + +1. **Dovecot exercises Maildir behavior, not a crate API.** Add black-box tests with Dovecot against the mounted filesystem for `new` to `cur` transitions, `S/F/R/T/D` flag renames, concurrent views of the same source message, copy-then-delete moves, expunge, restart, and forced unmount. +2. **FUSE cache invalidation can make correct state look stale.** Test notifier calls and TTL behavior under an already-open Dovecot indexer; never publish new state without invalidating every affected projected dentry/inode. +3. **Duplicate projected paths can race.** Route all mutations through the coordinator and make operations idempotent. A second delete should return the documented stale/not-found error, not delete an unrelated reused inode. +4. **Inotify is advisory, not a transaction log.** Test queue overflow, missing rename halves, watch deletion, burst delivery, and changes while reconciliation is scanning. Every such path must converge through a full reconciliation. +5. **Parser tolerance is part of observable behavior.** Maintain a corpus containing folded headers, repeated delivery headers, address groups, comments, quoted local parts, UTF-8/encoded display names, LF-only messages, malformed addresses, missing separators, and oversized headers. Only the extracted routing address—not the display name—feeds tag extraction. +6. **Maildir and SQLite cannot commit atomically together.** Preserve the source-first/disposable-index rule and inject crashes between each step to prove restart repair. +7. **Upstream churn remains possible.** Pin the lockfile, run dependency/security updates routinely, and isolate `fuser`, parser, watcher, and storage adapters. The complete Dovecot integration suite is the upgrade gate. +8. **The scale target needs measurement.** Acceptance requires 100,000 messages and 10,000 distinct tags, bounded headers, no body cache, and recorded startup latency/RSS. A complexity argument alone is insufficient. + +## Resulting implementation constraint + +The implementation specification should name this stack and ownership model, but should not expose crate types in the domain model. The stable internal seam is: a header scanner yields raw tags; a source catalog yields authoritative messages; an index persists derived metadata; an immutable projection answers reads; and one mutation coordinator owns all source/index changes. That boundary keeps the route open if any selected library needs replacement.