Choose the Linux FUSE and message-parsing implementation stack #3

New issue

Closed

opened 2026-07-22 22:12:33 +00:00 by phg · 2 comments

phg commented

2026-07-22 22:12:33 +00:00

Owner

Part of #1

Question

Against official documentation and upstream source, which maintained implementation language, Linux FUSE binding, RFC-message/address parser, filesystem-watching facility, and persistence approach best fit the required writable projection, concurrency model, near-real-time updates, 100,000-message scale, Nix packaging, and long-running systemd operation? Record a recommendation with rejected alternatives and risks.

Part of #1 ## Question Against official documentation and upstream source, which maintained implementation language, Linux FUSE binding, RFC-message/address parser, filesystem-watching facility, and persistence approach best fit the required writable projection, concurrency model, near-real-time updates, 100,000-message scale, Nix packaging, and long-running systemd operation? Record a recommendation with rejected alternatives and risks.

phg added the

wayfinder:research

label 2026-07-22 22:12:33 +00:00

phg referenced this issue

2026-07-22 22:13:01 +00:00

Validate the writable tag namespace with a concrete prototype #6

phg referenced this issue

2026-07-22 22:13:01 +00:00

Define synchronization, startup, and performance requirements #8

phg referenced this issue

2026-07-22 22:13:02 +00:00

Define the standalone CLI and NixOS module interface #9

phg referenced this issue

2026-07-22 22:13:03 +00:00

Define verification and release acceptance criteria #12

phg commented

2026-07-22 22:13:48 +00:00

Author

Owner

Claimed by Codex research agent /root/research_fuse_stack for resolution on branch research/linux-fuse-message-stack.

phg commented

2026-07-22 22:21:44 +00:00

Author

Owner

Resolution

Choose stable Rust with:

fuser 0.17 (default/no-libfuse build) for the inode-oriented Linux FUSE server and kernel cache invalidation;
mailparse 0.16.1 for bounded, header-only parsing and address parsing on every repeated delivery/visible-recipient header;
direct inotify on a dedicated thread for exact move cookies and overflow/watch failure handling;
rusqlite linked to the Nix-provided SQLite, using one coordinator-owned connection in rollback-journal mode;
a bounded multi-producer/single-consumer mutation coordinator and immutable in-memory snapshots for hot-path reads.

The source Maildir remains authoritative. SQLite is a disposable metadata/inode cache: startup performs a complete reconcile before mounting, and overflow or inconsistency rebuilds while the last complete snapshot remains served. FUSE writes and external watcher events share the coordinator; source rename/unlink happens before the index transaction, with reconciliation healing any crash gap. Message bodies are neither parsed nor cached.

The strongest rejected alternative is Go with go-fuse/v2; it remains viable for a Go-strong team. Rust fuse3 adds an unnecessary async runtime and had failed 0.9.0 docs generation; C/libfuse adds avoidable memory-safety complexity; mail-parser is richer than the header-only job; cross-platform notify hides Linux details this design needs. Principal risks are Dovecot/FUSE cache semantics, duplicate-path races, inotify loss/races, tolerant parser edge cases, and the unavoidable Maildir/SQLite crash gap; the findings prescribe black-box, overflow, corpus, crash-injection, and 100,000-message/10,000-tag scale tests.

Evidence and full trade-off analysis:

Branch: research/linux-fuse-message-stack
Commit: b6048de04f6996226fbba070b9919edb1c37c0db
Findings: docs/research/linux-fuse-message-stack.md

## Resolution Choose stable Rust with: - `fuser` 0.17 (default/no-libfuse build) for the inode-oriented Linux FUSE server and kernel cache invalidation; - `mailparse` 0.16.1 for bounded, header-only parsing and address parsing on every repeated delivery/visible-recipient header; - direct `inotify` on a dedicated thread for exact move cookies and overflow/watch failure handling; - `rusqlite` linked to the Nix-provided SQLite, using one coordinator-owned connection in rollback-journal mode; - a bounded multi-producer/single-consumer mutation coordinator and immutable in-memory snapshots for hot-path reads. The source Maildir remains authoritative. SQLite is a disposable metadata/inode cache: startup performs a complete reconcile before mounting, and overflow or inconsistency rebuilds while the last complete snapshot remains served. FUSE writes and external watcher events share the coordinator; source rename/unlink happens before the index transaction, with reconciliation healing any crash gap. Message bodies are neither parsed nor cached. The strongest rejected alternative is Go with `go-fuse/v2`; it remains viable for a Go-strong team. Rust `fuse3` adds an unnecessary async runtime and had failed 0.9.0 docs generation; C/libfuse adds avoidable memory-safety complexity; `mail-parser` is richer than the header-only job; cross-platform `notify` hides Linux details this design needs. Principal risks are Dovecot/FUSE cache semantics, duplicate-path races, inotify loss/races, tolerant parser edge cases, and the unavoidable Maildir/SQLite crash gap; the findings prescribe black-box, overflow, corpus, crash-injection, and 100,000-message/10,000-tag scale tests. Evidence and full trade-off analysis: - Branch: [`research/linux-fuse-message-stack`](https://git.s1q.dev/phg/fuse-mail-dir-by-tag/src/branch/research/linux-fuse-message-stack) - Commit: [`b6048de04f6996226fbba070b9919edb1c37c0db`](https://git.s1q.dev/phg/fuse-mail-dir-by-tag/commit/b6048de04f6996226fbba070b9919edb1c37c0db) - Findings: [`docs/research/linux-fuse-message-stack.md`](https://git.s1q.dev/phg/fuse-mail-dir-by-tag/src/commit/b6048de04f6996226fbba070b9919edb1c37c0db/docs/research/linux-fuse-message-stack.md)

phg closed this issue

2026-07-22 22:21:44 +00:00