phg/fuse-mail-dir-by-tag
0
0
Fork 0
Code Issues 10 Pull requests Projects Releases Packages Wiki Activity Actions
fuse-mail-dir-by-tag/docs/research/rfc-recipient-parsing.md

13 KiB
Raw Blame History

RFC recipient parsing boundaries

Research for Establish RFC recipient parsing boundaries.

Decision

Parse only the RFC message header section, preserve every occurrence of the four configured fields, and derive plus tags only from syntactically valid mailbox local-parts. Do not scan header text with regular expressions and do not salvage address-looking substrings from a field that failed complete syntactic validation. RFC 5322 makes the header section a sequence of fields terminated by the empty line, defines unfolding before semantic analysis, and distinguishes structured address fields from unstructured text (RFC 5322 §§2.1, 2.2.2–2.2.3).

Apply the already-agreed precedence as a two-tier algorithm:

  1. Parse every Delivered-To and X-Original-To occurrence and collect the plus tags from all valid occurrences.
  2. If that tag set is non-empty, return it and do not inspect To or Cc for tags.
  3. Otherwise, parse every To and Cc occurrence, flatten all mailbox members of all groups, and collect their plus tags.
  4. Deduplicate the selected tier's tags by exact equality of the semantic local-part suffix, preserving first-seen order internally for deterministic diagnostics and tests.

The fallback condition is therefore no valid plus tag from the delivery tier, not “delivery headers are absent” and not “delivery headers contain no valid recipient.” This makes an untagged or malformed delivery field fall back to visible recipients, while any valid delivery plus tag suppresses all visible-recipient tags.

Parsing contract

Header framing and occurrences

  • Field names are matched case-insensitively, and a folded field is unfolded by removing each CRLF immediately followed by whitespace before its structured value is parsed (RFC 5322 §§2.2–2.2.3). Folding is syntax, not a new field, so continuation lines must remain attached to the preceding occurrence.
  • Keep occurrences separate and retain document order. RFC 9228 intentionally permits a sequence of Delivered-To fields, one per delivery or address transformation, and says their relative order must not be changed (RFC 9228 §4). RFC 5322 normally limits To and Cc to one occurrence each, but its mandatory obsolete-input grammar permits repeated destination fields and says to interpret them as concatenated address lists (RFC 5322 §§3.6, 4.5.3).
  • Stop at the first empty line; never interpret address-like content in the message body. A header-only parser is sufficient. For example, mail-parser exposes parse_headers, which returns a message containing only headers, and header_values, which iterates every matching occurrence (mail-parser 0.11.5 MessageParser, Message::header_values).

Delivery tier

  • A Delivered-To occurrence is valid only when its complete unfolded field body, apart from the grammar's permitted FWS, is exactly one RFC 5322 addr-spec. The standardized field records one individual address and its grammar is FWS addr-spec FWS; a display name, angle-address, comma-list, or group is not valid in this field (RFC 9228 §4, RFC 5322 §3.4.1).
  • Treat each X-Original-To occurrence with the same one-complete-addr-spec rule. This is a project interoperability rule grounded in Postfix's contract: when original-recipient support is enabled, Postfix stores the original recipient address in this field, and its local delivery agent prepends one X-Original-To containing the recipient address given to Postfix (Postfix enable_original_recipient, Postfix local(8)).
  • Parse all occurrences from both names; neither name outranks the other. Multiple occurrences may produce multiple plus tags. A malformed occurrence contributes nothing but does not invalidate valid sibling occurrences.

Visible-recipient tier

  • Parse each complete To and Cc occurrence as RFC 5322 address-list, including the obsolete forms receivers are required to understand. An address is either a mailbox or a group; a group contains zero or more mailboxes. Extract only actual mailbox addr-spec nodes and ignore display names, group names, comments, and empty groups (RFC 5322 §§3.4, 3.6.3, 4).
  • Comments may nest and may contain escaped parentheses; folding may occur inside them. They are semantically separate from address data, while quoted pairs represent the escaped character without the backslash. These rules require a stateful grammar parser, not delimiter splitting (RFC 5322 §§3.2.1–3.2.2).
  • If one To or Cc occurrence does not parse completely as an address list, reject that whole occurrence and continue with other occurrences. This is the project's deterministic safety boundary: RFC 5322 requires parsers to understand its obsolete grammar and to handle still-malformed data without crashing, but does not require trusting partially recovered addresses (RFC 5322 §4).

Semantic local-part and plus tag

For each accepted addr-spec, obtain the parser's semantic local-part, not a substring found by searching the serialized address. RFC 5322 defines addr-spec = local-part "@" domain and permits local-parts in dot-atom, quoted-string, and obsolete forms; an @ can therefore occur as data inside a quoted local-part (RFC 5322 §3.4.1).

Canonicalize syntax only:

  • remove delimiting quotes;
  • resolve quoted pairs by removing the quoting backslash while retaining the quoted character;
  • remove comments and folding whitespace only where the grammar marks them as semantically invisible;
  • do not case-fold, Unicode-normalize, decode RFC 2047 encoded-words inside an addr-spec, or reinterpret the domain.

SMTP requires quoted forms of a local-part to compare equivalently, while preserving local-part case because it can be significant (RFC 5321 §§2.3.11, 4.1.2). RFC 2047 encoded-words are allowed in phrases and comments but explicitly forbidden in every part of an addr-spec, so an encoded-word-looking local-part is malformed rather than an alternate encoding of a plus tag (RFC 2047 §5).

Then split the semantic local-part at its first literal +:

  • no + means no tag;
  • an empty suffix means no tag;
  • otherwise the entire remaining suffix is one flat plus tag, including any later + characters;
  • an empty prefix is allowed, because + is an RFC 5322 atext character and this project has no base-recipient constraint (RFC 5322 §3.2.3).

SMTPUTF8 and identity

Accept a non-ASCII local-part only when the header bytes are valid UTF-8 and the address is valid under the RFC 6532 extensions. RFC 6532 extends atoms, quoted strings, and domains with UTF-8, while RFC 6531 leaves local-part delimiters and parsing otherwise unchanged (RFC 6532 §§3.1–3.2, RFC 6531 §§3.2–3.3). Invalid UTF-8 must never be lossily replaced because replacement could invent a tag or collapse two byte sequences.

Preserve the semantic suffix's exact Unicode scalar sequence and case. Do not normalize it during extraction or deduplication: internationalized-email guidance says unnormalized local-parts remain valid and warns that normalization behavior belongs to the destination system; local-parts also remain case-sensitive (RFC 6530 §10.1, RFC 5321 §2.4). Consequently Tag and tag, and canonically equivalent but differently encoded Unicode strings, are distinct plus tags. Quoted and unquoted syntactic forms that decode to the same semantic suffix are duplicates.

Malformed data and diagnostics

A field is malformed for this feature when header framing fails, UTF-8 is invalid where RFC 6532 syntax is needed, parsing does not consume the complete unfolded value, the value has the wrong top-level shape for its field, the addr-spec or its domain is incomplete, or an encoded-word occurs inside the addr-spec. Such a field contributes no recipient and no plus tag. The message remains available in the Source Maildir and other valid occurrences continue to be processed; emit a rate-limited diagnostic containing the source filename, field name/occurrence, and reason, but avoid logging the full address because delivery fields can disclose recipient and routing information (RFC 9228 §6).

Parser-library implication

mail-parser is a plausible header tokenizer for a Rust implementation: its documented API can parse headers only, configure arbitrary names such as Delivered-To and X-Original-To with header_address, retain repeated fields, and flatten groups through Address::iter (mail-parser 0.11.5 MessageParser, Address).

It is not, by itself, the validity oracle required above. Its public contract explicitly promises best-effort parsing of non-conformant messages and returns Option<Message> rather than per-field syntax errors; its address parser returns recovered Address/Empty values rather than a full-consumption Result (mail-parser crate contract, MessageParser::parse, address-parser source). This is an inference from its documented/source contract: an implementation may use it for framing and candidate AST extraction only if a strict wrapper validates complete field consumption, UTF-8, top-level shape, and semantic local-part decoding. Alternatively, choose a parser that exposes those checks directly. Whichever library is selected must pass the vectors below; no library-specific recovery behavior becomes part of the product contract.

Required acceptance vectors

Assume no other relevant fields unless shown. => lists the resulting plus tags.

Input condition Result
Delivered-To: mail+one@example and repeated X-Original-To: other+two@example one, two
Delivered-To: followed by CRLF + WSP + mail+folded@example folded
Delivered-To: plain@example; To: mail+visible@example visible (delivery tier produced no tag)
malformed Delivered-To: mail+bad@; valid To: mail+visible@example visible plus a delivery-field diagnostic
Delivered-To: mail+delivery@example; To: mail+visible@example delivery only
delivery fields absent; repeated To: mail+one@example and To: mail+two@example one, two (obsolete repeated-field input)
To: Friends: mail+one@example, mail+two@example; one, two
To: "sales+display" <mail+real@example> (comment+ignored) real
To: mail+real(comment+ignored)@example real after CFWS/comment semantics
To: "mail+quoted"@example quoted
To: "mail\+quoted"@example quoted (quoted-pair syntax is semantically invisible)
To: "a@b+tag"@example tag (the local-part's data @ is not the addr-spec separator)
To: mail+foo+bar@example the one flat tag foo+bar
To: +tag@example tag
To: mail+@example no tag
To: mail+same@example, other+same@elsewhere one same tag
To: "mail+same"@example, mail+same@example one same tag after semantic dequoting
To: mail+Tag@example, mail+tag@example distinct Tag and tag tags
To: δοκιμή+税@example as valid UTF-8/RFC 6532 税
same logical Unicode spelling in two different normalization forms two distinct tags
invalid UTF-8 in an otherwise address-looking local-part no tag and a diagnostic
RFC 2047 encoded-word used as a local-part no tag and a diagnostic
a syntactically bad To list containing one address-looking substring that occurrence contributes no tags; other valid occurrences still count
Delivered-To: Name <mail+tag@example> or a comma-separated delivery value no tag and a wrong-shape diagnostic

The repository's supplied sample exercises the ordinary Postfix/Dovecot path: both Delivered-To and X-Original-To contain the bare mail+analouge@philip-henning.com, so the selected set is exactly analouge; the duplicate visible To value is never consulted because the delivery tier is already non-empty.