PII redaction

The choice between the two forms is the first real decision and it is usually made by accident. Masking a card number to [REDACTED:CREDIT_CARD] is safe and lossy: a downstream reader loses the ability to tell two cards apart. Tokenising it to a placeholder that the same value always maps to within one conversation preserves referential structure — a model can still reason that the address in turn one and the address in turn six are the same customer — at the cost of holding a mapping somewhere. The rule that follows is not optional: secrets must be masked irreversibly even in tokenising mode, because a reversible placeholder for a live credential is a credential leak with extra steps, and the placeholder is the part that survives into logs.

Detection is pattern matching plus arithmetic, and the arithmetic is what makes it usable. A regular expression that matches sixteen digits matches order numbers, timestamps and part codes; the same expression with a Luhn check attached matches card numbers. Checksum-validated kinds therefore carry high confidence — Luhn for a payment card, mod-97 over the digit expansion for an IBAN, the weighted mod-11 check for a UK NHS number — while pattern-only kinds such as a US Social Security number or a UK National Insurance number carry lower confidence because nothing can be verified beyond shape. Credential shapes sit at the top of the range for a different reason: prefixed vendor key formats and PEM private-key headers are distinctive enough that a false positive is nearly impossible. Confidence should be exposed as a number rather than collapsed to a boolean, so a policy can treat a 0.7 phone-number guess and a 0.98 checksum-validated IBAN differently, which they deserve.

Two implementation details decide whether a redactor is usable in production, both invisible until they bite. The first is overlap: detectors must run most-specific-first and claim their spans, or a validated card number is also matched as a phone number and one of the two rewrites corrupts the other. The second is boundary precision, and it is worth a concrete illustration, because redaction rewrites the text a model then reasons over. In Token Observe’s detector set, an IBAN pattern that allowed its match to end on a trailing separator swallowed the space that followed the value, and the redacted prompt handed to the model read '<IBAN_1>for the payout' — a silent edit to the meaning of a sentence, produced by a control whose entire job was to be transparent. A redactor’s off-by-one errors are not cosmetic; they change what the model was asked.

Streaming is where most implementations quietly fail, because bytes already written cannot be recalled. A value that has only half arrived matches nothing — a JWT matches no pattern at all until its third segment lands — so a scanner that emits each chunk as it arrives will happily emit the first half of a credential and then redact the second. The fix is a hold-back buffer that never cuts inside an unbroken run of value characters, with a declared maximum run length; past that maximum the run is replaced wholesale and suppressed to its delimiter, which trades the ability to name the kind for the guarantee that neither half is emitted. Any implementation that cannot describe its hold-back behaviour is one that has not thought about the streaming case.

The limit is large, specific, and belongs in the same paragraph as the claim. Pattern-and-checksum detection does not see free-text personal data at all. 'Mrs Patel from the Leeds branch rang about her mortgage arrears' is personal data under any reading of the law and matches nothing, and identifier formats outside whichever national set was shipped — every country’s tax, health and identity number scheme not on the list — are equally invisible. That makes inline redaction a compensating control rather than a data-loss-prevention layer, and the honest framing is that it reliably catches structured identifiers and credentials and reliably misses narrative. Be precise about the legal vocabulary, too, because the words are routinely swapped: keeping a mapping from placeholder back to value is pseudonymisation, and pseudonymised data remains personal data. Only irreversible removal, with no retained key, is anonymisation.