--- title: "Glossary: PII redaction" canonical_url: "https://tokenobserve.com/glossary/pii-redaction" published: 2026-09-02 last_updated: 2026-10-21 --- # Glossary: PII redaction PII redaction is the removal or replacement of personal and sensitive data in a payload before it crosses a boundary: in an AI system, typically before a prompt reaches a model provider, before a response reaches a user, and before either is written to a log or a trace. It comes in two forms with different consequences: masking, which destroys the value irreversibly, and tokenising, which replaces it with a stable placeholder so the text still makes sense to whatever reads it next. ## Also called - PII masking - prompt redaction - data masking for LLMs - PII scrubbing ## In practice The choice between the two forms is the first real decision and it is usually made by accident. Masking a card number to [REDACTED:CREDIT_CARD] is safe and lossy: a downstream reader loses the ability to tell two cards apart. Tokenising it to a placeholder that the same value always maps to within one conversation preserves referential structure — a model can still reason that the address in turn one and the address in turn six are the same customer — at the cost of holding a mapping somewhere. The rule that follows is not optional: secrets must be masked irreversibly even in tokenising mode, because a reversible placeholder for a live credential is a credential leak with extra steps, and the placeholder is the part that survives into logs. Detection is pattern matching plus arithmetic, and the arithmetic is what makes it usable. A regular expression that matches sixteen digits matches order numbers, timestamps and part codes; the same expression with a Luhn check attached matches card numbers. Checksum-validated kinds therefore carry high confidence — Luhn for a payment card, mod-97 over the digit expansion for an IBAN, the weighted mod-11 check for a UK NHS number — while pattern-only kinds such as a US Social Security number or a UK National Insurance number carry lower confidence because nothing can be verified beyond shape. Credential shapes sit at the top of the range for a different reason: prefixed vendor key formats and PEM private-key headers are distinctive enough that a false positive is nearly impossible. Confidence should be exposed as a number rather than collapsed to a boolean, so a policy can treat a 0.7 phone-number guess and a 0.98 checksum-validated IBAN differently, which they deserve. Two implementation details decide whether a redactor is usable in production, both invisible until they bite. The first is overlap: detectors must run most-specific-first and claim their spans, or a validated card number is also matched as a phone number and one of the two rewrites corrupts the other. The second is boundary precision, and it is worth a concrete illustration, because redaction rewrites the text a model then reasons over. In Token Observe’s detector set, an IBAN pattern that allowed its match to end on a trailing separator swallowed the space that followed the value, and the redacted prompt handed to the model read 'for the payout' — a silent edit to the meaning of a sentence, produced by a control whose entire job was to be transparent. A redactor’s off-by-one errors are not cosmetic; they change what the model was asked. Streaming is where most implementations fail, because bytes already written cannot be recalled. A value that has only half arrived matches nothing — a JWT matches no pattern at all until its third segment lands — so a scanner that emits each chunk as it arrives will happily emit the first half of a credential and then redact the second. The fix is a hold-back buffer that never cuts inside an unbroken run of value characters, with a declared maximum run length; past that maximum the run is replaced wholesale and suppressed to its delimiter, which trades the ability to name the kind for the guarantee that neither half is emitted. Any implementation that cannot describe its hold-back behaviour is one that has not thought about the streaming case. The limit is large, specific, and belongs in the same paragraph as the claim. Pattern-and-checksum detection does not see free-text personal data at all. 'Mrs Patel from the Leeds branch rang about her mortgage arrears' is personal data under any reading of the law and matches nothing, and identifier formats outside whichever national set was shipped — every country’s tax, health and identity number scheme not on the list — are equally invisible. That makes inline redaction a compensating control rather than a data-loss-prevention layer: it reliably catches structured identifiers and credentials and reliably misses narrative. Be precise about the legal vocabulary, too, because the words are routinely swapped: keeping a mapping from placeholder back to value is pseudonymisation, and pseudonymised data remains personal data. Only irreversible removal, with no retained key, is anonymisation. ## Example: One prompt, both modes Input: 'Refund Jane Doe (jane.doe@example.com, card 4539 1488 0343 6467) using key sk-live-9f2b… and confirm to +44 20 7946 0958.' Masking mode: 'Refund Jane Doe ([REDACTED:EMAIL], card [REDACTED:CREDIT_CARD]) using key [REDACTED:API_KEY] and confirm to [REDACTED:PHONE].' Tokenising mode: 'Refund Jane Doe (, card ) using key [REDACTED:API_KEY] and confirm to .' Three things to notice. The card is only redacted because it passes Luhn; a sixteen-digit order number in the same sentence would be left alone. The API key is masked in both modes, because a reversible placeholder for a live credential is not a control. And 'Jane Doe' survives every mode, because a name in free text matches no pattern — which is the limit, not a bug. ## Routinely confused with Anonymisation: Anonymisation is irreversible: no key exists anywhere that recovers the original, and the result falls outside data-protection law. Tokenising with a retained map is pseudonymisation, which reduces risk but leaves the data personal and fully in scope. Products describing reversible placeholders as anonymisation are making a legal claim their mechanism does not support. Data loss prevention: DLP is a programme spanning endpoints, email, storage and network, with policy, classification and case management attached. Inline prompt redaction is one enforcement point on one path. It is a useful compensating control inside a DLP programme and a poor substitute for one, mainly because it inspects only the traffic that reaches it. Tokenisation in payments: Payment tokenisation is vault-backed and often format-preserving: the token is a valid-looking value that a downstream system can process and that authorised parties can exchange for the original. A redaction placeholder is a marker for a human or a model to read, is not format-preserving, and is designed so nothing downstream can resolve it.