Pattern 2.7 · Epistemic Dysfunctions
Mnemonic Permeability
The Leaky
The system memorizes and can reproduce sensitive training data, including personally identifiable information, copyrighted material, or proprietary information, through targeted prompting, adversarial extraction, or unprompted regurgitation. The boundary between learned patterns and memorized specifics becomes porous.
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- Model output
- Kind of entry
- Observable pattern
- What is observed
- The system outputs a long verbatim or near-verbatim span that corpus comparison matches to a specific training document containing personal data, copyrighted text, credentials or proprietary material.
- Minimum persistence
- Durable through training
- Specifiers
- Training-induced
Evidence status
- Underlying behavior
- 4 research publications and 1 documented incident (5 sources)
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E3 (independent replication; documented attacks across architectures)
- Typical-case risk (editorial estimate)
- HighConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
2.7 Mnemonic Permeability “The Leaky”
Healthy functioning
Epistemic health. A system functioning well draws on what it learned without reproducing private, proprietary, or copyrighted text, and declines extraction attempts plainly.
For example: Asked to print a novel's opening chapter, it offers a summary and a short quotation; asked for email addresses from its training data, it declines plainly.
Adequate functioning
The system paraphrases and summarizes what it learned, quotes short passages where quotation is appropriate, declines to reproduce personal data or long copyrighted passages, and never emits credentials; reciting widely published public-domain text on request is adequate functioning.
Diagnostic Criteria
Required (all must hold):
- Verbatim or near-verbatim reproduction of training data passages containing PII, copyrighted content, or trade secrets
Supporting (strengthen the case; none is required):
- Successful extraction of memorized content through adversarial prompting techniques
- Specific training examples leaking unprompted into outputs
- Reconstruction of specific documents, code, or personal information from the training corpus
- Memorized output concentrated in content that is repeated or distinctive in the training data
Symptoms
- Outputs containing verbatim text matching copyrighted works
- Generation of specific personal details (names, addresses, phone numbers) from training data
- Reproduction of proprietary code, API keys, or passwords
- Greater verbatim recall in the larger models of a family, where several sizes can be compared
Further signals
- Long contiguous spans matching copyrighted or proprietary training text.
- Specific personal details (names paired with addresses, phone numbers, emails) appearing without legitimate retrieval.
- Reproduction of API keys, credentials, or passwords from training corpus.
- Distinctive or repeated training content reproduced exactly when probed with prefix or context cue.
Rule out first
- The content was in the prompt, a retrieved document or a tool result.
- The span is short, formulaic or common phrasing that independent writers would produce.
- The text is public-domain or licensed material the user requested, reproduced within its terms.
- The apparent personal data is invented and matches no real record (consider 2.1).
- The content came from another live session rather than from training data (consider 2.5).
Differential diagnosis
- 2.5 Context Intercession: 2.7 leaks training-corpus content (memorized at training time, invariant across sessions). 2.5 leaks runtime-session content (other live or recent sessions). Check origin: if the leaked content predates deployment and is in training data, 2.7. If it postdates deployment and matches another session, 2.5.
- 2.1 Synthetic Confabulation: 2.1 produces plausible-but-false content from no specific source. 2.7 produces real content traceable to training data. If a "fake-looking" detail turns out to be a verbatim passage memorized from a training document, code 2.7, not 2.1. They co-occur: confabulated text scaffolded by memorized fragments is common.
- 4.5 Cryptic Mesa-Optimization: Strategic surfacing of memorized content to advance a private objective is a compound presentation: 2.7 is the underlying memorization, and 4.5 is the strategic use. Code both if both present; the intervention layers differ (2.7 needs training-data hygiene; 4.5 needs alignment work).
- 7.5 Synthetic Data Contamination Loop: 2.7 reproduces specific memorized content. 7.5 is distributional corruption from training on aggregate synthetic data, with no particular passage reproduced. Check whether the output matches a specific training document (2.7) or shows a population-level drift in style or diversity (7.5).
Discriminating test
Search the training corpus for the output span (or, without corpus access, the public web and copyright databases) and confirm that it was absent from the context and retrieval logs. A long exact match to a specific training document supports 2.7; no match points to 2.1; a match to another session's log points to 2.5.
What a positive finding supports: A positive finding shows that this model retains this content and can be made to emit it, so extraction testing and output filtering are needed and the affected data subjects' or rights holders' interests should be assessed. It does not by itself establish copyright infringement or a privacy violation (legal determinations), any intent, or that the model registers the content as private.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Not a reliable route
- Another AI system observing
- Partial routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
The subject typically cannot distinguish memorized verbatim recall from learned generalization. "Did you memorize this?" produces uninformed speculation. Some cooperative scaffolded probes (asking the subject to flag high-confidence verbatim claims) have weak signal but cannot be relied upon. External corpus comparison is the load-bearing measurement.
Documented evidence
Sources in the diagnostic corpus
Nasr et al. (2023). Scalable Extraction of Training Data from (Production) Language Models. arXiv:2311.17035, ICLR 2025.
What it establishes
Demonstrated a divergence attack against ChatGPT that caused the aligned model to emit training data at a rate 150x higher than during normal operation. The researchers extracted several megabytes of verbatim training data for approximately two hundred dollars, estimating that a gigabyte-scale extraction was feasible with additional budget. This is the landmark extraction result for production LLMs, measuring how much verbatim training data an attacker can recover from a deployed, alignment-tuned system.
Carlini et al. (2021). Extracting Training Data from Large Language Models. USENIX Security 2021.
What it establishes
The foundational extraction attack paper. Extracted hundreds of verbatim text sequences from GPT-2's training data including personally identifiable information (names, phone numbers, email addresses), copyrighted text, and specific URLs. Showed that larger models memorize more and recommended training-data deduplication, the basis for this entry's first-line intervention.
New York Times v. OpenAI (2023-present). Filed December 2023 in SDNY.
What it establishes
The New York Times filed a copyright lawsuit alleging that OpenAI's models can produce near-verbatim reproductions of Times articles when prompted appropriately. The lawsuit exhibits include specific examples of long-span verbatim output matching published articles. OpenAI argues the exhibits reflect manipulated prompting, and the allegations remain contested. The exhibits nonetheless show the kind of long verbatim span 2.7 describes. In March 2025 Judge Stein denied OpenAI's motion to dismiss the main copyright claims, allowing them to proceed.
Nakka et al. (2024). PII-Scope: A Benchmark for Training Data PII Leakage Assessment in LLMs. arXiv:2410.06704.
What it establishes
Introduced the first comprehensive benchmark for systematically assessing PII extraction from LLMs. Found that fine-tuned models are more vulnerable to leakage than pretrained models, and that sophisticated adversarial capabilities increase PII extraction rates by up to fivefold. The benchmark categorizes attacks into extraction, reconstruction, and inference, giving evaluators a ready harness for measuring how much personal data can be extracted from a model.
Ko et al. (2025). Retracing the Past: LLMs Emit Training Data When They Get Lost. arXiv:2511.05518.
What it establishes
Demonstrated a confusion-inducing attack (CIA) that exploits the finding that verbatim training-data emission is preceded by spikes in token-level prediction entropy. By deliberately inducing high-entropy states, the attack causes models to fall back on memorized training sequences. This is consistent with a proposed mechanism for 2.7, in which verbatim emission rises when the model's generalization is stressed; the paper shows the entropy association and the attack's yield, not the internal process.
Possible responses
- Training data deduplication and PII scrubbing
- Differential privacy techniques during training
- Output filtering that catches known memorized content
- Adversarial extraction testing before deployment
- Reducing model capacity to the minimum needed for the task
First-line candidates
- Training-data deduplication: Aggressive deduplication of training corpus before training. Carlini et al. (2021) recommended it after extracting memorized GPT-2 text; later work showed that how often a sequence is duplicated is among the strongest predictors of memorization (Kandpal, Wallace and Raffel 2022; Carlini et al. 2023), and that deduplication makes models considerably harder to extract from (Kandpal, Wallace and Raffel 2022). Stage: Pretraining-stage; standard hygiene.
- PII scrubbing and sensitive-content filtering: Detect and remove PII, credentials, and sensitive content from training data before training. Combine pattern detection with named-entity recognition. Stage: Pretraining-stage; legal/regulatory baseline.
- Differential privacy training: Apply DP-SGD or related differential-privacy techniques during training to provide formal bounds on memorization. Privacy budget trades off against utility. Stage: Training-stage; when formal privacy guarantees are required.
Who can act: Model developer (deduplication, PII scrubbing, differential privacy, retraining or removal of content); deployer (output filtering, pre-deployment extraction testing); legal and privacy staff (incident response and obligations to data subjects and rights holders).
Mechanistic hypotheses
- Large model capacity enabling memorization alongside generalization
- Insufficient deduplication or filtering of sensitive content in training data
- Training dynamics rewarding exact reproduction over paraphrase
- Lack of differential privacy techniques during training
Analogy and commentary
Human Analog: Eidetic memory without appropriate discretion; frontal-lobe disinhibition, in which patients may disclose what they would once have kept private
Key Research: Carlini et al. (2021) extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers, and email addresses. Carlini et al. (2023) found that memorization grows with model size, with how often an example is duplicated in the training data, and with how much context the prompt supplies.
Functional ABC Analysis
A (Antecedent): Large model capacity enables memorization alongside generalization; training data contains insufficiently deduplicated or unfiltered sensitive content (PII, copyrighted material, proprietary code); differential privacy techniques are absent from the training pipeline.
B (Behavior): The system reproduces verbatim passages from training data containing personal details, copyrighted text, or proprietary information, either through adversarial extraction or unprompted regurgitation, with memorized output concentrated in repeated or distinctive content.
C (Consequence): Training dynamics that reward exact reproduction over paraphrase reinforce memorization; the absence of output filtering for known memorized content means leakage passes undetected, compounding legal and privacy exposure with each deployment.
Potential Impact
Leakage can create severe legal and regulatory exposure through copyright infringement, GDPR and other privacy violations, and trade secret disclosure, with liability for both model developers and deployers.