Privacy & PII
Personal data going into AI systems, coming back out of them, and the fact that a trained model is very hard to un-train.
When not to use it
- Anonymisation as a compliance checkbox. Re-identification from a handful of attributes is well documented; "we removed the names" is not a defence.
- Sending regulated data to a third-party model without a contract covering retention and training use. This is a procurement question and it kills projects late.
- Fine-tuning on personal data without an erasure plan. You're creating an obligation you may not be able to meet.
Reach for something else instead
- Masking or tokenisation before the model sees anything — the data that never arrives can't leak.
- Self-hosted or on-device models when the data genuinely cannot leave, and the cost is worth it.
- Not using the personal data at all. Often the field wasn't load-bearing and nobody checked.
Sources & further reading
- Carlini et al. (2021), Extracting Training Data from Large Language Models — verbatim memorisation, demonstrated.
- Shokri et al. (2017), Membership Inference Attacks Against Machine Learning Models — determining whether a record was in the training set.
- Narayanan & Shmatikov (2008), Robust De-anonymization of Large Sparse Datasets — why anonymisation is weaker than it sounds, established long before this era.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Securing the source system and forgetting the vector store built from it. It's a full copy with the same obligations and usually none of the controls.
- Logging prompts for debugging and creating an undeclared PII store.
- Assuming deletion from the database means deletion from the model. It doesn't, and unlearning is unsolved.
At a glance
FieldSafety & Ethics
Core ideapersonal data in, and out
Hard problemmodels memorise, unlearning is unsolved
Best controldon't send it
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Privacy in RAG vs. fine-tuning — a retrievable store you can delete from vs. weights you largely cannot.