How do I know if ChatGPT made up a source?
You open the links. A fabricated citation is fluent, correctly formatted, and frequently almost real — the right journal, a plausible title, an article number that does not exist. It is not detectable by reading; it is only detectable by fetching. In 26 AI answers we checked, 4 contained a citation that does not hold up — 15%. None of them looked wrong on the page.
What a made-up source actually looks like
People expect an invented citation to look invented. They do not. These are real examples, from real answers, with public receipts:
Why "it looked fine" is not evidence
The failure mode is specific: a model reproduces the shape of a citation it has seen thousands of times, without the identifier that points at a particular paper. The result passes every test a reader applies — right domain, right format, right subject — and fails the only test that matters, which is whether the page is there.
What a paywall is not
This is where most checking goes wrong. ScienceDirect returns 403 to automated requests. PubMed answers 203. A checker that counts every non-200 as a fake citation will tell you most of a sound bibliography is invented. Of 429 citations we opened, 82 were paywalled or bot-blocked — nearly one in five — and every one of them was real. A checker you cannot trust is worse than no checker, because you stop reading it.
How to check one yourself
- Open every link. Not the first one. The fabricated citation is rarely first.
- Look for a truncated identifier — a URL that ends at the journal prefix.
- Read the page, not the title. A live page that does not say what was claimed is a different failure, and a more common one.
- Treat 401/403/429 as "could not open", never as "fake".
Or have it checked
Whysaid opens every source an answer cites, classifies each one — real, missing, invented, truncated, or merely paywalled — and issues a public receipt you can link to. The receipt is the point: a verdict nobody can open proves nothing.