Here is a backup restoring. Read it carefully, because on the face of it there is nothing wrong with it.
$ pg_restore -U app -d verify --exit-on-error backup.dump $ echo $? 0 $ psql -U app -d verify -c "\dt" List of relations Schema | Name | Type | Owner --------+-------------+-------+------- public | users | table | app public | orders | table | app public | line_items | table | app (3 rows)
The archive is well-formed. pg_restore exits zero with --exit-on-error set. Every table is present, with the right names and the right schema.
There are no rows in any of them.
This backup would pass every check that most backup systems perform, and it is worth precisely nothing. If you restored it during an incident you would have a database shaped exactly like production, containing none of production.
How this happens
Not through corruption. Corruption is the easy case — a truncated file fails loudly and you find out immediately. This is the quiet one, and it arrives through ordinary mistakes:
- A
--schema-onlyor--no-dataflag copied from a StackOverflow answer about migrations and never removed. - A backup user with permission to read the schema but, through a row-level security policy or a missing
SELECTgrant, not the rows. - A dump that ran against the wrong database — an empty staging copy with identical structure.
- A container that restarted mid-dump, leaving the schema written and the data never reached.
Every one of those produces a file of a plausible size that restores without complaint. And because the failure is silent, it survives until the day you need it.
Why the usual checks miss it
There are three checks people normally run, and each one passes.
"The file exists and looks about the right size"
A schema-only dump of a real database is not small. Three tables with indexes, constraints and sequences is several kilobytes before any rows. Size thresholds catch empty files, not empty backups.
"The archive parses"
For Postgres, pg_restore --list reads the archive's table of contents. It is a genuinely good integrity check — it fails on a truncated file — and a schema-only dump passes it perfectly, because there is nothing wrong with the archive.
"The restore exits zero"
It does. There were no errors. Restoring nothing successfully is still success.
MongoDB is worse, because the recommended check is misleading
If you back up MongoDB with mongodump --archive --gzip, the natural way to verify the result is mongorestore --dryRun, which parses the archive without writing anything.
It reads the archive prelude — the header listing which collections are inside — and stops.
Here is --dryRun against an archive deliberately truncated to 400 bytes:
$ head -c 400 good.archive.gz > truncated.archive.gz $ mongorestore --archive --gzip --dryRun < truncated.archive.gz preparing collections to restore from dry run completed 0 document(s) restored successfully. 0 document(s) failed to restore. $ echo $? 0
Exit zero on an archive with 400 bytes of a multi-megabyte file. The only case it fails is a completely empty file. If your Mongo backup verification is built on --dryRun, it is not verification — it confirms that a header exists.
A real restore does catch it. Restoring the same truncated archive into a scratch namespace exits 1 and produces one collection where the header claimed two. But you have to actually restore it.
The check that works
Restore the backup somewhere disposable, then compare row counts against production. That is the whole idea, and it is the only check in this article that catches every case above.
It works because it tests the thing you actually care about — is the data there — rather than a proxy for it. A schema-only dump restores three tables and zero rows against a live database with 8,415. That comparison cannot be passed by a backup that does not contain your data.
The shape of it, whichever database you run:
- Fetch the most recent backup from wherever it is stored.
- Verify its checksum against one recorded at the time it was written.
- Start a throwaway database container — on the same image as production, see below.
- Restore into it. It touches nothing live.
- List the tables or collections, and count the rows in each.
- Compare against production, allowing some drift — the dump is older than now — but not allowing zero.
- Tear it down and report.
Run weekly, it takes seconds and it moves your backups from a hypothesis to a fact.
$ ./restore-test.sh PASS backup freshness: 7h old PASS checksum matches (51b103a05e7c…) PASS restore completed without errors PASS restored copy contains 3 tables FAIL users: only 0 restored but live has 1,203 FAIL orders: only 0 restored but live has 8,415 restore verification: 2 passed, 2 failed
One trap worth knowing about
Run the verification with client tools from inside the database container, never from the host.
A custom-format Postgres dump carries an archive version, and an older pg_restore cannot read a newer one:
$ pg_restore --list backup.dump pg_restore: error: unsupported version (1.15) in file header
That is a host running Ubuntu's postgresql-client 14 against a postgres:16 container. The backup is perfect. The message is not about corruption at all, and if your verification is wired this way it will declare every healthy backup broken — which means you will either ignore the alert or lose a day to it. Using the binary inside the database container makes a version mismatch impossible. Full write-up of that error here.
Go and check yours
Not the file listing. Not the exit code. Restore your most recent backup somewhere disposable and count the rows.
If you got here because you built something with an assistant and are working out how to launch it, start here instead — this is one item on a longer list.
Most people have never done this once. Some of them find out that the backup they have been keeping for eight months restores an empty database — and they find out on a Tuesday afternoon rather than during an outage, which is the entire point.