To be safe (the minimum !) :
1) use good harware. Mainly by doubling all devices (power, etc …).
2) use RAIDx disks on a SAN (be careful on the ‘write back’ or ‘write through’ parameter !)
3) use ZFS if possible, or at least a journalized FS.
4) use ACID compliant database (postgresql …), use internal replication feature if possible, for HA and DR (see 6)
5) snapshots can be usefull sometimes, but rarely for databases because data can be inconsistent.
Don’t snapshot database directories, except if you can tell the database that you are backuping or snapshoting the data. (Postgresql allow it : SELECT pg_start_backup(‘label’); )
ZFS allow powerfull snapshots that can be sent to another remote node…
6) do regular database dumps in a (compressed) file for snapshots and backups. That can be done on a replicated slave database for performance.
7) do complete and incremental backups on tapes, disks and optical disks. Protect it in fireproof vault.
8) use system versioning for system configuration (/etc/, …) and protect it (tripwire …)
9) duplicate the whole in two datacenters in two different cities, at least, or two continents if possible.
cross the backups : Datacenter A is backuped on datacenter B and the contrary.
10) test data recovery and your disaster recovery plan (imply you wrote it :>) …)

And pray… if you are believer.

The unreliability of ext2 is just one specific example of a larger principle, in my opinion.

Just my $.02.

For a different perspective on this topic, I suggest reading http://www.xaprb.com/blog/2010/02/08/how-postgresql-protects-against-partial-page-writes-and-data-corruption/.

In practise, I can’t recall a snap of mine ever failing a recovery on EXT3, buts its not outside the realm of possibility.

Point being I suppose is that modern file system almost always survive a crash, so we tend to take that kind of durability for granted, but it really isn’t. The fact that a server comes back at all after an abrupt power loss is a testment to some very careful file system design. The fact that a database running on top of that file system recovers as well is still more good engineering, but both have to work to get your data back :).

So you are telling me i can loose all my data just by cutting power?
I won’t blame the filesystem here, i will blame the database.

I second that backing up by “snapshotting” a hot innodb db works. You just let the copy run crash recovery and voila you have a consistent backup copy of your database. I don’t use “snapshots” per se. My experience with LVM snapshots has been disappointing in terms of performance. I use RAID-10 and split mirrors.

We run master/slave MySQL replication…ZFS is the FS for both master and slave. We snapshot the filesystems on master and slave regularly. 3 times a week we copy one of those snaps off to a set of DataDomain backup servers.

We snap hot databases, mount the snaps elsewhere, and run a recovery all the time both via LVM and storage snaps (netapp).

In point of fact, I can’t recall ever having a snap we took this way fail the recovery process.

I keep reading about cases like this where it didn’t work, so I’m skeptical about proposals to use storage snaps in lieu of backups (because, inevitably, the snap you need will inevitable be the one that doesn’t recover), but as a general case tool to quickly clone databases, I think its phenomenally useful and infinitely faster than a dump/restore.

Point being I suppose that its entirely possible the customer in this case did do testing on this, even under heavy load, and it worked fine in testing. Doesn’t mean it was a bad test, probably more a failure of imagination (or research).

1) The primary server eating itself/disappearing into a puff of smoke
2) Some sort of logical error in the application or database corruption your data even though the server is technically just fine

Replication solves #1 *really well*.
Replication does not solve #2 at all since most common logical errors are happily replicated. As some have mentioned, running a delayed slave can mitigate this, but that only works if you notice the problem before the slave experiences it.

Traditional backups solve #1 weakly since the restore time on a large database can be prohibitive
Traditional backups solve #2 really well.

In the best of all possible worlds I usually recommend we run both, but depending on the business needs it doesn’t always make economic sense to run replication + backups.
In the enterprise environments I’m most familiar with the usual approach is to *always* run traditional backups, since they give absolute protection against both scenarios above, albeit with a high recovery time. For systems where we can’t afford the recovery time, we’ll also add in replication so that we can recover from some subset of failures quickly.