In proving that hot backing up virtual machines with REDO logs actually works, we did extensive testing. We have been using the Hot REDO methodolgy with our VMware ESX servers now for 2.5 years now. Here is some of the early testing we did.

All products that do HOT ESX backups all use the VMware commands to add REDO logs to the VMDK files. What makes the various backup products different is the manner in which the VMDK files are copied, how the backups are run and how the restorations work. Let's start off with how we do the actual Hot REDO Backup.

• The hard drives are identifed for a virtual machine. These are the VMDK files.
• For each VMDK file, we add a REDO log to it. If there are multiple VMDK files, then they are added as close to the same time as they can be.
  We do not use the VMware vmAddredo.pl script, but our own for adding and committing REDO logs with vmware-cmd.
  
• We copy each VMDK and compress/encrypt it and move it to your backup server through FTP.
  
• FULL backup. If your VMDK file is 70 gig, we read 70 gig from your /vmfs and compress it and move 30-50 gig across the network. This depends on what you are running and how it compresses. A 100 mb network can move 35 gig of data an hour. Coming off the /vmfs we easily reach good network speeds.
• INDEX backup. If your VMDK file is 70 gig, we read 70 gig from your /vmfs and compare it to the original VMDK from the FULL backup. Blocks that are different from the FULL backup are extracted and compressed and copied to the backup server on the fly.
• For both FULL and INDEX backups, no additional local space is required on the ESX host itself. Except for the REDO logs themselves.
• After each VMDK is backed up, a second REDO file is added, then they are committed.
• Hot backups are one of the great unrealized benifits of running VMware.
  

The best backups are still suspended virtual machines that are copied, then resumed. But that's not really an option in most environments.
With HOT backups the virtual machine is restored to a state as if it has been reset. As if you walked up and pushed the reset button.
We have yet to see a HOT backup that would not work.

To the testing:
We basically did the same test over and over again using different disk configurations and operating systems. We created raid 0's and raid 5's in software for each operating system, then put the disk under heavy I/O and ran the backups. Then the backup of the virtual machine was brought up on another host and validated. We did each test numerous times, most at least 3 times, others as many as 10.

Our premise here was that if software striping and software raid survives the REDO backups, then any data would also survive.

Tests run using Windows NT server, which we created a basic 4 gig winnt_c.vmdk

• We added 2 additional VMDK files, then under NT formated them as a raid 0 span, as the D: drive.
  Then we did our stress testing and afterwards the backups came up perfectly, no errors in the file system.
• We created a 6 drive (6 VMDKs) raid 5 in software as the D: drive.
  Then we did our stress testing and afterwards the backups came up perfectly, the raid was online, but had to be reprofiled.
  This is because software raid 5 cannot survive a power reset without rebuilding.

We did the same tests using Windows 2000 and Windows 2003

• In addition, we added an additional VMDK file and using Windows Dynamic disk, made our C: drive bigger.
  Backups came up perfectly, no errors in the file system.

For our Testing in Linux we used a copy of our Oracle Database server.

• We put the Oracle Database under heavy load and ran the backups.
  The backups came up perfectly and Oracle knew it was not properly shutdown and recovered.
  This test we ran 10+ times with different data being written to the database.
• We also created a Linux server with a software raid 5 using 6 different VMDK files. When restoring the backup, same as the NT, the raid was online and survived, but had to be rebuilt.
  
• We regularly restore copies of the Production Oracle Applications and Database virtual server to create new environments for the developers.
  So in essence we test our DRP all the time.

In Conclusion: There are lots of opinions about how to backup virtual machines.
We at PHD Consulting have been successfully backing up Production virtual environments for years with great success using REDO backups.

We have done many successful Disaster Recovery Drills with our end customers.
With VMware, any kind of recovery is just waiting for the tape to roll and then you power on the virtual machine.

We welcome scenarios that our backup methodology may not work, because we then test them out and enhance esXpress backups to cover all situations.