PHD esXpress backups, Patent Pending esXpress v3.1 Restores release README copyright 2007, PHD Technologies, Inc., www.p2v.net, www.esxpress.com by ron mckelvey, Sept 2007 This file is /home/phd/bin/RESTORES.txt ################################################################################ www.esxpress.com or backup.p2v.net BASIC ESXPRESS TERMINOLOGY -------------------------------------------------------------------------------- -IN THE CONSOLE- A console restore is simply the ability to restore a VMDK file or a VMX to the VMFS from within the VI3 console of a host. In esXpress v3 this function is the standard restore procedure. Currently to restore an esXpress v3 backup, you have to do it in the console, and not in a VBA. We consider the ability to restores virtual machines in the console an important feature. If restores were limited to a VBA only, then you would not be able do any restores until your complete virtual framework was up and running. By enabling restores in the console, you are always provided the ability to restore your backups. We consider this a crucial DR feature. A console restore is a menu driven text mode UI. You are walked through the complete restoration process of a VMDK file or a VMX file from beginning to end. Nothing more then hitting the 'Enter' key is usually required. -SIMPLE REPLICATION- If you were to take a backup, and restore it on another host, that is basically a restoration. If this happened automatically, then we could call it a replication of a Virtual Machine. This is how we define 'Simple Replication'. -IN THE GUI- Now that the restore queue has been implemented, GUI restores are coming. -MANUAL RESTORES- Above all, you should always be able to restore your backups. This is a simple statement, that we at PHD stand behind. With esXpress backups, you can always restore your backups, whether you have access to our software or not. With our product, only backups are licensed. There is no licensing involved with doing restorations. You own your data, not us. -FULL BACKUPS- When a FULL backup is made, it is just a gzip'd copy of the file-flat.vmdk file (gzip is the standard Linux compression utility). This file can easily be restored to any version of VMware or even other virtual platforms. With VMware, you just need to make a STUB file, and you can use your FULL backup after unzipping it. Restoring a FULL is as simple as: zcat fullbackup.gz.phd > newfile-flat.vmdk -DELTA BACKUPS- esXpress Delta backups are more then just backup files. The delta backup file itself it an executable program that will allow you to find and locate and combine the FULL and DELTA archives together to create the VMDK file. Restoring a DELTA backup manually is fairly simple. If the DELTA and FULL backup are together on a share, you can just shell the archive. Example: sh the_delta_file.phd and a restore menu will appear. If the FULL is on the local share with the DELTA, then it should automatically find it. If not, you can tell it where it is, or even grab it on the fly from your FTP server. Once in the DELTA Restore menu, you can perform many tasks including: restore a VMDK, verify the FULL archive, and a few others. This delta backup is scriptable. See the help in the restore menu itself. THE ESXPRESS RESTORATION PROCESS -------------------------------------------------------------------------------- The way restores work in esXpress is different from most backup solutions. The backup and restore programs in esXpress are essentially 2 completely different applications. One does not have anything to do with the other. In fact the licensing for esXpress is only in the backup engine, not the restore programs. The Full backup is always restorable because it is nothing more then a gzip'd copy of the -flat.vmdk file. The Delta backup is actually a self restorable archive. It can restore itself without the need of the esXpress software. You own your backup archives, not us. You know with esXpress your backup archive are yours, and that you can always restore them, forever. esXpress does not keep a grid or database of backups it has made like other products, but will connect and index the backup targets when needed. This way you can restore a FTP server, restore your backup tapes, and then have your VI3 hosts go out and index it. Then you can start doing restores. With the mass/auto restore you can setup numerous restore jobs at once, submit them to restore, and then watch it all restore automatically. When choosing to restore a backup, the backup targets will probably need to be indexed unless they were recently indexed. If it's been more then 5 hours, then the targets will be re-indexed automatically for you. Because of this indexing ability of esXpress, you can safely remove, move around or restore backups to a backup share. Then just re-index the share, and you can now restore your backups. It does not matter to esXpress if the delta is on the local drive and the Full is somewhere else. As long as both the Delta and Full are included in the index of the backup targets. As of 3.0-9 you can easily restore backups from version 2 of esXpress also. Just note, for the v2 backups, they will show up in the restore menu with their hostname as the VM Name. This is because the naming conventions used in version 2 of esXpress were different then those used in esXpress v3. DELTA vs FULL RESTORE -------------------------------------------------------------------------------- When restoring backups, it is always better to restore a Delta backup over a Full backup, but a Full is faster. When a Delta backup is restored, esXpress knows the size of the VMDK. On restoration esXpress will create the VMDK file on the VMFS first, then import the backup into that pre-allocated VMDK file. This is the proper way to write to the VMFS. Because of this import, you can restore multiple VMDK files to the same VMFS at the same time. If you are restoring a Full backup, it is restored as a plain file. It is no different then using the copy command or tar to write out the VMDK file. These methods do not pre-allocate the VMDK first, but write out the new file in chunks. If you were to copy multiple files to one VMFS at the same time, these files would effectively be interleaved. They would be severely fragmented. If you still need to restore a Full backup, only do one at a time per VMFS, and you will be OK. If you do more, then they will be fragmented and performance can suffer in the running VMs. When esXpress makes a Full backup, it always makes an empty delta too, even in the free version. The Delta backup contains the metadata information about the backup, along with VMX file and the index maps. When esXpress Delta backups are restored, the restored file is checked, block by block on restoration against the index map. If there is a problem, then the restore will be aborted. If you were to lose a Full backup, and try to rename another one to replace it, it will not work. The checksums will not be the same and the restore process will be aborted. ESXPRESS BACKGROUND RESTORE QUEUE -------------------------------------------------------------------------------- When esXpress does a restore, it can do it a number of ways. When you use the phd menu to restore a VMDK, you have the option to submit it to the restore queue. When you choose a Delta backup to restore, at the end you have the option of submitting it to the restore queue. Or you can run it in the forground like esXpress has done until now (version 3.1). Note - Background Restores are a licensed feature of esXpress. (Always restore Delta backups over a Full backup. Restoring a Full might be faster, but when you restore a Delta backup it has many advantages. The VMDK will be pre-allocated and the backup imported into the VMDK. Each block of the Full and the Delta are compared against the index map to validate the checksum of all data. esXpress knows the proper name of the VMDK to restore.) In the phd text menu, option (C) Replication / Restore Options menu under the (C) Configuration menu has the options for the restore queue. --It needs to be enabled, it is by default. --You can configure how many concurrent restore jobs to run at once. The default is 1, with a max of 4. Do not run 4 unless you increase the MHZ reserved for the console. The restore queue is /etc/phd/restore When backups are submitted for restore, a control file is created in this folder, then the phd_daemon will pick it up, and run the restore. If you define 2 restores to run, then 2 will run at a time. The log for each restore is also kept in this folder. From the Replication menu you have then option to clean-up and delete the restore jobs. (auto purging coming) (Running 2 in a normal console is OK, but increasing console CPU Mhz will help keep the console from bogging down. Do not run more then 2 unless you increase the CPU allocation. But do experiment and test.) From the (B) Backup Restore Status menu you can see the restore queue. It shows all the restore jobs in the /etc/phd/restore folder. The restore status for each is shown if it is complete, or Waiting to run. For each restore you can see who submitted the restore job, along with the log for each. If the restore is currently running you can watch the log as it runs. Remember to hit ^C (Control C) to exit the live viewer. When a backup is complete you can also view the log, this will use nano or vi. The restore queue is only for Delta archive restores. It will not work for a Full backup. Even in Free mode, when a Full backup is made, an empty Delta backup is also made. The restore queue is also a licensed feature of esXpress. This means to auto replicate or mass restore, or for even backgound restore, you need a licensed copy of esXpress. If you choose to restore a Full backup, it cannot be run by the restore queue in the background. You must run the restore in the foreground, on the console or through putty (ssh). This is how all esXpress restores were run until now (versin 3.1, with the restore queue) SIMPLE REPLICATION / MASS RESTORE -------------------------------------------------------------------------------- Simple replication is defined as having one Host backup to your Backup Target, then have others Hosts check the Backup Target and look for new Delta backups. When new backups are found they are automatically restored. This is what we call simple replication, even one to many replication. Unlike other replication products, simple replication is included with esXpress. It is included at no additional charge, but does require a licensed copy. esXpress can only replicate from the available backups. This means you always have a copy on the backup server, in addition to the replicated copies. Other products copy the data directly from one VM to another VM, with no backup. HOW IT WORKS: The default action is to look for the newest backups and restore them. But you can also define a VMDK as -3, this means restore one at least 3 days old instead. This way you can keep a copy of the Exchange server VMDK auto restored on another LUN, and a second copy from last week. The Replication Actions are defined from the Restore Menu, which is item (E) from the phd Main Menu in the console. From the (R) 'Replication Action' menu you define the VMDKs you want to replicate, and manage the restore jobs. When the replication runs and chooses Delta backups to restore, they are submitted to the esXpress Restore Queue. Choose option (B) to manage the Backup Restore Queue. All the options, the queue and log for restores are in the /etc/phd folder. /etc/phd/restore is the restore queue itself. Restores are submitted to this folder then processed by the backround phd_daemon. This is enabled by default and requires the phd_deamon to be running. The restore queue is a licensed feature of esXpress. /etc/phd/restored.log is a list of previously restored VMDKs. This file is checked after then Backup Targets have been indexed and compared against VMDKs defined in the /etc/phd/vmdks.auto file. If you delete this file or edit it, then you can restore a previously replicated restored backup. /etc/phd/vmdks.auto is where the replication is configured. This is a simple text file describing with VMDKs to restore and where to restore them to. You can edit the /etc/phd/vmdks.auto manually or from the phd menu. Option (E) in the Replication Menu. This file contains some example on how to configure the replication. You also also pre-load this vmdks.auto file with a list of all known VMDKs on the backup targets. This is option (F) on the Replication Menu. Afterwards the Backup Targets will be indexed, and all unique VMDKs will be appended to the vmdks.auto file. Then you can easily edit this file, to include the VMDKs you want to restore. When each VMDK is loaded, it is defaulted to restore to the VMFS as defined in the (C) Replication / Restore Options in the (C) Configuration Menu. The /etc/phd/vmdks.auto file looks like this: ## /etc/phd/vmdks # Copyright PHD Technologies Inc, 2007 # Part of PHD esXpress backups, www.esxpress.com # # This file defines the VMDKs to restore and # which VMFS to restore them on. This is meant for mass restores. # # By default, the most recent backup will be restored. # USE_DAYS=-1 will not restore a backup newer then yesterday. # USE_DAYS=-3 will not restore a backup newer then 3 days ago. # # USE_DAYS=-3 # # You can also set the USE_DAYS value on a per VMDK basis instead of globally. # # UUID can be the correct UUID, or *, or % to accept # any UUID where the VMDK name and the SCSI_ID match # # When the default restore path is created, it will use the VMFS defined in # the Mass restore menu, off the Restore Menu. # # RESTORE:VM_NAME|UUID_OF_VM|SCSI_ID|VMDK_NAME|Complete_Restore_Path|Use_Days # RESTORE:CRM|564d122a-0ed7-3b92-2f46-aee9456b2074|00|CRM.vmdk|/vmfs/volumes/ISCSI02/ronzo/CRM.vmdk RESTORE:CRM|564d122a-0ed7-3b92-2f46-aee9456b2074|00|CRM.vmdk|/vmfs/volumes/ISCSI02/caleb/CRM.vmdk|-3 The concept is that you define a list of VMs and VMDKs you want to mass restore or replicate. Every hour (or as defined) the backup targets will be re-indexed and matching backup archives will be matched against the /etc/phd/vmdks.auto file. When defining which VMDKs for replication you need to know: 0. Line must start with RESTORE: 1. Which VM to restore. This is the folder where the VMX file lives. 2. Each VM is unqiue by the UUID, enter the UUID or * (For any). Be careful of DUPES. 3. The SCSI ID (or * For any) of the VMDK you want to restore. The backup archives are name 00- which mean SCSI ID 0:0 4. The Name of the VMDK you want to restore. 5. And the Full path to restore the VMDK to. This is usually /vmfs/volumes/some_vmfs/some_folder/machine.vmdk 6. Optionally, how many days behind the restore should be. If you make this -7, then only a backup at least 7 days old will be restored. The example above showing CRM.vmdk is configured to restore the most current VMDK and a copy from 3 days ago. This allows you to backup one VM, and have multiple copies of it restored, for multiple purposes. ON TO RESTORES FROM THE PHD MENU -------------------------------------------------------------------------------- From the (E) Restore Menu you can restore VMDK files and VMX files from within the console, and create STUB files if needed. Creation of STUB is not required but you may at time need to create a new STUB file. Console restores are VMDK restores. You select the VMDKs you want to restore. If a VM as 4 VMDKs then you need to select and restore all 4 VMDKs for this VM. To restore a VMDK you need to choose: (C) Restore through the ESX Console Then on the next menu you can search by VM Name or VMDK name. Or restore a VMX file. After a Delta backup is restored, you are asked to restore the VMX file at that time. The restore VMX option is here incase you just want to restore a VMX only. Choose: (V) Select by Virtual Machine name Now you should get a list of all known VMs. The VM name here is from the folder on the backup target. The folder name on the backup target is based on the date of the backup and the original folder name of the VM from the ESX host where the backup came from. Version 2 backups will show up here by host name, not VM name. If you choose (F) Select by VMDK name, then a list of all unique VMDK name are shown. Sometimes these menus can have a lot of choices, and moving around the menu can be tiresome. Some quick keys to remember. If you hit (B) or (Q) you will jump to the bottom, and pressing (1) will bring you back to the top. SELECT A BACKUP TO RESTORE -------------------------------------------------------------------------------- Once you select a VM and press enter, you are shown a list of all dates for this VM. This is all the backup archives found on the backup targets. Choose a date or All Dates. Then the backups that match this VM name are shown. This will include all VMDKs that match this VM name. Be warned, if you have the same VM name on different hosts, they will show up here together. For each backup listed, you are told from which target it was found, the size, the type of backup (Full/Delta), the date, the SCSI ID and the VMDK name. You can choose the Filter option and filter the output by various ways. Once you select a VMDK to restore, you are then shown a summary of what you want to do. Here you are shown the full path to the backup file and the backup target information. Once you select to continue, the connection is checked to make sure the backup can be successfully accessed. This can can a few seconds as the header for each backup is actually downloaded and verified. If this is a Delta restore, then you are asked to choose which Full backup to use. If the Delta backup file you choose has a UUID then esXpress will show with an asterisk '*' next to the Full backup it thinks is correct by a matching UUID. If you choose the wrong Full backup because you have Dupe VM names, then the restore will be aborted because esXpress will know it is not correct when it validates the blocks on restore. You could be shown multiple Full backups that match the Delta you are restoring. If you were backing up to VMFS and NET, you have two Fulls, and both will be shown on the restore menu. SELECT WHERE TO RESTORE -------------------------------------------------------------------------------- After the connection to the Full and Delta have been verified, you are now asked where to restore this backup to. You are shown a list of where this VMDK file currently exists and the option to restore elsewhere. Warning, if your VMFS Nice names have funny characters in them, then esXpress is not happy. Instead of ISO's, use ISOs. The single quote is a problem at this time. Other characters may cause issues too. You can either choose to over-write an existing VMDK file or choose a new path location. If you choose to over-write, then that VM has to be powered off. If the VM is running, and then selecting it will not hurt it, as esXpress will tell you that it can not select that location because it cannot get a write lock. Once you choose a location, you are asked to name the VMDK. esXpress will suggest a name. If you are restoring a Delta, then esXpress knows the name. If it is a Full backup then esXpress can only guess at what the name should be. AND THE RESTORE IS STARTED -------------------------------------------------------------------------------- You are asked for a final verify before starting. Once you select to continue you are asked if you want to submit this restore job to the background restore queue. Saying 'Yes' will add this restore job to the queue. You can keep submitting jobs to the restore queue and they will be run in order. For this example we selected 'No' to submit to the restore queue. The restore is started in the foreground. Do not close a putty window if you are running a restore in the foreground. Your restore will probably die and the esXpress process might now properly clean-up leftover files. If you are doing a Full, then the restore starts and you are shown a status of how much and how fast. When restoring the Full, esXpress is literally grabbing the file from FTP/SSH and passing through gzip and writing the file out. (Make sure you read Delta Restores vs Full Restores above) A Full restore. -------------------------------------------------------------- esXpress v3.1, Doing FULL Backup Restoration, www.esxpress.com Restore FULL Backup from Location: 192.168.201.2:21/local/esxpress/backups Folder : 2007.09.20-CRM.564d122a-0ed7-3b92-2f46-aee9456b2074 File : 00-CRM.vmdk.gz-070920-1601.phd Restore to: VMFS: /ISCSI02/ronzo VMDK: CRM.vmdk FLAT: CRM-flat.vmdk Restoring FULL Backup from NET FTP Target 405 MB processed Avg(15.0 MB/s) Cur(13.7 MB/s) If you are restoring a Delta backup, the process is a little more involved internally. The Delta backup needs to be downloaded to the local VMFS first. Then the Delta backup is executed like a program, which then pulls the Full and makes the new VMDK file on the fly. When the delta is being restored you are shown real stats on how much, what percent, how fast and how much time remaining. Delta: 67% Full: 32%, 1300 mb of 4096 mb, at 11 meg/sec, Elapsed: 01:50s Remain: 04:14s When restoring a Delta archive, the VMDK is pre-created on the VMFS first using vmkfstools. Then the restore is imported into the existing VMDK. Because the Delta is importing the backup into the VMFS, and it is validating the data of the Full and the Delta blocks against the index maps, it is not as fast as a Full restore. When the data blocks are verified on restore, any bad data will cause the restore to abort. Even one flipped bit will cause it to abort. After the Delta restore is complete, you are asked if you want to restore the VMX file also. When you restore the VMX file it is the copy form when the backup was made. If you restored to a different path and setting, you must check the configuration of this restore VMX in the VI3 client. esXpress will not edit or update the VMX to reflect changes you made on restore. You must do this yourself from the VI3 client Afterwards the Delta backup file is removed, but only if it was downloaded. Delta backups on the VMFS as a backup target will not be deleted. If you want to test the DR ability of esXpress you can simply run the delta backup file on your backup host and restore a backup there. You do this by 'sh the_delta_archive.phd' and a restore menu will be shown. If your backup host is a Windows server, you just need install Cygwin first and you can run the self restoring Delta arhive. (Install Perl and Lynx too with Cygwin). The Delta backup is a run-able Linux program that will use the Full backup and recreate the VMDK file. No additional software is required. To restore an esXpress delta backup on any Linux or ESX host, without the esXpress software, you need to only: sh (the delta file) Example: sh 00-Win2K3.vmdk.delta-2007.01.29-0220-070124-2111.phd Then the delta restore menu will be shown. From this menu you can restore the backup or just validate it. This is how Delta backups are restored. Once the Delta has been downloaded, the delta file is then executed by the phd menu. The restoration of the Delta files can also be scripted easily! ################################################################################ ################################################################################ ################################################################################