esXpress v3.1 release README copyright 2007, PHD Technologies, Inc., www.p2v.net, www.esxpress.com by ron mckelvey, June 2007 This file is /home/phd/bin/README.txt THIS IS A POINT RELEASE www.esxpress.com or backup.p2v.net MAKE SURE YOU ***************************************************************************** Go to the Configuration Menu and go through all the Options, Backup Target Options, FTP Configuration Options, SMTP Options, FLB (File Level Backup) Option, Snapshot on Snapshot and Helper Options. For the Helper Options, you MUST set a VMFS and a Virtual Network Switch to use. ON RPM PACKAGES ***************************************************************************** Using RPM packages are the easiest and best way to install software on the ESX platform. It is the method that VMware itself uses. To install esXpress on your host, 2 packages are required. The first is the VBA package, it is around 70 meg in size. The second is the esXpress package, usually about 4 meg in size. IF THIS IS AN UPGRADE ***************************************************************************** --If you have esXpress installed, you need to remove it and re-install it If you have the GUI running, make sure you exit out of it first. To remove the esxpress package run: rpm -e esxpress This v3.1 release requires a new esXpress VBA package, if you are already using version 3.0. If you already have v3.1 installed, you just need to replace the esxpress, package and not the VBA rpm. You need to copy the .rpm files to your VI3 host. Then as the 'root' user you need to remove the older esXpress and install the new packages. esXpress needs to be removed and re-installed, not -U rpm -e esxpress rpm -e esxpressVBA rpm -i esxpressVBA-3.1-1.esx.i386.rpm rpm -i esxpress-3.1-17.esx.i386.rpm Then run: phd import Make sure you ran 'phd import' IF THIS IS A NEW INSTALL ***************************************************************************** --You need to copy both .rpm files to your VI3 host. Then as the 'root' user you need to install both esXpress rpm files. One is the VBA appliance, and the other is the backup programs. rpm -i esxpressVBA-3.1-1.esx.i386.rpm rpm -i esxpress-3.1-17.esx.i386.rpm --Before starting the esXpress GUI, you need to configure a VMFS for the VBA helpers to use, along with a default Virtual Network for them also. Type the command 'phd' from service console. The PHD Text Menu will be shown. All actions in esXpress can always be done through the service console or a putty session (ssh) along with the GUI. At the main 'phd' menu are all the options you can do. Our initial goal is to configure the minimum, and start up the esXpress GUI. 1. At the main menu, hit 'U' for Setup Quick Menu. 2. You are now at the Quick Menu. If you do not know how to navigate and use the phd text menus, then hit 'enter' to read the help on menus. 3. Choose option 'H' Configure Helper Options. Here we need to set the VMFS that the esXpress VBA Helpers will live on. 4. At the top is the VMFS_For_Helpers, hit 'Enter', then 'Enter' again. You are now shown a list of available VMFS. Choose the VMFS you want esXpress to use for its virtual helpers. Local Space is perferred. When the VBAs are running, they require 3GB each. 5. Choose 'OK' or 'ESC' or 'Cancel' and you should be asked to save your changes. Choose Yes. 6. Now your back at the Quick Menu. 7. Choose (O) Network Options for VBAs. 8. The very first line is Network_For_Helpers, press 'Enter' twice to bring up a list of found VM Network names. Choose the one the VBAs should use. Even the GUI does not require a network, the VBA still needs a valid network. 9. Choose 'OK' or 'ESC' or 'Cancel' and you should be asked to save your changes. Choose Yes. 10. Now your back at the Quick Menu. 11. Choose (G) Start the esXpress GUI Helper. Confirm with Yes. 12. Make sure the GUI is starting correctly, choose (T) to Tail the esXpress log. **If you do not get shown a list of available VMFS, this might be due to a limitation of esXpress. Make sure none of your VMFS Names have a single quote. Example: ISO's, change into: ISOs Other characters might cause issues also. Use your VI3 client and look at this host, a new VM should appear. Its name should be '_esXpress: GUI Helper'. Open a console to this VM to access the esXpress v3 GUI. Once on the main GUI menu, choose configuration to configure esXpress v3. Go through the GUI and look at all the pages of options. The GUI has extensive help built in. (All suggestions are welcomed!) To actually run backups you need to properly configure a backup target or use VMFS space for backups. A Backup Target can be a FTP server, SSH server, or even a SMB server if you want to backup across the network, or choose a VMFS to backup to. When configuring FTP, SSH, or SMB the account should have admin access to the share, all rights (to make and delete files and folders). You should use a backup folder such as '/backups' instead of just '/'. The user and password you use should not have any funny characters such as back slashes, @'s, %'s or quotes. We have seen both '$' and '!' causing issues too. Make sure you test with backups and restores. If you are using Windows, and want to use FTP, make sure you use FileZilla. Google FileZilla, it is on sourceforge. We used to recommend other Windows FTP servers, but in our larger environments, they did not work. While FileZilla has no problem handling 25 hours and 50 TB of backups. If you are using SSH, this means a Linux or UNIX type ssh account, not sftp. Access to certain shell commands are necessary for esXpress SSH backups to work. This includes the dd and the ls commands along with permissions to create, rename, delete files and folders. When backing up to VMFS, you must have enough space to create a VMDK of the same size as the VMDK that is being backed up plus 300 meg. Even if you are doing a Delta backup of a VMDK that is 40 gig, the VMFS backup must be created at 40 gig + 300 meg. Then after the backup is finished, the backup is extracted from the VMDK and the VMDK is removed. Let's backup one VM, and see if it works. 1. Go back to the esXpress GUI, back to the main menu, and choose Live Log Viewer. if you are using ESX 3.5, then the log viewer is currently diabled. 2. Choose a VM to test with. It can be a real VM, or you can create a dummy VM to test with. 3. In the text based 'phd' menu, choose option 'O' to backup one VM and hit enter, choose a VM and start the backup. You can choose option 'T' to tail the backup log and watch the progress. or In the GUI, or actually the VI3 client, you can issue a backup command to a VM. Choose the VM you want to backup now, and append '[xnow]' to the display name of the VM. That is open square bracket, xnow, then close square bracket. This command is not case sensitve, we just show it in lower/upper case to understand the command easier. Changing the name sounds a little severe at first, but you are only temporary changing the DisplayName. Once the esXpress back-end accepts the command, it will rename the VM back. 4. If you are in the esXpress GUI, you can force the phd daemon to wake up sooner by pressing the clear lock button while in the live log viewer. 5. The backup of this VM should start, you should see if it works or not. a. The VM selected will have its snapshots validated. b. A snapshot will be added to the VM. c. A VBA helper will be created, and the VMDK from your VM will be attached to it. d. The VBA then is powered up, and your VMDK is backed up in the VBA, which means all the disk I/O and CPU is all in the virtual space. NOT IN THE CONSOLE. e. After the backup it complete, the VBA is shutdown. f. The snapshot is removed from your VM. 6. If it is successful, do it again. This time a delta backup should run. 7. Do run a backup with [xTEST] mode. Add '[xtest]' to a VM name and a fake backup will be run. Instead of doing a backup, some benchmarch tests will be run against the VM, in an attempt to get the MAX network, disk, CPU and write speeds. This will also test if your backup target can accept a backup file greater then 4GB in size. ***************************************************************************** IMPORTANT: READ THIS ***************************************************************************** By default once esXpress v3.1 is installed and configured, it will attempt to backup every VM on that host, every night, unless you tell it not too. All VMs are included automatically. THEY HAVE TO BE EXCLUDED NOT TO BE BACKED UP. If you create a new VM, and do not exclude it, it will be backed up on the next automatic cycle. (Depends on esXpress configuration options) IF YOU INSTALLED ESXPRESS, IT WILL AUTOMATICALLY TRY AND BACKUP ALL YOUR VMs TONIGHT AT 00:01 HOURS. PLEASE TURN OFF BACKUPS IF THIS IS NOT DESIRED. ***************************************************************************** IMPORTANT: READ THIS ***************************************************************************** ***************************************************************************** Getting Better VBA Performance: --By default esXpress is using gzip for compression, but you can also use lzop instead, which can be twice as fast as gzip. --The default Number of CPUs in the VBA is set for 1. Making it 2 will help performance in the VBA. In ESX 3.5 only 1 CPU will be used. --esXpress will use all the CPU that you give it. Without any limits, each CPU in a VBA will use 100% if allowed. --Set How Aggressive esXpress will run. Set this more MORE and the VBAs will run more backup threads. This will use more CPU. This only applies to Delta Backups and when the VBA as 2 or more CPUs. --Make sure you have given your VI3 COS (the Linux Console part of VI3/ESX) more then the minimum memory of 272 meg. The VMware back-end for VI3 is now Java based, and it runs much better with the 800 meg (or at least 512 meg) of COS Memory. Also increase the CPU share for the console. By default VMware will give it 400 mhz. * When snapshot actions are done, they are done by the VMware hostd process that lives in the console. The act of adding or removing a snapshot is all done in the console. * When you are backing up a VM, it will have a snapshot file, that is growing in size as the VM writes to its disk. After the backup is compelte, the snapshot needs to be commited back to the VMDK. This is all done in the console. * Please give the console as much Memory and CPU as you can. Changing this will make all the difference. Also make sure after patching you check the memeory settings again. --Do the phdcat speed tests on your /vmfs/volumes to get some baseline disk speeds. (search for phdcat on support.esxpress.com support board) --After you know what your max disk speeds are, you should enable throttling in esXpress. If you do not enable throttling, the VBAs will try to run as fast as they can. They will grab all the disk I/O that is available. On some SANs, having 4 hosts, with 4 VBAs each, all trying to read as fast as they can, and the SANs cannot handle it. Better to enable throttling and let esXpress time-slice the disk I/O. ***************************************************************************** ***************************************************************************** WARNING!! --esXpress is designed to use the VMware platform to its full realization of its features. Because of this, improperly installed or badly designed systems will cause esXpress to fail. But these failures will bring to light any limitations in your virtual infrastructure. Many customers do not realize they have a problem until they install esXpress. Please test on one host, then two, before rolling it out everywhere. --As of version 3.0-15, esXpress is now more aggressive in the VBA. esXpress will use the all the processor that is given to it. esXpress will use 100% of all the CPU that is given to it. If you give the VBAs 2 CPUs then they will run at 100% each. Use resource pools to control and limit the amount of CPU that the VBAs will use. ***************************************************************************** In our esXpress 2 product, the complete processing was done in the console space, even with the console limitation, esXpress 2 on newer hosts are achieving backup speeds of 200+ GB/Hour. By running in the console space, esXpress had to have limits, such as one backup at a time, single cpu, using FTP as the only available transport. With VMware stressing that processing has to be moved out of the console space we at PHD have taken a new direction in how to backup VMDK files. http://www.vmware.com/vmtn/resources/516 With esXpress v3.1 all the limits are gone! PHD is introducing our VBA Technology with the latest release of esXpress v3. A 'VBA' is a Virtual Backup Appliance. All processing in esXpress v3 is now done completely in the virtual space, using our VBA Helper. FEATURES --esXpress v3.1 will create FULL and DELTA backups of your VMDK files. Only esXpress offers Delta Backups for VMDK files. --File Level Backups are included with Enterprise. Define specific folders in a VMDK, and esXpress will zip them up and put them on a Backup Target. --Simple Replication is included with esXpress. If Host A backups up to a FTP Target, others Hosts B and C can check for new backup archives and restore them automatically. Thus allowing standby replicated VMs at no additional cost. --Only esXpress backups are completely restorable without having our software. A Full is nothing more then a gzip copy of the -flat.vmdk file. A Delta backup is actually self restorable. If you 'sh the_delta.phd' a restore menu you will be presented with the restore options. This even works in Windows with CYGWIN installed. --No processing of disk, network or heavy CPU in the console. All I/O happens in the virtual space. In our Patented VBA Process. --Run as many backups on each host as you want, instead of just one. --If you have a 4 CPU Host server, you can set the auto backups to use up to five VBA helpers. Max Helpers = # of CPU's + 1 --Completely scalable horizontally and vertically. --Different compression methods and level are available for Full and Delta backups. They are configured separately. Can even be done at a VM level. --In this release is only FTP/SSH/SMB and/or backup to VMFS is supported. (SSH is supported, not SFTP.) --Support for Snapshot on Snapshot Backups, including separate Targets. --Real Encryption using AES256 for your Full and Delta Backups. Encryption is done with gnugpg, the open source version of pgp. --The ability to Throttle your backup speeds to limit impact to the Host. --Some extra backup target options are available, such as separate top level folders for DELTA or FULL, or make the date another folder split. --Backup targets will now roll over to the next on failure. If FTP #1 gets a FTP failure, then FTP #2 will be tried, and so on. --Integration with the VI3 client and VC. Get backup status and feedback; also issue backup commands. --Resource Groups. Put your esXpress VBA Helpers in the resource group then you can control their 'resources' by using VMware. Do give each VBA at least 100 mhz each. --No Single point of failure. --Each host is still autonomous in that it is responsible for backing up all the VMs on it. --Updated Backup Status in the VM notes (configurable) --Backup Target management. Separate auto delete options for Network vs VMFS --Skip a Delta backup if the VMDK has not changed since the last backup. --The ability to backup VMs in parallel mode. If a VM has multiple VMDKs, then run multiple VBAs at once to back them up in parallel. --As of version 3.1 of esXpress, up to 16 VBAs are supported. This requires a BIG license code. --After each backup run you will still get an emailed report. That is one per Host that is running esXpress. --A RETRY backup option is enabled by default. After a backup run, the failed backups will be tried again. --Detailed reporting of what backup archives exist where, and how much space is being used. Also includes report of Snapshot usage on the VMFS. SOME ESXPRESS v3 BACKUP THEORY. --Suppose a 25 host environment using all 4-way servers. --Using esXpress 2, you were able to backup 25 VMs at once, that is one per host in the console of each ESX Host. If your using a proxy to backup your VMs, then you are limited to the number of backups the proxy can run at once. (If your proxy host has 4 CPUs, then you can run 4 backups at a time max to achieve full speed.) --Now with esXpress v3.1, you have configured your hosts to run 3 VBA's each, and now you are getting 75 VMs backups at once in the virtual space, not just the 25 VMs being backed up in the console. --No extra hardware is required to perform the backup of the VMDK files, all work is done by the VBA helper. (With the exception of your backup targets) VI3 CLIENT INTEGRATION: --When VMs are being backed up, a new VBA helper will be created for each. --The name of this helper, is the name of the VM and the VMDK that it is be backed up. Example: esXpress: 'W2K3 VM1', disk # 1/1: 'W2K3 VM1.vmdk', #13443a --As the backup of the VM progresses, the Notes field in the VBA will be updated to reflect the current backup stats and status. Example: _esXpress: Creating Index Map, 2%, 100 mb of 4096 mb at 25 meg/sec, elapsed seconds: 3 _esXpress: Creating Index Map, 73%, 3025 mb of 4096 mb at 47 meg/sec, elapsed seconds: 63 --When a backup of a VM is started, its Notes field is updated to reflect that a backup is starting. This is configurable. Example: _esXpress: 2006-09-07 13:18 - RUNNING BACKUP --When the backup is complete, the finished status of the backup is updated in the Notes of the VM. This is configurable. Example: _esXpress: 2006-09-07 14:13 - OK - 1/1/1 disks, (0.8%) 33m/4g/4g, 01:50s, 37mb/s (130gb/hr) _esXpress: 2006-09-07 13:00 - OK - 2/2/2 disks, (0.0%) 0m/2g/2g, 01:56s, 17mb/s (59gb/hr) ISSUING COMMANDS: --You can issue backup commands to esXpress by modifying the VM display name. Commands are in the form of: 'Open Square bracket', 'x', 'command', 'Close Square bracket'. ex: [xCOMMAND] --Just like in esXpress 2, you can control if a VM gets backed up hourly or not at all by inserting text into the displayname of the VM. [x0] Do not backup [x1] through [x9] [x1] Backup this VM one extra time a day [x9] Backup this VM nine extra times a day --With esXpress v3.1, you can also issue backup commands by adding text to the VM displayname. The PHD_DAEMON has to be enabled for this feature to work. By default it is enabled when you install the esXpress v3.1 rpm. This background daemon uses almost no resources. It only looks for tasks, and issue the commands to startup and run a VBA. --If you were to add [xNOW] (commands are case in-sensitive, examples are shown in uppercase) to the displayname of a VM, the phd daemon would see the '[xnow]', when it does, the daemon will rename the VM, and remove the '[xNOW]' from the displayname, then issue the phd-INDEX command to backup this VM. It could take up to a minute before the phd daemon picks up the displayname change. ESXPRESS COMMANDS FROM VI3 OR VC: --To bring up and run the esXpress GUI Helper. Go to any VM on the host and add: [xGUI] or [xCFG] or [xCONFIG] and the GUI Helper VM will be started. --To Delta Backup a VM now: These keywords will issue the phd-INDEX command, to Delta backup the VM. The '--force' is used, so powered off VMs will be backed up also. Example: [xINDEX] or [xDELTA] or [xNOW] or [xBACKUP] --To Special Backup a VM now: These keywords will issue the phd-INDEX command, to Delta backup the VM. The '--force' is used, so powered off VMs will be backed up also. Example: [xSPECIAL] or [xONE] For a Special FULL Backup, use: [xSPECIALFULL] or [xONEFULL] --To FULL Backup a VM Now: This keyword '[xFULL]' will issue the phd-FULL command, to Full backup the VM. The '--force' is used, so powered off VMs will be backed up also. Example: [xFULL] --Backup ALL VMs Now: When backing up all VMs, the normal backup all rules are used, the '--force' command is not used with the following keywords. The Delta backup all VMs: [xALL] Full backup all VMs: [xALLFULL] or [xFULLALL] --Wipe Indexs, so next backup is a FULL. [xWIPE] to wipe the indexs of the one VM [xWIPEALL] or [xALLWIPE] to wipe the indexs of all VMs on the host. --To clear the notes field of one VM of esXpress information, use [xCLEAR], [xNOTES] To clear the notes field of ALL VMs, use [xCLEARALL], [xALLCLEAR], [xALLNOTES] PLEASE READ THE RESTORE.txt FILE ALSO. ESXPRESS ROADMAP --We would like to list and tell you about all the development we are doing, but in todays world market, that's not a good idea. --Our goal (the development team) is to put out a monthly point release to include bug fixes, but mostly feature enhancements. --On speaking of bugs, since esXpress v2 for ESX 2 was released May 2006, there have only been 5 different versions, with very minor changes. --esXpress v3 for VI3 was released December 2006, version 3.1 was Jan 2008. --On all these releases to the public, none of them have any show-stopper bugs in them. There is no version of the day at PHD Technologies. --We believe in a quality QA process. --All ideas and suggestions from customers and non-customers are always welcomed. thank you PHD Technologies Inc www.p2v.net or www.esxpress.com