Design and Implementation Overview
On vSphere, backups are usually done by taking a snapshot, to efficiency obtain a static image of the virtual machine. Snapshots are a view of a virtual machine at a certain point in time, and enable quick and clean backup operation. Snapshots also provide an incremental backup mechanism called changed block tracking.
To back up virtual machines on vSphere, VMware recommends a two-language solution. First use Java to code the backup program that contacts the host, takes a temporary snapshot, records virtual machine configuration, and (later) deletes the snapshot. Then use C++ or C to code the VDDK program that transfers virtual disk data from the snapshot to backup media.
For restore, VMware recommends a two-language solution. First use Java to code the program that instructs the virtual machine to halt, or re-creates the target virtual machine from recorded configuration. Then use C or C++ to code the VDDK program that transfers saved data from backup media to virtual disk.
The Backup Process
These are the high-level steps to back up a virtual machine running in vSphere:
1
2
Tell the host to take a snapshot of the target virtual machine, using the vSphere API. Use the quiesce flag, but not the memory flag, which is incompatible with quiesce. The virtual machine continues to run, while the snapshot provides a static (quiesced) view.
3
4
5
Communicating With the Server
In a typical vSphere deployment with many ESXi hosts, an instance of vCenter Server manages the ESXi hosts, and can move virtual machines from host to host (vMotion) to balance load and possibly save electricity by powering off an ESXi host. VMware therefore recommends that backup applications communicate with the vCenter Server instead of with individual ESXi hosts.
The vCenter Server provides location transparency for vSphere Web Services developers. The vCenter Server tracks virtual machines as they move (through vMotion) from one ESXi host to another, and vCenter Server directs SDK operations to the ESXi host that currently runs a virtual machine. Using the vSphere Web Services API, it is possible to back up all the virtual disks associated with a virtual machine.
The handling of the vCenter or an individual ESXi host is essentially equivalent when using the vSphere SDK. With vCenter management, there is no need to contact individual ESXi hosts directly. The remainder of this chapter uses the term vSphere to indicate either a vCenter Server or an ESXi host.
To reduce the resources used by vSphere, VMware recommends that the number of connections (or Sessions) be minimized. It is in the best interests of any program that communicates with vSphere to create one Session and share it with all elements of the program that need to exchange information with vSphere. This means that if your program supports multiple threads, your program should multiplex the use of connection objects by use of access control locks (mutex and the like).
It is also important that all vSphere SDK operations proceed from one instance of the “Session” object that your application requests after logging into vSphere. Using the vSphere API your application can create objects that are “Session specific” and therefore would not be known to other portions of your application that might use a different Session.
Information Containers as Managed Objects
VMware documentation introduces you to the concept of the managed object and its handle, called a managed object reference (moRef). You might be tempted to get configuration and status information of managed objects using a piecemeal approach. This has the severe disadvantage of creating a lot of chatter over the server connection, so it is very slow. A mechanism has been created to provide status information efficiently: the PropertyCollector, discussed in PropertyCollector Data.
More About Managed Objects
The documentation for the vSphere API and object model introduces a large number of managed objects. There are five basic types of managed objects that describe the organization of a server. Other managed objects can be considered as details expanding on these five basic types:
It is a characteristic of all managed objects that they have a moRef to the managed object that serves as the parent to the managed object. This parent moRef allows you to reconstruct the object hierarchy exposed by the vSphere SDK. In general the hierarchy is a tree-like structure along the lines of:
Root Folder > Datacenter > ComputeResource > ResourcePool > VirtualMachine
There are variations on this theme, depending on whether you connect to vCenter or directly to an ESXi host, but the overall organization is like the structure above. Each managed object also has a Name property.
The virtual machine that you want to back up, and the snapshot you take of it (the extensible managed object VirtualMachineSnapshot) are both designated by their moRef.
Managed Object References
A managed object reference (moRef) is actually a handle and not the managed object itself. While it is certain that a moRef always contains a unique value, the unique value is only relative to the instance of vSphere to which you are connected. For example, if vCenter Server manages a cluster of ESXi hosts, each ESXi host maintains its own managed object reference namespace and the vCenter must maintain a managed object reference namespace representing all of its servers. So when an ESXi host is represented by a vCenter, the vCenter must ensure that the managed object references are unique. The vCenter accomplishes this by creating unique managed object reference names inside its own namespace, which differ from the names that ESXi uses for the same managed objects.
A vSphere instance (vCenter or ESXi) tries to keep the moRef for a virtual machine consistent across sessions, however consistency is not guaranteed. For example, unregistering and reregistering a virtual machine could result in a change to the moRef for the virtual machine. Thus, it is a bad idea to store a moRef and expect it to work correctly in future sessions, or with a different vCenter Server.
Unique ID for a Different vCenter
On one vCenter Server, the moRef uniquely identifies a virtual machine. If you need to track and inventory virtual machine backups across multiple vCenter Servers, you can use moRef together with instanceUuid. You can see the instanceUuid at the following browser path:
https://<vcserver>/mob/?moid=ServiceInstance&doPath=content.about
For direct connections to ESXi, the host address and moRef uniquely identify a virtual machine. However this moRef could be different from the one that vCenter Server returns, hence the fallback to instanceUuid. The instanceUuid was new in VMware vSphere 4.0. In previous releases, the fallback was to Uuid.
Gathering Status and Configuration Information
To save configuration of a virtual machine so you can restore it later, you can use the PropertyCollector to get the virtual machine configuration.
The PropertyCollector is the most efficient mechanism to specify, at the top level, all of the managed objects that are of interest to your application. It has methods for providing updates that indicate only changes to the previous state of these objects. There are two mechanisms for acquiring these updates:
Wait for updates – “Wait for updates” is basically a blocking call to the PropertyCollector. This is only useful if you dedicate a program thread waiting for the call to unblock. The advantage of this mechanism is that there is no traffic on the communications thread unless something must be reported.
The PropertyCollector is powerful but requires great attention to detail. Backup-related features of the PropertyCollector are covered in Low Level Backup Procedures of this document. The next section provides some background about PropertyCollector.
PropertyCollector Data
This document assumes that you want to keep up with changes in the configuration of the vCenter Server, and therefore plan to use the update tracking capability of the PropertyCollector.
The PropertyCollector requires two fairly complex arguments: the PropertySpec and the ObjectSpec. The ObjectSpec contains instructions to the PropertyCollector describing where to look for the desired data. Because configuration information in vSphere is organized like a directory tree, the ObjectSpec must describe how to traverse the tree to obtain the desired information. The net result is a complex, nested, and recursive list of instructions. Fortunately, once you have determined the location of all the desired information, the ObjectSpec needed to determine the layout of a vSphere object hierarchy can be a static unvarying object. See the code example in section Understanding an ObjectSpec.
The PropertySpec is a list of desired property information. Formulating a list that includes all of the desired information can take some effort to compile, but once determined, this can be a static object also.
The data returned from the PropertyCollector is a container class called PropertyFilterUpdate, which contains an objectSet with an item-by-item list of changes to object properties. Every item in this container is identified with one of the following keys: enter (add), leave (delete), and modify. On the first data request, every data item is included, and “enter” is marked for every data item.
The PropertyCollector presents its results in what amounts to random order. Since all managed objects have a “parent” property, you can reconstruct the configuration hierarchy by building a tree in memory, using the parent identification to organize. The root folder is identified as the only folder without a parent.
Useful Property Information
In the data returned from PropertyCollector, you can find most of the information that is useful for backup in the Virtual Machine managed object, including the following:
VMware supports many virtual disk implementations. The disk implementation type is important because:
For more information about the Java APIs, read the first several chapters of the VMware vSphere Web Services SDK Programming Guide, and related pages of the Web-based VMware vSphere API Reference Documentation. Both are available at http://www.vmware.com/support/developer/vc-sdk. Examples in this chapter assume that you have set up the vSphere SDK as described in documentation.
Doing a Backup Operation
After your program obtains information about what is available to back up, it can perform a backup. The three steps to the backup process are:
Extract Backup Data from the Target Virtual Machine, and save configuration information.
Prerequisites for Backup
To complete a backup, the calling program requires the permissions shown in Required Permissions to Complete a Backup.
Create a Temporary Snapshot on the Target Virtual Machine
The low-level procedure for creating a snapshot of a virtual machine is documented in the section Creating a Snapshot. Set the quiesce flag True to make the file system quiescent, otherwise the snapshot might represent a transitional system state, with inconsistent data. Restoring such data might be destructive.
Another flag named memory allows you to include in the snapshot a dump of the powered on virtual machine's in-memory state. This is not needed for backup, so set this flag to False.
Changed Block Tracking
This feature, first available in vSphere 4, provides the foundation for incremental (or differential) backup of virtual disks. Your application can back up only changed data as indicated by the QueryChangedDiskAreas method. Virtual machines with virtual hardware version 7 and later support changed block tracking. These virtual machines contain changeTrackingSupported in the capability field of the VirtualMachine managed object. See Changed Block Tracking on Virtual Disks for details.
Extract Backup Data from the Target Virtual Machine
Associated with the snapshot you just created are “versions” of the virtual disks. To identify these disks, you obtain a moRef to the snapshot you just created. From this snapshot moRef, you can extract the disk names and paths. How to do this is demonstrated in section Backing Up a Virtual Disk.
To read the data in a virtual disk, it is necessary to use the VixDiskLib. This library isolates the programmer from the details of extracting data from a virtual disk and its redo logs. For example, when doing backup you call functions VixDiskLib_Open() and VixDiskLib_Read(), among others. VixDiskLib allows access to disk data on sector boundaries only; the transfer size is some multiple of the disk sector size.
When accessing disks on ESXi hosts, VixDiskLib release 1.0 transferred virtual disk data over the network. Later VixDiskLib releases contain API enhancements so you can request more efficient data paths, such as direct SAN access or HotAdding disks to a virtual backup appliance. These efficient data paths requires minor code changes, such as calling VixDiskLib_ConnectEx() instead of plain connect.
Part of virtual disk information is metadata: a number of key/value pairs describing configuration of the virtual disk. Metadata information can be extracted from a virtual disk using the VixDiskLib functions VixDiskLib_GetMetadataKeys() and VixDiskLib_ReadMetadata(). You should save metadata keys along with the backup, in case you need to re-create the virtual disk.
The VixDiskLib API allows a backup application to perform a full backup of a virtual machine. The newer VixMntapi library can extract information about a guest operating system from its virtual disks, so your backup application can determine the type of operating system that is involved. This allows mounting the volumes to device nodes, so your application can perform file-oriented backups and restores.
Delete the Temporary Snapshot
As the last part of the backup process, you should delete the temporary snapshot. It is no longer needed, worsens virtual machine performance, and takes up storage space that could be put to better use.
The Restore Process
Your software can follow one of two restore scenarios: either revert to a saved state, or disaster recovery:
To bring an existing virtual machine to a previous state
1
2
3
Transfer the disk images from backup using VixDiskLib. Revert-to and delete the snapshot, if created.
To completely re-create a virtual machine (disaster recovery)
1
2
3
Transfer virtual disk data to the newly created virtual disks using VixDiskLib. Virtual disk data includes disk formatting information, so you do not need to build any kind of file system on the virtual disks.
Doing a Restore Operation
The two scenarios of restore operation are described below.
Prerequisites for Restore
To complete a restore, the calling process requires the permissions in Required permissions to complete a restore.
For security reasons, programs are not granted write access to the disks of a running virtual machine. Before you shut it down, you should determine the run-state of the virtual machine.
Run-state information is available from the PropertyCollector, and if you keep this information up-to-date, your application already knows the run-state of the virtual machine. To change the run-state you must have the moRef of the virtual machine. Use this moRef in a PowerOnVM_Task call through the server connection. For virtual machine shutdown, call the PowerOffVM_Task method.
Restoring an Existing Virtual Machine to a Previous State
The following steps restore a virtual machine to a certain saved state:
1
2
3
Restoring disk data requires that you obtain the current names of virtual disks. This process is similar to the one described in Extract Backup Data from the Target Virtual Machine, except in this case you obtain this information directly from the virtual machine and not from a snapshot. The target for the saved disk data must be the actual disk name (including any sequence number) because the current incarnation of a virtual machine may be derived from one or more snapshots.
Restoring disk data requires use of VixDiskLib. The VixDiskLib_Write() function opens the virtual disks so your program can write data. VixDiskLib functions transfer data to even-sector boundaries only, and transfer length must be an even multiple of the sector size.
The virtual disk already exists, so it is not necessary to restore the disk configuration information mentioned in Extract Backup Data from the Target Virtual Machine.
4
With SAN transport mode, revert-to and delete the snapshot that you created in Step 2. Failing to perform this step with SAN mode could yield a virtual machine that cannot be powered on.
Creating a New Virtual Machine
The process of building a virtual machine from backup data involves the following steps:
1
To create a new virtual machine, you use the information about virtual machine configuration that you derived and saved during the backup process. You might allow users of restore software an opportunity to rename the virtual machine during recovery in case they want to clone or move the virtual machine. Also you might consider offering them an opportunity to change virtual machine layout (for instance, storing virtual disks on a different datastore). Creating the virtual disks is also done at the time when you create the virtual machine. This process is fairly complicated. See the section Low Level Backup Procedures for details.
2
This process is similar to restoring the contents of virtual disks (Step 3 under Restoring an Existing Virtual Machine to a Previous State) with the following exception: you must call the VixDiskLib_WriteMetadata() function to write all the disk configuration key/value data into the virtual disk before restoring any backed-up data to the virtual disk. Then call VixDiskLib_Write() to restore the virtual disk data, as described in Step 3 above.
3
Accessing Files on Virtual Disks
It might be necessary for a backup application to access individual files or groups of files on the virtual disks. For example, data protection applications might need to restore individual files on demand.
You can find the interfaces to accomplish this in the VixMntapi library associated with VixDiskLib. The VixMntapi library allows disks or volumes of a virtual machine to be mounted and examined as needed. VixMntapi provides access at the file system level, whereas VixDiskLib provides access at the sector level.
To mount a virtual disk
1
2
Call VixDiskLib_Open() to open all of these virtual disks. This gives you a number of VixDiskLib handles, which you should store in an array.
3
Call VixMntapi_OpenDiskSet() to create a VixDiskSetHandle, passing in the array of VixDiskLib handles that you created in step 2.
4
Pass VixDiskSetHandle as a parameter to VixMntapi_GetVolumeHandles() to obtain an array of VixVolumeHandle pointers to all volumes in the disk set.
5
Call VixMntapi_GetOsInfo() to determine what kind of operating system is involved, and decide where important pieces of information are to be found.
6
For important volumes, call VixMntapi_MountVolume() then VixMntapi_GetVolumeInfo(), which reveals how the volume is set up. (Unimportant volumes include swap partitions.)
7
If you need information about how the guest operating system sees the data on this volume, you can look in the data structure VixVolumeInfo returned by VixMntapi_GetVolumeInfo(). For example, VixVolumeInfo::symbolicLink, obtained using VixMntapi_GetVolumeInfo(), is the path on the proxy where you can access the virtual disk’s file system using ordinary open, read, and write calls.
Once you are done accessing files in a mounted volume, there are VixMntapi procedures for taking down the abstraction that you created. These calls are:
VixMntapi_DismountVolume() for each volume handle
VixMntapi_FreeOsInfo() and VixMntapi_FreeVolumeInfo()
This leaves the VixDiskLib handles that you obtained in the beginning; you must dispose of them properly.
More VADP Details
The preceding sections explained how to contact vSphere and extract information from it, and how to back up or restore virtual disks. The following sections cover the same information at a lower level.