VDDK 5.5 Release Notes

Release date: 22 SEP 2013 | Builds: ESXi 1331820, VDDK 1284542
Last document update: 7 JUL 2015.
Check frequently for additions and updates to these release notes.

Contents



About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 5.5 is an update to support vSphere 5.5 and to resolve issues discovered in previous releases. VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software.

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 5.5 and all of its update releases support vSphere 5.0, 5.1, and 6.0 (except new features).

This release was tested with (and supports) the following 64-bit operating systems to perform proxy backup:

  • Windows Server 2003 R2
  • Windows Server 2008
  • Windows Server 2008 R2
  • Windows Server 2012
  • Red Hat Enterprise Linux RHEL 5.9, 6.2, 6.3, and 6.4 (as of 17 JAN)
  • SUSE Linux Enterprise Server SLES 10.4, 11.1, and 11.2.

VMware Consolidated Backup (VCB) has a writeup (knowledge base article) showing the support matrix for storage multipathing. VMware does not currently provide a similar support matrix for VDDK. Customers should seek this information from their backup software vendors.

For general information about this development kit, how to obtain and install the software, programming details, and redistribution, see the VDDK documentation landing page.

New in This Release

Higher limit on VMDK size. In vSphere 5.5, virtual machine disks can be larger than 2TB (two terabytes). The > 2TB virtual disk can reside on a VMFS-5 partition, but not on VMFS-3 or NFSv2. Check with your NAS vendor for information about disk size support on NFSv3.

More robust handling of linked clones. The snapshot and linked clone facility was improved for this release. VDDK programs can now open any disk in the chain of redo logs, not just the base disk.

No installers, packaged as ZIP. This VDDK release no longer uses the Windows .exe installer, but is instead delivered as an extractable ZIP archive.

Discontinued 32-bit libraries. Both the Windows and Linux packages are intended for 64-bit systems. The 32-bit libraries and binaries are no longer supported. For Visual Studio, 64-bit debugging tools must be installed.

Interoperation with VSAN. This VDDK release is able to back up virtual machines residing on VSAN. Despite its name, VSAN does not support SAN transport.

Compatibility Notices for Partners

New support for RHEL 6.4 proxy. On 17 January 2014, RedHat Enterprise Linux 6.4 was qualified as a supported proxy OS with VDDK 5.5.

Support for SEsparse disks. Virtual disk libraries can handle SEsparse disk as implemented in vSphere. SEsparse disks are used for snapshots of > 2TB disks, and for linked clones in the VMware View environment (VDI). Backup and restore of SEsparse is supported for the advanced transports (NBD, NBDSSL, HotAdd, and SAN) but not for the host-based file transport.

HotAdd behavior change in VDDK 5.5 with multiple disks. Rather than HotAdding all disks at connect time, as in earlier VDDK releases, in VDDK 5.5 each disk is HotAdded at VixDiskLib_Open() time, followed by reconfiguration for the disk. This affects performance as each open adds to the reconfiguration delay. However this change was necessary for the linked clone fixes mentioned above. Because programs may HotAdd just the base disk instead of a child disk chain, it is incorrect to HotAdd all disks at once. Similarly during HotRemove, each disk is removed at VixDiskLib_Close() time, followed by reconfiguration for that disk.

Cannot restore linked clones using SAN transport. This release includes revised snapshot and linked clone libraries that allow read access anywhere in the snapshot chain on any platform. During development VMware noticed errors when trying to restore multiple linked clones using SAN transport, which supports only one snapshot. To avoid errors the VDDK libraries were revised to explicitly disable writing to a disk that is not the base disk in a snapshot hierarchy. The libraries throw an error on open instead of waiting for fatal disk write errors. To restore linked clones, VMware recommends use of HotAdd, NBDSSL, or NBD transport. You can still use SAN transport to read (and back up) any disk in a snapshot hierarchy, and a linked clone backed up over SAN transport can be restored over other transports.

Disk mount and vdisk manager removed. The vmware-mount and vmware-vdiskmanager utilities are discontinued in this release.

Switch to Microsoft runtime 9.0. VDDK 5.5 libraries were compiled with Visual C++ 9.0 libraries, for compatibility with the glib called by VMware Tools. Developers must ensure that consumers of their software have installed the 9.0 runtime. Here are the Visual C++ 2008 SP1 redistributable packages: for x86 and for x86-64. This change does not affect Linux libraries.

DNS. To avoid SSL certificate errors, DNS service must be configured in the backup proxy, consistent with vCenter Server and ESXi hosts in a datacenter.

Linux proxy SAN backup can fail with powered on VM. If any virtual machine I/O operations are occurring for a VMDK on a LUN when the Linux proxy starts a SAN transport backup, proxy systems before SLES 11.2 and RHEL 6.2 receive an EIO instead of EBUSY error, causing backup to fail. This is more likely to occur with a powered on VM. The cause is a Linux kernel bug when handling SCSI reservation conflicts, fixed in newer kernels, and encountered on the following releases:

  • SLES 10.4 and SLES 11.1
  • RHEL 5.9, RHEL 6.0, and RHEL 6.1

For older compatibility notices from the VDDK 5.1 release, see the VDDK 5.1 Release Notes.

Recently Resolved Issues

The VDDK 5.5 release resolves the following issues.

  • Sporadic application exit during SAN restore. An issue was discovered where a stack overflow could occur during SAN restore on a physical proxy if any of its disks had 29 or more partitions. This resulted in the backup application exiting unexpectedly or crashing. Memory allocation algorithms were changed to prevent this issue.

  • HotAdd forward compatibility fails with error 13.

    When VDDK 5.1 or 5.1 U1 programs access HotAdded virtual disk on ESXi 5.5 hosts, VixDiskLib_Open fails when retrieving metadata before a disk gets HotAdded. The message is “Error 13 - You do not have access rights to this file.” This was fixed by changing the next VDDK 5.1 update release to pass the OPEN_LOCK option when retrieving disk metadata before HotAdding a disk. The fix will allow software compiled with VDDK 5.1 U2 to HotAdd disk on ESXi 5.5 hosts.

  • TCP timeouts no longer fail on Linux in NBD mode.

    On Linux systems, attempting to open a disk when already using NBD transport failed because of an attempt to configure the TCP socket keepalive with an invalid value. This was fixed by implementing TCP timeouts for NBD transport. The workaround of setting a long timeout in VixDiskLib_InitEx is no longer necessary.

  • Storage vMotion and changed block tracking.

    In earlier releases, VM cold relocation (virtual machine powered off) resulted in loss of changed block tracking (CBT) state. In this release, if source and destination datastores are accessible to both hosts, CBT state is maintained after cold relocation of VMs.

  • Backing up the HotAdd proxy causes left-over virtual disk.

    In all earlier releases, HotAdding virtual disk to back up the proxy itself resulted in left-over virtual disk. This disk could not be removed while the proxy was running, because ESXi hosts disallow removal of HotAdd disk, even by the vSphere (Web) Client. Now, VDDK throws the error "Cannot hot-add the proxy's own disks to itself" and switches to NBD mode if a program tries to HotAdd the proxy's virtual disk, thus preventing the problem. In a future release, VMware plans to set the VIX_E_NOT_SUPPORTED error also, but VDDK 5.5 does not produce this error code.

  • Snapshot delta files not consolidated after backup.

    In NBD mode, or after SAN backup, log messages appeared for *-delta.vmdk files saying “Device or resource busy” and snapshot consolidation failed with “The file is already in use” error. Reads or writes to disk failed because the connection was closed due to idle time. See the VDDK 5.1 U1 Release Notes for details. The issue is fixed in this release.

  • With application level quiescing, Changed Block Tracking overstates changes.

    When calling QueryChangedDiskAreas to get changes between application level quiesced snapshots, in previous releases the areas returned represented a larger amount of changes than the actual amount of changes. Application level quiesced snapshots are used to back up powered-on VMs running Windows Server 2003, 2008, 2008 R2, and 2012. The issue is fixed in this release, and in ESX 5.1 Patch 02. If vCenter Server 5.5 manages any ESXi 5.1 hosts, the patch should be applied to them. See KB 2052143 for details.

  • Calling PrepareForAccess more than once returns success.

    In vSphere 5.0 and 5.1, VixDiskLib_PrepareForAccess() returned successfully even if programs called it multiple times before calling VixDiskLib_EndAccess(). Multiple calls indicate a problem in program logic. In this release, the VIX_E_FAIL error is returned when programs make a redundant VixDiskLib_PrepareForAccess() call.

  • Connect call when non-existent snapshot is specified.

    If a program called VixDiskLib_ConnectEx() passing as third parameter the managed object reference (moRef) of a deleted or non-existent snapshot, the program crashed when dereferencing the snapshot moRef. This has been fixed in this release.

  • Default log level decreased to reduce space consumption.

    In this release the logging level was changed from TRIVIA (6) to INFO (3), resulting in smaller and more comprehensible backup logs. This change also has the benefit of reducing hangs in diskLibPlugin.dll.

  • Hang in connect or cleanup due to intermittent race condition.

    Windows processes could sometimes hang when calling VixDiskLib_ConnectEx() or VixDiskLib_Cleanup() due to a race condition while loading or unloading libraries. This has been fixed in this release.

  • Problem in GetMetdataKeys with a large number of snapshots.

    When backing up a virtual machine with 27 or more snapshots, the third call to VixDiskLib_GetMetadataKeys was unable to retrieve any metadata, possibly causing a software crash. This has been fixed in this release.

The VDDK 5.5 release resolves the following issues, which were also resolved in VDDK 5.1 U1.

  • Same glib version for VDDK and VMware Tools.

    The VDDK libraries for Windows are now built with Visual C++ version 9 to match the VMware Tools build environment, avoiding glib discrepancies that sometimes caused intermittent core dumps seemingly from VDDK code. See Compatability Notices above.

  • SLES 10 SP2 requires newer OpenSSL library.

    In this release, OpenSSL is updated from version libssl.so.0.9.8t to version libssl.so.0.9.8x so VDDK works correctly with SLES 10, SLES 10 SP1, and SLEs 10 SP2.

  • No segmentation violation during library disconnect.

    When many Linux processes ran simultaneously, some of them crashed with SIGSEGV when calling VixDiskLib_Disconnect() before exit. This sometimes occurred on Windows too. The issue is fixed in this release.

  • HotAdd open fails when volume spans multiple disks.

    When a Windows HotAdd proxy was configured with a volume that spans multiple disks, VixDiskLib_Open() failed to open the HotAdded virtual disk of the target virtual machine. The issue is fixed in this release.

  • Changed Block Tracking could return full thin-provisioned disk.

    On VMFS file systems holding very large thin-provisioned disk (> 200GB) calls to QueryChangedDiskAreas formerly returned incorrect results if the number of ioctl calls for query exceeded a fixed limit. This caused QueryChangedDiskAreas to return all areas (the full disk) rather than only the in-use areas of thin-provisioned disk. This resulted in longer backup time and excessively large backup data. It was especially noticeable on Linux ext2 and ext3 file systems. The issue is fixed in this release.

Known Issues and Workarounds

The following VDDK issues were reported after the vSphere 5.5 GA release.

  • Clone function fails connection with at-sign in password.

    If a user sets the access password to include at-sign (@) the VixDiskLib_Clone function will fail with “CnxAuthdConnect” or “NfcNewAuthdConnectionEx“ error. The issue is seen only when connecting directly to an ESXi host, and when the virtual machine's MoRef was not specified. This behavior is a regression from VDDK 5.1. The workaround is to set the root (or other login) password to avoid at-sign.

  • Slow read performance with NBD transport.

    When reading disk data using NBD transport, VDDK makes synchronous calls. That is, VDDK requests a block of data and waits for a response. The block is read from disk and copied into a buffer on the server side, then sent over the network. Meanwhile, no data gets copied over the network, adding to wait time. To some extent, you can overcome this limitation by using multiple streams to simultaneously read from a single disk or multiple disks, taking advantage of parallelism.

  • Writing to snapshot disk may be noticeably slower than writing to base disk.

    With all transport modes, calling VixDiskLib_Write() to a snapshot is sometimes faster than calling VixDiskLib_Write() to the base disk. However in certain circumstances, for instance with non-1KB buffer sizes, writing to the snapshot disk might be 2-3 times slower. Increasing the buffer size improves write speed to the base disk, but not to the snapshot disk. Note that taking a snapshot before recovery is no longer required in vSphere 5.5, except it is still mandatory for SAN transport.

  • Changed Block Tracking problem at 128GB boundaries.

    The QueryChangedDiskAreas function returns incorrect sectors after extending a virtual machine's VMDK file with Changed Block Tracking (CBT) enabled. See KB 2090639 for details.

  • Restore of virtual machine crashes during socket operations.

    During virtual machine restore, crashes have been reported due to an unexpected exception thrown by the vmacore library calling the async_connect function. This may be caused by network conditions such as loopback being rejected by a firewall. A fix has been identified and will be available in a future release.

  • Assert failure when getting NFC ticket during multiple VM backups.

    Before opening a virtual disk, the VMware libraries log into vCenter Server and acquire an NFC ticket. When multiple backup applications run on one host, or when one backup application simultanously backs up multiple virtual machines, the VMware libraries apparently have timer-related issues. Assertion failure when getting an NFC ticket causes the application to crash. A fix (insert negligible timeouts) was tested and will be available in a future release.

  • Cleanup function does not remove the HotAdd disks.

    After virtual disks are HotAdded with VixDiskLib_Open, they are not automatically cleaned up (hot-removed) by VixDiskLib_Cleanup. After disk is HotAdded or mounted, if a backup application crashes during backup due to network failure or power outage or any other issue, the leftover HotAdded disk must be cleaned up manually. Starting the backup application after a crash and calling VixDiskLib_Cleanup thereafter will clean up the mount directory, but will not hot-remove the disk. The fix, available in a future release, is to clean up or hot-remove any HotAdd disk based on mount information when VixDiskLib_Cleanup is called.

  • Connection may hang when NBD backup exceeds documented NFC buffer limit.

    ESXi 5.x hosts have a 32MB buffer allocated for NFC connections. When the dedicated ESXi memory is consumed, the next connection attempted can hang, although other established connections will continue normally. The following message appears in the ESXi host's vpxa.log file: “[NFC ERROR] NfcCheckAndReserveMem: Cannot allocate any more memory as NFC is already using bytes and allocating morebytes will make it more than the maximum allocated: 33554432. Please close some sessions and try again.” The workaround is to reduce the number of concurrent backups targeting the same ESXi host so as to remain within the documented limit. See KB 1022543.

  • Backup of >2TB disks could fail silently with HotAdd transport.

    When using HotAdd transport, backing up disks larger than 2TB can fail silently on Windows virtual machines. This issue occurs when disk.enableUUID is set True in the virtual machine. VDDK abruptly halts backup and wrongly returns success status. See KB 2068424.

The following VDDK issues were reported during the vSphere 5.5 testing cycle.

  • Always reboot after uninstalling VDDK on a Windows proxy.

    Attempts to install or uninstall the Windows vstor2 driver using the provided scripts can be unsuccessful under certain circumstances. When uninstalling VDDK from a Windows proxy, you should reboot after the uninstall. If this is not done, it may prove impossible to (re)install the vstor2 driver until the proxy is rebooted.

  • Threads still running after VixDiskLib_Exit call.

    When the VDDK plugin is loaded, several threads are started that cannot be shut down until the calling application exits. For instance on Windows Server 2008, VDDK libraries may leave threads running after the VixDiskLib_Exit call. Still-running threads are from the vCenter vmacore library and cannot be unloaded until the application finishes. The leftover threads should be harmless, but in some cases the calling application can terminate unexpectedly. This is a known issue that will be addressed in the first update release. Even with that fix, the threads will continue to run as before. The fix will address the logging attempts that were causing problems. Terminating the threads will not be addressed in VDDK 5.5 due to the extensive nature of required changes to the code.

  • Error code may not be set correctly before VDDK exit.

    An application calling VDDK libraries should return its own exit code, but QuickExit (part of the vmacore library) may terminate the VDDK application before it can set the proper exit code. This problem began when VDDK 5.1 was modified to call QuickExit. A fix has been identified, but it might never be applied to a VDDK 5.5 update release due to the extensive nature of required changes to the code.

  • Advanced transport requires more memory on SLES 11.1.

    On a SLES 11.1 proxy using SAN or HotAdd transport, you need to increase the maximum locked memory, memory size, and virtual memory. Do this by specifying unlimited for the ulimit -l, -m, and -v options. RHEL has higher default limits than SUSE Linux, or no limits.

  • Metadata write is not supported for HotAdd and SAN transport.

    When a program calls VixDiskLib_WriteMetadata with HotAdd or SAN transport, the function returns an error saying “The operation is not supported” and supplied metadata never gets written to virtual disk. The workaround is to close the disk from HotAdd or SAN mode, reopen the disk using NBD or NBDSSL mode, then call VixDiskLib_WriteMetadata.

  • HotAdd transport not possible with SATA virtual disks.

    ESXi 5.5 adds support for SATA disks in virtual machines created with, or upgraded to, virtual hardware version 10. However HotAdd advanced transport is not supported for SATA drives. So disk library code may reject requests for SATA disk with this error: “Cannot use [this] transport to mount virtual machines with non-SCSI disks.” This caveat does not apply to SATA disks in SCSI, iSCSI, or FC connected storage arrays.

  • Read-only mount of Windows ReFS with VixMntapi.

    On Windows Server 2012 systems with Resilient File Systems (ReFS), the VixMntapi library can mount ReFS partitions read-only, but not read/write.

  • Boot disk should be scsi0:0 for use with mount API.

    When a virtual machine's boot disk is not on scsi0:0, the VixMntApi_GetVolumeInfo function does not correctly return the inGuestMountPoints (such as drive letter C:) or the numGuestMountPoints. On a scsi0:0 disk the correct information is returned, but not otherwise. For example with a scsi0:2 boot disk, the two properties are not populated. This issue was recently reported against VDDK 5.0 and later.

  • Error 87 with Windows 2012 NTFS.

    NTFS virtual disk from Windows Server 2012 might produce “Error 87” when a backup proxy mounts the disk read-only on earlier versions of Windows Server. The VDDK log contains the error “Volume/NTFS filesystem is corrupt (87).” If customers encounter this issue, one workaround is to mount these virtual disks read/write so that the system has a chance to repair them.

  • Possible segmentation violation if running multiple backup processes.

    When many simultaneous backup processes are running, some of them might crash with a SIGSEGV after many iterations. This is due to a possible race condition in VixDiskLib, which can be reduced by calling VixDiskLib_Exit() at the end of your program.

  • Windows proxy HotAdd can cause differing disk signatures.

    When backup is done on a Windows proxy with HotAdd transport, Windows may rewrite the disk signature if it finds two identical signatures, resulting in the first sector of the backup not matching the original. To avoid this issue for backups, read the first sector of the disk using NBD transport, then read the remaining sectors using HotAdd transport. To avoid this issue for restores, write the entire disk using HotAdd transport, close the disk and its connection handle, open a new disk connection handle, and rewrite the first disk sector using NBD transport.

For VDDK issues that were identified during the vSphere 5.1 release, see the VDDK 5.1 Release Notes.