New Query Allocated Blocks Function

As of vSphere 6.7, developers can call VixDiskLib_QueryAllocatedBlocks to accelerate the backup process, especially with large virtual disks. By knowing which blocks are allocated, an application can avoid reading irrelevant sectors and thus reduce data transfer during backup.

Function Signatures

The VixDiskLib_QueryAllocatedBlocks function synchronously retrieves a list of allocated blocks. On an open VMDK, it queries for allocated blocks by chunks. A chunk is a defined as a block of consecutive sectors. The calling program can specify chunk size to control granularity, along with starting sector and extent. 

VixError
VixDiskLib_QueryAllocatedBlocks(VixDiskLibHandle diskHandle,
                                VixDiskLibSectorType startSector,
                                VixDiskLibSectorType numSectors,
                                VixDiskLibSectorType chunkSize,
                                VixDiskLibBlockList **blockList )

Parameters

  • diskHandle [in] Handle to an open virtual disk.
  • startSector [in] Absolute offset, possibly zero (0).
  • numSectors [in] Number of sectors to query.
  • chunkSize [in] Minimum number of sectors covered by a chunk containing data.
  • blockList [out] Buffer containing a VixDiskLibBlockList.

Return: VIX_OK on success, otherwise a suitable VIX error code.

Backup programs must call the VixDiskLib_FreeBlockList function to deallocate the returned blockList when done with it. 

VixError 
VixDiskLib_FreeBlockList(VixDiskLibBlockList *blockList)

Parameters

  • blockList [out] Buffer with above returned VixDiskLibBlockList.

Return: VIX_OK on success, otherwise a suitable VIX error code.

About Query Allocated Blocks

This function supplements the QueryChangedDiskAreas function, which is the vSphere API for changed block tracking (CBT). Backup software can start by calling VixDiskLib_QueryAllocatedBlocks to get a list of allocated blocks, instead of calling QueryChangedDiskAreas with special change ID ("*") to return only active portions of a virtual disk. Calling QueryAllocatedBlocks is easier, and its implementation is simpler. Consequently the special change ID ("*") is deprecated in this release. It had issues with pre-existing snapshots, and was slow when used with extremely large disks.

Query allocated blocks does not depend on CBT. You can enable CBT either before or after the call. Although new in vSphere 6.7, QueryAllocatedBlocks works against vSphere 6.5 and 6.0 as well.

VixDiskLib_QueryAllocatedBlocks can be used on object-oriented datastores such as vSAN and VVols (virtual volumes). It is especially helpful for backing up thin-provisioned disks. To skip copying non-allocated regions, programs call QueryAllocatedBlocks to get the list of allocated blocks returned in the final parameter. This works for large sparsely allocated virtual disks even when CBT is not enabled.

Query allocated blocks should be called in the same thread as open and close disk, not from read and write worker threads.

One limitation is that VixDiskLib_QueryAllocatedBlocks returns the whole VMDK for NFS, rather than just the allocated blocks as for VMFS, vSAN, and VVol datastores.

In file transport mode, VixDiskLib_QueryAllocatedBlocks depends on the disk library to support local VMDK files. In NBD(SSL) transport mode, an NFC message queries the bitmap of allocated sectors. In SAN and HotAdd transport modes, raw disk in the guest OS does not contain allocated sector information, so the function depends on the aforementioned NFC message to get a bitmap of allocated sectors.

Use Cases for Query Allocated Blocks

To accelerate the initial full backup of allocated disk:

  1. Open the VMDK file.
  2. Query allocated blocks. If you have separate threads for open and read, call it in the open-close thread.
  3. Read and save the allocated blocks.
  4. Close the VMDK file. Repeat steps for subsequent VMDK files.

For an incremental or differential backup:

  1. Call QueryChangedDiskAreas with changeId of the epoch (point in time of last backup) to find all areas that have changed since the last backup. This is A1.
  2. Use VixDiskLib_QueryAllocatedBlocks to find all valid data sectors. This is A2.
  3. The intersection of the two sets (A1 ∩ A2) yields the regions that have changed and are allocated. The backup application should save the data in these regions (the intersection).
  4. The area A1 minus the intersection (A1 – ( A1 ∩ A2)) yields the regions that were changed but unmapped. The backup application can record those regions as zeroed without reading them.

This is highly useful when backing up an AUTO-UNMAP enabled VMDK. Since unmapped blocks can be recorded as "changed block" by CBT, without the QueryAllocatedBlocks call, those unmapped blocks get backed up unnecessarily. With the features together, as described in above, the real changed blocks (namely excluding unmapped) can be calculated.

Code Sample for Query Allocated Blocks

In the development kit, see the vixDiskLibSample.cpp program for its sample code related to chunkSize . The DoGetAllocatedBlocks routine calls the query allocated blocks function, after being requested by the -getallocatedblocks command line argument.