VMware vSphere Data Protection (VDP) Error and Time Synchronization

Using vSphere Data Protection (VDP), I have seen the following error a few times and it has appeared in forums so I thought it would make sense to post a short article on it.

“The most recent request has been rejected by the server. The most common cause for this error is that the times on the VDP appliance and your SSO server are not in sync.”

That is an example of a well-written error message – thank you, VDP developers. As the error suggests, it is likely an issue of time difference between the VDP appliance and the VMware vSphere Single Sign-On (SSO) server it is linked to. To resolve the issue, make sure the vSphere host that is running the VDP appliance has its NTP client configured properly and running. In the vSphere Web Client, this is done by selecting a host, clicking the Manage tab, and then clicking Time Configuration.

Do the same for the host running the SSO server using the same NTP server, of course. If you are running the SSO server on a Windows server (virtual or physical), make sure Windows is configured with the proper time zone for its location, as well. Last, but not least, make sure the latest version of VMware Tools is installed in your virtual machines. Most virtual appliances, including the vCenter Server and VDP appliance, come with VMware Tools already installed.

In a few situations, I have seen the need to restart VMware Tools in the vCenter Server and VDP virtual appliances to force a time sync with the vSphere host they are running on. This can be accomplished by running the following commands at the command line of each appliance:

(vCenter Server virtual appliance)# service vmware-tools-services restart

(VDP virtual appliance)# service vmware-tools restart

In a VMware virtualized environment and in just about any environment, for that matter, it is best practice to make sure…
1. DNS name resolution is configured and working properly – forward and reverse lookup, long name and short name.
2. All hosts are configured to use an accurate and reliable source of time – preferably, the same source.

Limitations of vSphere Data Protection (VDP) File Level Restore (FLR)

Purpose

This article provides information on the limitations of vSphere Data Protection (VDP) File Level Restore (FLR).

Resolution

Limitations of VDP FLR

  • FLR operations result in failure if you are using an older version of VMware Tools. Ensure to install the latest version of VMware Tools in the target virtual machines.
  • These virtual disk configurations are not supported by FLR:
    • Unformatted disks
    • Dynamic disks
    • GUID Partition Table (GPT) disks
    • Ext4 filesystems
    • FAT16 filesystems
    • FAT32 filesystems
    • Extented partitions
    • Virtual disk with more than one partition
    • Two or more virtual disks mapped to a single partition
    • Encrypted partitions
    • Compressed partitions
  • ACLs are not restored in FLR.
  • Symbolic Links cannot be restored or browsed.
  • You cannot restore more than 5000 folders or files in the same operation.
  • In logical volumes managed by LVM:
    • One physical volume (.vmdk) must be mapped exactly to one logical volume.
    • Only ext2 and ext3 formatting are supported.
  • When partitions are created, the lower ordered indices must be filled first. You cannot create a single partition and place it in the partition index 2,3, or 4. The single partition must be at partition index 1.

    For more information, see the File Level Restore Limitations section in the vSphere Data Protection Administration Guide

vSphere Data Protection (VDP) – FAQ

General

Q: What is VMware vSphere Data Protection (VDP)?
VDP is a robust, simple-to-deploy, disk-based backup and recovery solution. VDP is fully integrated with VMware vCenter Server and the VMware vSphere Web Client. VDP enables centralized and efficient management of backup jobs while storing backups in deduplicated destination storage.

Q: Is VDP replacing VDR (VMware Data Recovery)?
Yes. VDR is not supported with vSphere 5.1 and higher. VDR is being deprecated, but will be supported as detailed in the VMware Life Cycle Policies.

Q: Will I be able to use my backups from VDR with VDP?
Yes. A migration tool is included with the 5.1.10 release of VDP. This tool handles migration of data and restore points. Backup jobs cannot be migrated. For more information, see:

Q: Do I need additional licenses to use VDP 5.1?
VDP is included with vSphere Essentials Plus and higher.

Q: How is VDP deployed?
Like VDR, VDP is an appliance that is delivered as an OVA template. Each OVA is preconfigured with destination datastore sizes of 0.5 TB, 1 TB, and 2 TB. Note that the actual storage consumed by the appliance is greater than the destination datastore sizes.

Q: How does VDP compare to VDR?
VDR was a first generation solution bundled with vSphere 4 for the rapidly growing backup market and experienced rapid adoption by VMware customers. However, in the constant effort to deliver more value to customers, VMware has been actively working on improving data protection and disaster recovery with enhanced backup and replication solutions. This led VMware to introduce a new, more robust product in the form of VDP. To maximize customer value, VMware decided to collaborate with the EMC Avamar team, who has world-class industry leading expertise in backup and recovery technology to build the underlying foundation for VDP.

Q: Is it possible to mount a CIFS of NFS share directly from the appliance and use these as backup data destinations?
Mounting a CIFS or NFS share directly from or “within” the appliance is not supported. The VDP appliance (.vmdk files) can be deployed to a NFS share mounted by a vSphere host. Windows based NFS shares are not supported.

Q: Are there any limits for VDP?
VDP supports these specifications:

  • Each vCenter Server can support up to 10 VDP appliances.
  • Each VDP appliance supports backup for up to 100 virtual machines.
  • Only 1 VDP appliance can exist per ESXi/ESX host.
  • Support for 0.5 TB, 1 TB, or 2 TB of deduplicated backup data.

Q: Can VDP appliances work together as a unit sharing information such as backup job details, capacity information, etc?
No. Each VDP appliance operates independent of other VDP appliances.

Configuration

Q: VDP comes in 3 different OVA sizes: 0.5 TB, 1 TB, and 2 TB. Why is this, and is this all the space I need for the appliance?
VDP comes preconfigured with the destination datastores already attached. The OVA sizes indicate the size of the destination datastore. Additional space is required for the operating system and VDP operations (checkpoints, logs, etc). The amount of disk space required is:

OVA Size Disk Space Required
0.5 TB 850 GB
1 TB 1600 GB (1.57 TB)
2 TB 3100 GB (3.02 TB)

Notes

  • In VDP 5.5 one OVA size is available, you will be prompted to choose the size of your deduplication store during configuration.
  • When a VDP appliance is deployed, additional space cannot be added to an existing appliance. If more destination datastore capacity is needed, a new VDP appliance can be deployed (up to 10 appliances can be deployed per vCenter Server).

Q: Which VDP OVA size should I choose?
This depends on your environment: How many virtual machines are being backed up, the various OS types, and the amount of daily I/O being generated.

When a new VDP appliance is created, the destination datastore begins to fill rapidly on each new virtual machine backup added. This is due to the fact that each virtual machine being backed up contains unique data. To help reduce the amount of unique data being backed up, and therefore reduce the rate of growth on initial virtual machine backups, it is best to group similar backups together. For example, the initial backup of two Windows 2008 virtual machines requires less destination datastore space as similar data results in better deduplication efficiency. The initial backup of a Windows 2008 virtual machine and a Linux virtual machine would likely require more destination datastore space as there are unlikely to be similar data. After the initial backups, the VDP appliance backs up less unique data during subsequent backups. This is due to the Changed Block Tracking functionality in vSphere and deduplication in the VDP appliance.

However, if you are unsure of destination datastore capacity requirements, it is better to deploy a larger VDP appliance. When the appliance has been deployed, additional capacity cannot be added.

Q: Can I expand my destination datastore within VDP?
Yes, if using VDP Advanced. If using VDP Basic and require additional space, a new VDP appliance must be deployed (up to 10 per vCenter Server). It is not possible to migrate data from the original VDP basic appliance to a new, larger appliance. 

Q: Is DNS required for VDP to work?
Yes. Full DNS resolution (shortname, FQDN, forward, and reverse) must be configured correctly for VDP to work. If there are DNS issues, VDP operations, such as backups, are impacted. A DNS host record for a VDP appliance should be created prior to deploying the VDP appliance.

Q: What’s a blackout window?
The blackout window is the portion of each day reserved for performing server maintenance activities, such as garbage collection, which require unrestricted access to the server. Garbage collection deletes the orphaned chunks of data that are no longer referenced within any backups stored on the system. By default, the blackout window begins at 8 AM local server time and continues uninterrupted for three hours until 11 AM that same morning. This can be customized.

Note: No backup or administrative activities are allowed during the blackout window, however restores can be performed.

Q: What’s the difference between the blackout window and the maintenance window?
The maintenance window is the portion of each day reserved for performing routine server maintenance activities such as integrity check validation, whereas the blackout window performs server maintenance which requires unrestricted access.

Note: Backups can be initiated during the maintenance window, but doing so impacts both the backup and maintenance activities.

Q: Can I set a blackout window to 0?
No. A blackout window is required. It is possible to customize when the blackout window occurs.

Q: What are checkpoints and when would I be required to use them?
A checkpoint is a system-wide backup taken for the express purpose of assisting with disaster recovery. Checkpoints are scheduled and created once a day during the maintenance window. VDP stores two checkpoints (one validated and one not validated).

Q: What happens if rollback doesn’t work?
If a rollback fails, then deployment of a new VDP appliance is required. Migrating configuration and backup data from one appliance to another is not supported in VDP 5.1, this functionality was added in VDP 5.5.

Backup

Q: If I create a backup job during the backup window, will it start backing up immediately?
No. The backup jobs are scanned at the start of the backup window. If the job does not exist at this time, it will not run. It will be scheduled to automatically start at the next backup window. However, you can manually start a backup job during the backup window.

Q: Can I schedule backup jobs to run at different times?
There is only one backup window per VDP appliance. The backup jobs are run during this backup window. It is not possible to create separate backup windows for each backup job. When creating a backup job, it is possible to define how often the backup runs. For example, daily, weekly, or monthly.

Q: How many backups can run in parallel?
Each VDP appliance can simultaneously back up a maximum of eight virtual machines.

Restore

Q: Am I able to do a file-level restore (FLR) of my Linux virtual machines?
The file-level restore is web-based, so a supported web browser is required. At the time of writing, a command-line FLR is not available.

Q: Are there any prerequisites to use the FLR?
A web browser is required to connect to the VDP appliance. VMware Tools must also be running within the virtual machine where the file-level restore is occurring.

Q: Am I able to restore a backed-up virtual machine’s virtual disk to different datastores?
It is possible to restore a virtual machine to a datastore other than the original. However, all files that make up the restored virtual machine are restored to the same datastore.

Q: Can I perform a Restore Rehearsal, as I used to do in VDR?

Yes, restore rehearsal functionality is available in the VDP 5.1.10. For more information, see the vSphere Data Protection 5.1.10 Release Notes.

See Also