Recently I’ve discovered a nifty little issue in one of our environments. This environment consists out of VMware Cloud Director (VCD) in combination with VMware vSphere 6.7 hosts and Veeam v10. In this environment we make use of the VCD Veeam UI Plugin so that customers can use the Veeam Enterprise Manager plugin UI right from the VCD UI to create, edit and restore backup jobs. One of the features of this is that the customers can use Veeam File Level Restore (FLR) to restore their backed up GuestOS Files themselfes, without our intervention.
This works pretty well and we haven’t had any issues so far, except for this one. The customer wanted to restore a couple of files from one of their backed up VM’s. This however didn’t work as expected and the customer was provided with the following information:
The file cannot be accessed by the system.Failed to open file [\\?\C:\VeeamFLR\filexxxxxx] in readonly mode.--tr:Error code: 0x00000780Failed to open file [\\?\C:\VeeamFLR\filexxxxxx] for reading.--tr:Server unable to process [get_file] command.read: End of file--tr:Cannot read data from the socket. Requested data size: .--tr:Failed to read data from throttled device.--tr:Client unable to process [get_file] command. Source: [\\?\C:\VeeamFLR\filexxxxxx]. Target: [\\?\C:\ProgramData\Veeam\Backup\Temp\filexxxxxx].
Pretty obvious right? Well not really.
Troubleshooting the issue
So I started troubleshooting this issue. The following was list of items I checked:
- Can I do the restore myself under my administrator account without any issues? The answer was no, the same issue occurred.
- Can I manually see and copy the file from the FLR location (Which is the Mount Server configured on the Veeam backup repository)? The answer was yes I can see the files, but no I cannot copy the files, I received the following error
- What issues do I get when I do the FLR through the Veeam UI instead of through the Veeam Enterprise Manager UI? Well, I actually received the same error, as can be seen below:
- Do we see anything else from the Veeam Enterprise Manager UI that might help us figure out this issue? Well I did receive the following error message from the UI:
The last error message actually helped a lot. It turns out that while talking with the customer, the customer is using Windows Data Deduplication within the GuestOS to deduplicate files and save storage. This was actually the first time I’ve seen any VM use this inside the GuestOS. After carefully reading the Veeam documentation for GuestOS file restores, I’ve noticed the following line:
- [For restore to the original location] The Veeam Backup & Replication console and mount server associated with the backup repository must be installed on a machine running Microsoft Windows Server 2012 or later. Data Deduplication must be enabled on the mount server.
- [For restore to a new location] The Veeam Backup & Replication console must be installed on a machine running Microsoft Windows Server 2012 or later. Data Deduplication must be enabled on this machine.
- The version of Microsoft Windows Server on the mount server and Veeam Backup & Replication console must be the same or newer than the version of the VM guest OS.
After reading this I figured out that our Veeam Backup repository server didn’t have the Windows Data Deduplication role enabled on it. You don’t actually need to have the feature deduplicate data on your hard drives, but you do need to have the feature installed if you want to be able to restore deduplicated files that were backed up.
How to install Windows Data Deduplication Role
If you want to install the Windows Data Deduplication role you should follow the next few steps:
- Login to your Windows Server and open up Server Manager.
- Go to Manage -> Add Roles and Features -> Next -> Next -> Next
- Go to File and Storage Services -> Expand the view and click Data Deduplication -> Next.
- Now press on Install so that the feature is installed.
If you are using Windows Server 2016 and newer you don’t need to reboot. If you are using Windows Server 2012 (R2) you might need a reboot. Once you’ve done this the feature will be installed and you will be able to restore GuestOS files that were dedupped on the original backup source.
Fix number two
Now that that was out of the way I tried the restore again. This time I was greeted with another error message:
Alright so the issue isn’t completely fixed yet. This message however was quickly deciphered since I found a Veeam KB that accurately described the issue I was facing. It turns out that between the several major Windows Server versions such as 2012 (R2), 2016 and 2019 the Data Deduplication capabilities have changed. This isn’t that bad, but it is when you want to restore a Windows Server 2019 data deduplicated file, with a Windows Server 2012 R2 Veeam Mount Server. The differences between the two Data Deduplication features/versions is too much. They don’t have the same capabilities and therefore they cannot read the files properly. This means that the Data Deduplication capabilities must match between the source and destination. There are two possible fixes for this issue:
- Upgrade the Veeam Backup Repository Mount server to the highest available version off Windows Server, which at this point in time is Windows Server 2019. The Data Deduplication versions are backwards compatible with lower versions of Windows Server. But not forward compatible.
- Change the Mount Server on the Veeam Backup Repository to a server that has the same or higher OS.
The second option was the easiest for us, we also had another Veeam Backup Repository which was Windows Server 2019. So after configuring the newer server as a Mount Server, the restore worked.
Restoring GuestOS files from a GuestOS that has Windows Data Deduplication enabled created an issue for us. Once we found out that the Data Deduplication feature needed to be installed on the Veeam Mount Server, we quickly also found out that the Data Deduplication capabilities differ between the major Windows versions. So all-in-all, if you want to restore data that has been deduplicated, make sure you use a Veeam Backup Repository Mount Server that is at least the same or a newer Windows Server version.