The problem
Today we have another fascinating error with one of our tenants in our Veeam Backup for Office 365 environment. This tenant and its jobs have been around for months, and nothing actually changed on our side, or the tenant side. But all of a sudden our customer wasn’t able to run a OneDrive only backup job anymore. There was one site that was causing issues, which resulted in the following error message in the Veeam Backup for O365 GUI, and the job failed to start altogether:
26-1-2021 14:36:34 :: Failed to process OneDrive: Mail | (https://tenant). Input string was not in a correct format. :: 0:00:05
Since this small error message wasn’t giving me a lot to work with, I double checked the logs in the regular log file location and searched for the tenant:
C:\ProgramData\Veeam\Backup365\Logs>findstr /spin "tenant-name" *.log
This did yield a bit more, but nothing that was useful in any way. I talked to the customer and he also didn’t know of any changes that were made to the tenant. We also noticed that another backup job, that holds all Sharepoint sites, did work without any issues. Even for the affected site. This seemed weird to me, so I created a support ticket with Veeam (You can reference case number 04608132 if you want).
After some more troubleshooting we found that the backup job was failing over several dozen “.tmp” files that were present in the OneDrive:
3-2-2021 08:16:38 4 (2832) Warning: Block request failed (block ID: 113, range: 0-*, duration: 00:00:00.1946711, resource: xxxxxxx/DPdata/DP{I1800.TMP,
As it turns out, there is currently a small bug in the Veeam Backup for Office 365 5.0a release. If a given OneDrive box has files that have the following format: "DP{T4589.TMP” it will fail with the beforementioned error message. The serialization of these objects goes wrong and creates the error.
Conclusion
Fortunately, this is fixable! We’ve received a small fix with 4 new DLL’s that we had to place in the following folders on the Veeam Backup for Office 365 backup server (Since it’s not publicly available yet as far as I know, create a case with Veeam and reference my case number to get the files):
C:\Program Files\Veeam\Backup365
C:\Program Files\Veeam\Backup365\Packages\Proxy
Once you did this, you will have to restart the Veeam services (Veeam Backup for Microsoft Office 365 Service and Veeam Backup for Microsoft Office 365 RESTful API Service) on the backup server, and upgrade the backup proxies. This can be done through the GUI by going to Backup Infrastructure -> Backup Proxies -> Right click a proxy -> Upgrade. This will push the patched DLL’s to the proxy servers and activate the fix. Once this is done the error is gone.
Happy backupping!
0 Comments