Recently I’ve been getting an error on one of my replication jobs in one of our Veeam environments. The error message was a bit vague, and to be honest, I’ve never seen it anywhere behore. The error message was:
The given key was not present in the dictionary.
If you browse to your log file location, usually under:
C:\ProgramData\Veeam\Backup\%JOBNAME%
You can find a more detailed error message like the one below:
[11.09.2019 21:06:28] <181> Info [SnapReplicaVmTarget] Getting disk '2006' datastore from DatastoreMap
[11.09.2019 21:06:28] <181> Error The given key was not present in the dictionary. (System.Collections.Generic.KeyNotFoundException)
[11.09.2019 21:06:28] <181> Error at System.ThrowHelper.ThrowKeyNotFoundException()
[11.09.2019 21:06:28] <181> Error at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTargetDatastoreMap.GetDiskDatastoreFromMap(Int32 diskKey)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.SyncDisks(IEnumerable`1 diskBackupObjects, CViReplicaProfilesResolver profilesResolver)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.PrepareConfig(CBackupClient sourceClient, CBackupObjectsCollection backupCollection, CViSnapReplicaVm replicaVm)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.ProcessConfig(CBackupClient sourceClient, CBackupSet backupSet)
[11.09.2019 21:06:28] <181> Error The given key was not present in the dictionary. (System.Collections.Generic.KeyNotFoundException)
[11.09.2019 21:06:28] <181> Error at System.ThrowHelper.ThrowKeyNotFoundException()
[11.09.2019 21:06:28] <181> Error at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTargetDatastoreMap.GetDiskDatastoreFromMap(Int32 diskKey)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.SyncDisks(IEnumerable`1 diskBackupObjects, CViReplicaProfilesResolver profilesResolver)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.PrepareConfig(CBackupClient sourceClient, CBackupObjectsCollection backupCollection, CViSnapReplicaVm replicaVm)
[11.09.2019 21:06:28] <181> Error at Veeam.Backup.Core.CViSnapReplicaVmTarget.ProcessConfig(CBackupClient sourceClient, CBackupSet backupSet)
[11.09.2019 21:06:28] <181> Info [AgentsSession] Disposing...
And however vague this error message is, this is it. There are no other log entries anywhere else in the system that could help out more. Just for the records of this issue, this environment is running Veeam version: 9.5 U3a.
After researching this error, I have found only four results on the web that could possibly help me, but none of them helped me out. One suggestion was that the SSO/PSC connection to the vCenter was faulty. However, this was not the case because we have several other replication jobs running on the same vCenter without any issues. The other solution from another blogger had something to do with permissions on a dvSwitch that were missing in his case. Nevertheless, this also is not the case for us for the same reason as I have mentioned before.
So, what did help? Well, I did a few things but the key thing I did was to delete the configuration entry of this virtual machine from the Veeam database. This is actually a rather silly step to execute, but I’m guessing the value for the virtual disks in the Veeam database were corrupted somehow. You can easily do this from the Veeam GUI following the next couple of steps:
- Login to your Veeam environment.
- Go to your backup or replica under ‘Backups’ or ‘Replicas’.
- Search and find the virtual machine that is giving this error.
- Right click this VM and select ‘Remove from configuration’.
- Go back to your backup or replica job.
- Re-configure the virtual machine in the backup or replication job.
- Re-run the backup or replication job.
That’s it, after this your backup or replication job should work again. One thing you can also do is to rescan your vCenters so that the Veeam database gets populated with new entries that could be missing. The good thing about this solution is that you do not even have to do a remapping of the replica virtual machine to the job. This happens automatically. I also did not have any ‘hash recalculation’ jobs for this replica once the fix was applied. Which Veeam usually does when something happened to a replica.
Applying the fix for the error also meant that we didn’t have to re-do the full replication for this virtual machine. Normally I would’ve just thrown away the replica VM and replicated the VM again, but since this was a very large machine this just wasn’t an option.
I hope this helps anybody that is having this issue!
**Edited and updated with new findings on 2020-01-04.
0 Comments