In November and December 2019, Microsoft released a few new patches just like any other month to patch certain issues in Windows Server 2012. It has recently come to my attention that these patches seem to break Windows Server 2012 virtual machines! They appear to cause a reboot loop, which cannot be interrupted.
Since this was happening on a relatively large scale I didn’t know that the reboot loop was because of some faulty Windows patches. I figured I would just check the environment to see if anything infrastructure related was causing the issues. In a reboot loop this is highly unlikely, but hey, since this was happening on more than one virtual machine we just want to be sure right?
After some troubleshooting it seemed that, the reboot loop was a result of patching Windows Server 2012 virtual machines the incorrect way. Both the November 2019 Servicing Stack Update (SSU) KB4523208 and the December SSU KB4532920 for Windows Server 2012 seem to be causing the reboot loops according to some users on Twitter and Reddit. It seems that these two patches, as with most SSU patches, need to be installed exclusively (which Microsoft does mention in the installation procedure). This means they should be installed before any other patches get installed. When you install patches in the environment with WSUS or even unattended, the installation sequence will be ignored and the rollup patch gets installed before the SSU patch, which in this case makes the virtual server break. Some have said that patch KB890830 is also responsible for the reboot loop. When you are experiencing this issue, the servers keep getting stuck in the following screen:
Stage 2 of 4
Preparing to configure Windows
Do not turn off your computer
After which the virtual machine will just tell you:
Notifying services that Windows is shutting down.
And it will reboot. So how do we fix this? Well the resolution is quite simple but it depends on where you are in the process:
- If you haven’t installed the November or December SSU patch, you can proactively take care of the situation by following the installation guidelines for the patch. Which means, install these patches before the rollup patches. This way you should be fine.
- Did you already install the patch and are you experiencing the issue? Then you will have to boot into Safe Mode and reboot (twice sometimes). When the VM boots into Safe Mode the installation completes or fails and you can reboot and use the virtual machine as usual.
However, what do you do when you cannot boot into Safe Mode? If you are for example using EFI boot, booting into the Windows “Advanced Boot Options” menu is disabled. Some say it is disabled by default in Windows Server 2008 R2 and above to have faster boot times on the VM, even when using Legacy BIOS mode. I myself have not been able to proof this to be true though. I have done tests on Windows Server 2012, 2012 R2 and 2016 VM’s with Legacy BIOS enabled and every time I could hit F8 and boot into the Windows Advanced Boot Options menu.
So what do we need to do to get into the Windows Advanced Boot Options menu when we use UEFI Boot for our VM’s? Well we can get into the UEFI Boot Manager by pressing F2 when a VM boots. You will receive the UEFI Bios Manager screen such as below:
From there on you can do all sorts of things, but you can’t get the Windows Advanced Boot Options menu without doing some tricks. Below are three ways of entering the Windows Advanced Boot Options menu when using UEFI Boot:
- Connect the Windows 2012(R2) ISO.
- Start the VM hit F2 for the UEFI Boot Manager.
- Run the Virtual CDROM.
- Choose Use a Device -> UEFI Internal Shell (Unsupported Version).
- Press F8 on the selected GuestOS for advanced options.
- Press Save Mode (with options) and continue.
OR
- Start the VM and press F2 during boot.
- In the UEFI Boot Manager go to “Enter Setup” and select “Boot from a File”.
- Choose file with “NO VOLUME LABEL”. Press Enter twice.
- Go to “Microsoft” and press Enter.
- Press Enter on “Boot”.
- Go down to “bootmgfw.efi” and press Enter.
OR
- Start the VM and press F2 during boot.
- In UEFI Boot Manager go to UEFI Internal Shell (Unsupported Version).
- Let the shell load, do
map -r -b
(-b to break after one screen). - Choose the fs0 mapping or blk0 mapping by entering
fs0
: orblk0:
. - Enter
dir
orls
to view content of mapping/disk. - Go to the “UEFI” folder with
cd EFI
. - Enter
dir
orls
to view content of mapping/disk. Docd Boot
to enter the Boot folder. - Execute bootx64.efi by just entering
bootx64.efi
. - Press F8 on the selected GuestOS for advanced options.
- Press Save Mode (with options) and continue.
Windows Safe Mode Boot menu
If none of these work, it is probably because the option that shows the Windows Advanced Boot Options menu has been disabled in the Boot Configuration Data (BCD) configuration inside the GuestOS. You can check this by executing the following command from within the GuestOS and looking for the displaybootmenu option:
bcdedit
If you want to enable this menu you can do this by entering the following command:
bcdedit /set {bootmgr} displaybootmenu yes
If you want to disable the boot menu again, you can disable the menu manually by executing the following command inside Windows. Some have this setting disabled by default for security reasons.
bcdedit /set {bootmgr} displaybootmenu no
You can check if this had any effect by executing:
bcdedit
The displaytoolsmenu option should say “no”.
The following is a bonus! If you looked closely in the previous screenshots you can see that the UEFI Bios screenshots show a line called “Windows Boot Manager”. This option calls the Windows “Advanced Boot Options” menu from within the bios. This will work when you have the displaybootmenu option enabled. So calling this will save you time, because you don’t have to use one of the three procedures explained before to get to Windows Safe Mode. But what if you don’t have this line, like in the screenshot below? Well that is probably because your UEFI boot partition is corrupt or faulty.
It took me some time to figure this out, but don’t worry, I got you covered. There are ways to fix this, without doing a restore or destroying your virtual machine. Just follow the below steps and you will be fine:
- Either do this from within the GuestOS or from the Windows installation ISO.
- On a command line enter the following command:
bootrec /rebuildbcd
- This will search for Windows installations not included in the BCD store and will try to add it. If this command returns:
Scanning all disks for Windows installations. Please wait, since this may take a while... Successfully scanned Windows installations. Total identified Windows installations: 0 The operation completed successfully.
Then continue to step 4. But if this command returns:Scanning all disks for Windows installations. Please wait, since this may take a while... Successfully scanned Windows installations. Total identified Windows installations: 1 [1] D:\Windows Add installation to boot list? Yes/No/All: Y
Then end the procedure by executing step 3.
- Reboot the machine and you will see the Windows Boot Manager line in the bios.
- Since the bootrec command cannot find any Windows installation that haven’t been included in the bootrec configuration yet (obviously because we already have Windows installed), we will have to remove the current entry and reconfigure it. On a command line enter the following command:
bcdedit /export C:\bcdexport
- When you are using a Windows ISO to recreate the BCD configuration you will probably have to mount the recovery partition. This parition isn’t mounted by default. To do so just enter the following commands:
diskpart
list volume
select volume X
assign letter=v
list volume
(to check if the volume is mounted under a drive letter)
Then execute the command in the beginning of step 4 to export the BCD store as a file.
- When you are using a Windows ISO to recreate the BCD configuration you will probably have to mount the recovery partition. This parition isn’t mounted by default. To do so just enter the following commands:
- With the next step we will have to change the attributes on the BCD store file so that we edit it. Enter the following command:
attrib v:\EFI\Microsoft\Boot\bcd –h –r –s
(If you can’t find the file on “C:\boot\”, just search for it withdir bcd /s
like I did. The attrib command removes the hidden, read-only and system attributes from the BCD store. - Rename the old BCD store file so that it cannot be used anymore:
ren v:\EFI\Microsoft\Boot\bcd bcd.old
- Now execute step 1 again. This time you should receive the correct output. The bootrec command should now have added the refreshed BCD store file.
- Once this is done you can reboot the virtual machine.
Fix Windows EFI Partition
And now it is back again! The Windows Boot Manager option is back in the UEFI Bios. So there you have it, three ways of entering Windows Safe Mode when using UEFI boot for your VM’s on VMware vSphere. I hope this blogpost can help people that are having issues with the November and December 2019 SSU patches.
2 Comments
Baskar N · October 27, 2020 at 7:20 am
Thank you,
We had the Auto restart issue,
The solution got worked in Vmware 5.5 Environment with 2012 R2 standard.
After deployed patches the server got hang-up in restart loop.
Option-1 resolved- Went to Safe Mode 2 times OS restared and worked in normal boot.
Thanks a lot for the detailed solution for the issue..
Bryan van Eeden · October 28, 2020 at 11:34 am
No problem! Happy to help!