How to fix the Single-Sign On issue in the VMware Cloud Director 10.3.3 update

Introduction

Last week VMware has release VMware Cloud Director (VCD) 10.3.3. After a small wait from the previous 10.3.2a version but this version was the one I was lingering for! This release would have fixed a lot of the operational issues we have on the environment, which I will post several other blog posts for. Next to this we had to update our VCD environment just like everybody else due to the release of the following critical VMSA: https://www.vmware.com/security/advisories/VMSA-2022-0013.html

This VMSA fixes a critical remote code execution vulnerability (CVE-2022-22966) which should be fixed a.s.a.p. I got several e-mails from my VMware contacts, so I figured we should install this directly. Which we did the same evening and then the issues starting happening.

The release notes at the time of writing this blogpost do not state any of the issues we had. So please be aware off the following issue we had.

The Issue

The upgrade to VCD 10.3.3 went smooth. The upgrade procedure is very easy and straightforward in the newer versions. After the upgrade we tested the environment enxtensively. The next morning we received some issues about not being able to login to the VCD portal with our SSO application, which we found strange since we tested more or less all and everything from the portal. Local user login either through the Provider or the Tenant did work at this point in time.

We did notice that in VCD 10.3.3 there is a new button on the default Tenant/Provider URL. Previously if you browsed to the portal.tdl/provider or tenant URL you were automatically redirected to your configured SSO solution, albeit SAML, OpenConnect or anything else. In VCD 10.3.3 the default URL has changed in the sense that there is a direct possibility to login with local configured Provider/Tenant users and you have to press an additional button to be redirected to your SSO solution. You can see an example below:

Usually this button should redirect to your configured SSO solution. Which may look like the following if you are using Workspace One Access as your Identity Provider (IDP).

In our environment once we clicked on the “Sign In with Single Sign-On” button actually redirected back to the portal.tld/login page. On this page there is another default credential field after which you can press “Sign In”. This however does nothing. The “Sign In” button does not provide any feedback, not even through the browsers development tools.

The reason why this is happening is something that is not obvious at all, but something we have found out while troubleshooting:

1. A provider/customer logs in to their respective URL (Pre VCD 10.3.3) and will receive an auto redirect to their configured SSO solution. This can be seen below:

2. A provider/customer logs in to their respective URL (VCD 10.3.3) and will no longer receive the “Direct IDP ” redirect, but will instead be greeted with the login screen mentioned earlier in this post. This reflects to the following:

3. So far so good. Now once we click on the “Sign In with Single Sign-On” button it should receive the “IDP Redirect” like Pre VCD 10.3.3, except it doesn’t as you can see below:

As you can see it instantly redirects to the portals base login URL, which does nothing since it’s not connected (with a cookie for example) to any tenant/provider login. At this point it means customers cannot login to the enviroment with their configured SSO solution.

In our case we have connected Workspace One Access as our IDP. As a temporary workaround you can give your customers the “Launch URL” for the webapp within Workspace One Access. This URL is the direct URL from a specific Tenant to the IDP, which ensures that the correct sequence is followed and the tenants can login as usual with SSO.

The fix

The fix is something that I wouldn’t have though of if not given a hunch and some information by VMware support. It seems that this is related to the fact that we are using (or rather were at this moment) a custom branding theme with VCD. And a bit more specific, it’s because we are using a custom color in our branding. More information regarding custom branding can be found here. Basically with custom branding you can create a system wide or tenant specific branding so that the VCD portal feels like something from your company rather than a generic product that is being used and therefor increase the cloud experience. You can alter the logo, menu’s, colors and more.

Because we are using a custom color, which is defined in a hex code, it seems that a javascript error occures somewhere in the page (Uncaught TypeError: Cannot read properties of undefined (reading 'style')) . You can also see this in the browser’s development tools directly when opening the provider/tenant url:

If we have a closer look at the log file we can find the following line in the vcloud-container-info.log files:

2022-04-15 12:07:55,551 | SECURITY | pool-jetty-9026           | SsoFilterUtils                 | No matching sso interaction cookie for org System. Redirecting back to org selection page | requestId=0e10141ff460,request=GET https://xxxx.xxx/login/system/saml/login/alias/vcd,requestTime=xxxxxxxxx,remoteAddress=xxxxxxxxxx:xxxxxx,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=text/html application/xhtml+xml application/xml;q 0.9 image/avif image/webp image/apng */*;q 0.8 application/signed-exchange;...

The failed function should do something in regards to setting the colors for the branding. If a custom color is entered, it looks something as below:

function adjustHeaderBranding() {
        document.getElementsByClassName('header')
[0].style.background = '#FFFFFF';
    }

However, the “[0]” property is not defined, which causes the code to fail . If a default color of “null” is set, the two lines within the function are not present, and so the “[0]” property isn’t used, so the script does not crash.

Because the browser crashes on the function “adjustHeaderBranding” the next function is not called, which is the function that sets the following two cookies (sso-preferred and sso_redirect_org). That would be the following function:

function simplePageLoad() {
        hide('blankCanvas');

        adjustHeaderBranding();

        let ssoLoginButtonEl = document.getElementById('ssoLoginButton');
        if (!!ssoLoginButtonEl) {
            ssoLoginButtonEl.addEventListener(
                'click',
                function () {
                    // Set sso preference cookie
                    document.cookie = 'sso-preferred=yes';
                    document.cookie = 'sso_redirect_org=' + "System";
                }
            );

            document.getElementById('loginButton').addEventListener(
                'click',
                function () {
                    // Set sso preference cookie
                    document.cookie = 'sso-preferred=no';
                }
            );
        } else {
            document.cookie = 'sso-preferred=no';
        }
    }

These two Request cookies are essential in redirecting to a SSO solution. To demonstrate this, I’ve pasted two screenshots below from a working and non-working environment:

Working environment:

Not working enviroment:

Because of this, as mentioned before, the IDP location is not known to VCD and the redirect is not done.

VMware has confirmed to me that this is a code bug and engineering will have this fixed for VCD 10.3.3.1. An ETA for that release has not yet been made. To workaround the issue we need to remove the custom color branding from the environment. The easiest way to do this is to login to the provider tenant with the administrator@system user and follow the following steps:

1. Go to the API Explorer by clicking on the Question Mark in the upper right corner and click API Explorer. You can also directly go to it by opening your portal URL and adding /api-explorer/provider after it.
2. Now look up the “branding part”. You can see the following couple of calls (there are more).

3. Now use the first API Call called “Gets the system level branding” (GET /branding). Try it and press on Execute. You can see an example custom branding below:

{
  "portalName": "Unified Cloud Platform",
  "portalColor": "#454545",
  "selectedTheme": {
    "themeType": "BUILT_IN",
    "name": "Default"
  },
  "customLinks": [
    {
      "name": "help",
      "menuItemType": "override",
      "url": "https://support.XXXX.nl"
    },
    {
      "name": "imprint",
      "menuItemType": "override",
      "url": null
    },
    {
      "name": "about",
      "menuItemType": "override",
      "url": "https://www.XXXXX.nl/over-ons"
    },
    {
      "name": "vmrc",
      "menuItemType": "override",
      "url": "https://www.vmware.com/go/download-vmrc"
    }
  ]
}

Now have a close look at the third line which says what “portalColor” we have. In the API documentation it states the following:

A custom color defined by the administrator and used to theme the site. The color should be in rgb hex color format such as “FFFFFF” all capitalized. Can be set to an empty string to use the selected theme’s default colors.

4. We need to remove the “portalColor” value. This can be done by using “null” als value, without the “” marks. Enter the entire field from the previous code snippet, change the portcalColor value and use it in the following call: PUT /branding:

{
  "portalName": "Unified Cloud Platform",
  "portalColor": null,
  "selectedTheme": {
    "themeType": "BUILT_IN",
    "name": "Default"
  },
  "customLinks": [
    {
      "name": "help",
      "menuItemType": "override",
      "url": "https://support.XXXX.nl"
    },
    {
      "name": "imprint",
      "menuItemType": "override",
      "url": null
    },
    {
      "name": "about",
      "menuItemType": "override",
      "url": "https://www.XXXXX.nl/over-ons"
    },
    {
      "name": "vmrc",
      "menuItemType": "override",
      "url": "https://www.vmware.com/go/download-vmrc"
    }
  ]
}

5. At this point you can redo the API call from Step 3 to view that the “portalColor” value has been set to “null”.
6. Don’t forget to alter any Tenant specific custom color branding, because you also need to do this. You can do this be executing the same code but with the following API Call: PUT /branding/tenant/{org}

There you have it. Once this is done the SSO integration works flawlessly again. I hope that you guys read this blog post if you’ve encountered this issue to save yourselfes a lot of time troubleshooting this. In our environment it was pretty impacting since we only use SSO integrations.

Edit: It seems a KB has been published to address this issue: https://kb.vmware.com/s/article/88257