Introduction

Recently we were in a situation in which we upgraded one of our Workspace ONE Access (formerly known as VMware Identity Manager) environments to the at that moment newest version 21.08 (Release notes here). All in all the upgrade went fine, however it seems that after the upgrade we had a couple of weird issues in the environment, that ultimately didn’t allow us to provision/assign new users/groups from the connected Active Directory to our Web Applications.

The issue

As mentioned earlier we were not able assign new users or groups to our Web Applications. This behaviour outed itself as such:

Workspace One Access unable to find the user
Workspace ONE Access unable to find the user

and

Workspace One Access unable to assign/find user to Web Application
Workspace ONE Access unable to assign/find user to Web Application

After the upgrade, and during the troubleshooting for this issue we did check all of the most basic items which in my opinion are:

  • WS ONE Access cluster health
    • The WS ONE Access cluster health was also 100%. You can check this by going to the Administration Console -> Dashboard -> System Diagnostics Dashboard -> Checking the DB connection and simply checking all of the nodes. An important thing to notice is the ElasticSearch Health.
  • WS ONE Access Node health
    • The Node health is 100%. You can check this by going to the Administration Console -> Dashboard -> System Diagnostics Dashboard -> click on a node and examine all the fields. If you need to refresh the checks with the refresh button on the right side.
  • WS ONE Access Elasticsearch health

The ElasticSearch cluster health can be checked by running the following commands and examining the output:

In the above output you can see that the cluster status is green, there are three nodes connected and no ‘unassigned’ shards available. The active shards percentage is also 100%. So far so good! You can also fire the following command to look at what WS ONE Access appliance is the master for the ElasticSearch application:

  • Back-end SQL cluster health.
    • The health for the SQL cluster was 100%. The user was found within the database in the table called: saas.Users.
  • WS ONE Access Connector (Directory Sync status) and health.
    • The health was 100% and we found the user “bvetest” being synced to the environment in the Directory Sync service log files. These can be found at: C:\Program Files\Workspace ONE Access\Directory Sync Service\logs

So as you can see, everything seems to be fine and healthy. We were troubleshooting for a while before we said let’s just examine a specific shard/index to see if we can see anything on that end. An index is made up of one or more shards. Each shard is as the official documentation states, a Lucine index, which you can think of as a self-contained search engine that indexes and handles queries for a subset of data. Once data is written to a shard, this gets pushed to the disk, after which it’s available for the search engine to use.

You can examine a shard by executing the following command (I’ve removed unrelevant data from the shard):

Now that is pretty interesting. We see some settings that are held in the shard, but we can also notice that the ‘read_only_allow_delete‘ value is ‘true‘. At this point we figured that at some point the environment went below the disk space watermark that ElasticSearch (especially the older versions) use to prevent damage to the cluster. If the threshold for free disk space if crossed, ElasticSearch can do two things, either lock up (read-only) the cluster, or put the indices on read-only. The latter part happend here. If we have a look at the log file /opt/vmware/horizon/workspace/logs/analytics-service.log we can also see that there are some errors in regard to pushing new data into elasticsearch:

If you really wanted. You can even relate the mentioned users in the above output with the ID’s for the users in the database to find which specific users are mentioned/having issues.

If we execute the following command it will display the settings for all shards:

We noticed that most, if not all shards were in read-only. Thankfully the disk space issue was already resolved but in ElasticSearch 6.x (which is running on WS ONE Access 21.08 appliances) it does not automatically revert the read-only indices once the disk space is back. So this is something we have to do manually by entering the following command (this is for all indices):

After this we can again read the shard and see if the setting changed:

Now that we’ve done this, we checked the environment again to see if we were now able to find the users and new groups within the Workspace ONE Access UI. It takes a couple of minutes for ElasticSearch to process everything, but once this is done the result is that it worked!

Workspace One Access able to find the user
Workspace ONE Access able to find the user

and

Workspace One Access unable to assign/find user to Web Application
Workspace ONE Access unable to assign/find user to Web Application

If we take a quick look at the log file at /opt/vmware/horizon/workspace/logs/analytics-service.log:

We can see that ElasticSearch is once again syncing the documents and maken them usable in the searchengine.

There you have it, some insights in WS ONE Access and the ElasticSearch engine that is running on these appliances. WS ONE Access can sometimes be quit complex to work with since there are a lot of moving parts required to have it up and running. I hope this blogpost helps you with troubleshooting your issues.

Leave a Reply

Your email address will not be published. Required fields are marked *