Don’t forget to do this one thing while upgrading to Workspace ONE Access 22.09!

Introduction

If you can remember I’ve made a blogpost quite recently about the issues that got up when we updated one of our Workspace ONE Access instances to 21.08. Our entire ElastichSearch engine eventually got in read-only mode which prevented us from using the product like we should be able to. Now I have another upgrade story for you guys. This time it’s about the upgrade we did to the latest version which is 22.09. It’s something you surely don’t need to forget. Because if you don’t, you will be in for a long restore procedure!

The issue

So like I said, the other day we upgraded the environment to 22.09 (eventually). It took a very long time because we had some issues with the upgrade.We actually followed the entire procedure that is usually well documented on the Workspace ONE Access upgrade guide. Because we have a hardened environment we don’t allow the Workspace ONE Access nodes to connect to the internet, everything is done through a proxy which is in front of it. This is why we use the Offline upgrade procedure, specifically the one with the updateoffline.hzn script.

So as the upgrade guide described, we did all the prerequisites for the offline upgrade, we even checked and did the regular upgrade prerequisites just to be sure. So after upgrading the first node the issue already popped up. We were (eventually, I will make another blogpost for this) able to login to the freshly upgraded node through the ‘break-glass’ URL to check if the environment was still working. At that point it seemed there was not much wrong. We were able to login to the new UI which is now fully Clarity enabled and looks a lot better. To give a quick look it should look like below once you’ve upgraded the first node:

Everything seemed to work, we had one upgraded node and two old nodes. The old nodes were together in an ElasticSearch cluster, and the new node was a single member (the previous master in the ElasticSearch cluster) for the now used OpenSearch cluster. OpenSearch is new in the 22.09 version. It’s just a different name since it’s a fork from the original ElasticSearch application. The only thing we noticed is that OpenSearch didn’t have any status at all, not a ‘OK’ or a ‘Bad’ status, nothing. If I recall right it was a NULL status. OK now that is not how it should be. Once we checked the cluster with the following commands it became quite obvious that the upgraded had failed for the first node for atleast the OpenSearch side:

curl 'http://localhost:9200/_cluster/state/nodes,master_node?pretty' <-- to check what Node is the Open/ElasticSearch Master node
curl 'http://localhost:9200/_cluster/health?pretty' <-- to check cluster health
curl 'http://localhost:9200/_cat/nodes?v' <-- to check detailed information for the hosts.

We doublechecked the upgrade log file which is located at: /opt/vmware/var/log/update.log. This logfile did not have anything that pointed at the fact that OpenSearch was not working anymore. It wouldn’t even start on our side. Everything succeeded without issues and the upgrade said:

Update completed, current version : 22.09.0.0 Time : Sat Oct 15 15:07:50 UTC 2022

So if everything was succesful, what was our issue? Well the OpenSearch log file at /opt/vmware/opensearch/logs/horizon.log did have some clues for us:

Caused by: java.lang.IllegalStateException: The index [[v4_2022-07-19/B3WXVpqhRu-6ejGuShgZcg]] was created with version [5.6.15] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in OpenSearch 6.x before upgrading to 1.3.5.
Date Entered: 10/15/2022

Hmm alright. So it seems that OpenSearch is seing older versions of the indices that are not compatible with the current version. Why is that? We followed everything in the upgrade guide to the letter. Well as you can maybe already point out, we still had older indices which are not supported with OpenSearch 1.3.5. Because of this the application wouldn’t start. This meant that we had to do a full revert on the environment. This took me a while but in the end made it work and now we had a fresh environment again. Fortunately there is an easy way to fix this. You have to do the following for this:

1. Upgrade the environment as described in the current Upgrade guide here.
2. Make sure to follow all prerequisites!
3. Once you start the upgrade it will go through some basic tests to see if your environment is able to upgrade. This should work OK.
4. At this point the upgrade script will ask you to do a re-index of all incides in the ElasticSearch environment. DO NOT skip this! You absolutely have to do this even though the upgrade guide for the 22.09 version does not mention this at all (as of the current date 22-10-2022).
5. If the procedure asks you to delete the incompatible indices, press Y. Again don’t skip this part either.
6. Once everything is re-indexed you can see the following as output in the screen (or the aforementioned upgrade log):

v4_2022-08-24 created on version 5.6.15
Reindexing required for v4_2022-08-24
Deleting 5.6.15 indices with old format
Would like to delete the incompatible indices ?
Enter [y] to Continue...y
..........................................
WARNING: Since elasticsearch version 5.6.15 index exist in system which is incompatible with opensearch, it is recommended to delete this index : v4_2022-08-18
---- Deleting index ----
Deleting index v4_2022-08-18
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    21  100    21    0     0    172      0 --:--:-- --:--:-- --:--:--   172
{"acknowledged":true}Deleted old formatted index v4_2022-08-18
Processing for index: v4_2022-08-24 ...
WARNING: Since elasticsearch version 5.6.15 index exist in system which is incompatible with opensearch, it is recommended to delete this index : v4_2022-08-24
---- Deleting index ----
Deleting index v4_2022-08-24
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    21  100    21    0     0    197      0 --:--:-- --:--:-- --:--:--   200
{"acknowledged":true}Deleted old formatted index v4_2022-08-24
INFO: Successfully completed reindexing of indices for migration process.

As you can see it will now re-index/re-create/delete every index if needed to make sure we can go ahead with the upgrade to OpenSearch 1.3.5. Once this is done and the upgrade log shows the upgrade is succesful, continue and go check your environment on issues. If there are no issues, continue the upgrade. Make absolutely sure you don’t do the re-index a second time. This is only required on the first upgraded node (preferably the ElasticSearch Master node).

Everything should now go according to the upgrade guide. Once this is done don’t forget to upgrade your Workspace ONE Connectors to the newest version available.

As to why VMware did not document this in the upgrade guide, that is completely unknown to me. Even the official OpenSearch upgrade procedure talks about this here. The previous version (21.08) did have this part in the upgrade guide. I’ve talked to VMware Support and provided them with feedback on the documentation. So hopefully this will get reflected very soon. Until then, you can use this blogpost to continue your upgrade! After that, enjoy the latest version of Workspace ONE Access which also provides means to enable regular RFC6238 Time-Based One-Time Password Algorithm (TOTP) authentication to be used. This was previously not possible in the On-Premises versions of Workspace ONE Access.