How to fix a vCenter 6.7 to 7.0 U3h upgrade failure

Introduction

Sometimes you can find some issues in older environment while upgrading that you haven’t seen before. This blogpost is about one that I’ve never seen before, but luckily is very easy to fix.We were upgrading the our environment from 6.7 U3x to 7.0 U3h. The upgrade eventually failed on the second step in the upgrade process where the services get started. It could not start vPostgreSQL. Let’s dive in!

Troubleshooting

After initially troubleshooting we didn’t find anything so we tried the upgrade again. But the second time gave us the same error after we did some more digging and found the following log entry in the log called /var/log/firstboot/vpostgres-firstboot.py_9697_stderr.log:

2023-01-21T08:49:15.154Z Upgrade import failed due to Error while deploying schema: psql.bin:/storage/seat/cis-export-folder/vcdb/VCDB_schema.dump:100:  ERROR:  could not open extension control file "/opt/vmware/vpostgres/13/share/extension/amcheck_next.control": No Such file or directory
2023-01-21T08:49:15.154Z Upgrade import step failed
2023-01-21T08:49:15.154Z vPostgres firstboot(action=firstboot) failed

Further diving into this it seems that this “amcheck_next.control” is a residue formed by executing the following script “vpgsql-consistency-check.sh” to check if the database can read all the data it has. The workings of the script can be found in VMware KB #53062. If you open up the script and look closely you can see the following:

function amcheck {
        amcheck_command=$1
        message=$2
        ## check if exists amcheck extension
        if [ -r /opt/vmware/vpostgres/current/share/extension/${extname}.control ]; then
           # amcheck extension Exists
           echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG
           PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -v ON_ERROR_STOP=on -w -c "$amcheck_command" >> /dev/null 2>>$ENV_ERROR_LOG
           if [[ $? == 0 ]] ; then
              echo -e "$(/bin/date "+%m-%d-%YT%T") Successfully finished the $message" | tee -a $ENV_ERROR_LOG
           else
              echo -e "The VCDB failed the $message. Please review $ENV_ERROR_LOG file for details.\n"
              error=1
	      exit 1
           fi
        else
           # amcheck extension is not installed on this system
           (echo -e "amcheck extension is not installed");
        fi
        #echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG
        #PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -w -c $amcheck_command 2>>$ENV_ERROR_LOG
}

It seems to check for the presence of the extension. A bit further in the script it executes the function. This also puts a pg_extension entry in the vPostgreSQL database, which apparently isn’t there anymore in the newer vPostgreSQL versions since those versions don’t use that library anymore. To check if this extension is present in your vPostgreSQL database execute the following commands:

root@vc01 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
psql.bin (9.6.24 (VMware Postgres 9.6.24.0-19410862 release))
Type "help" for help.

VCDB=# select * from pg_extension;
   extname    | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
--------------+----------+--------------+----------------+------------+-----------+--------------
 plpgsql      |       10 |           11 | f              | 1.0        |           |
 amcheck_next |       10 |        16388 | t              | 2          |           |
(2 rows)

VCDB=#

Here we can see the second entry which is called ‘amcheck_next’. This was the result for the consistency check mentioned earlier. Now the only thing we have to do is drop the extension by executing the following command (Make sure you have a backup just in case):

drop extension amcheck_next;

Once this is done the following should be the result if you execute the steps mentioned earlier again:

root@vc01 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
psql.bin (9.6.24 (VMware Postgres 9.6.24.0-19410862 release))
Type "help" for help.

VCDB=# select * from pg_extension;
   extname    | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
--------------+----------+--------------+----------------+------------+-----------+--------------
 plpgsql      |       10 |           11 | f              | 1.0        |           |
(1 row)

VCDB=#

There you have it! Now your vCenter Server upgrade will run through fine. It’s strange that the pre-upgrade checker does not notice you if there are missing libraries because it does check on this. If we would’ve known this earlier it would’ve made the upgrade that much quicker! I hope this helps for everybody that is facing this issue.