How to fix a vCenter 6.7 to 7.0 U3h upgrade failure

Introduction

Sometimes you can find some issues in older environment while upgrading that you haven’t seen before. This blogpost is about one that I’ve never seen before, but luckily is very easy to fix.We were upgrading the our environment from 6.7 U3x to 7.0 U3h. The upgrade eventually failed on the second step in the upgrade process where the services get started. It could not start vPostgreSQL. Let’s dive in!

Troubleshooting

After initially troubleshooting we didn’t find anything so we tried the upgrade again. But the second time gave us the same error after we did some more digging and found the following log entry in the log called /var/log/firstboot/vpostgres-firstboot.py_9697_stderr.log:

2023-01-21T08:49:15.154Z Upgrade import failed due to Error while deploying schema: psql.bin:/storage/seat/cis-export-folder/vcdb/VCDB_schema.dump:100: ERROR: could not open extension control file "/opt/vmware/vpostgres/13/share/extension/amcheck_next.control": No Such file or directory

2023-01-21T08:49:15.154Z Upgrade import step failed

2023-01-21T08:49:15.154Z vPostgres firstboot(action=firstboot) failed

Further diving into this it seems that this “amcheck_next.control” is a residue formed by executing the following script “vpgsql-consistency-check.sh” to check if the database can read all the data it has. The workings of the script can be found in VMware KB #53062. If you open up the script and look closely you can see the following:

function amcheck {
        amcheck_command=$1
        message=$2
        ## check if exists amcheck extension
        if [ -r /opt/vmware/vpostgres/current/share/extension/${extname}.control ]; then
           # amcheck extension Exists
           echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG
           PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -v ON_ERROR_STOP=on -w -c "$amcheck_command" >> /dev/null 2>>$ENV_ERROR_LOG
           if [[ $? == 0 ]] ; then
              echo -e "$(/bin/date "+%m-%d-%YT%T") Successfully finished the $message" | tee -a $ENV_ERROR_LOG
           else
              echo -e "The VCDB failed the $message. Please review $ENV_ERROR_LOG file for details.\n"
              error=1
	      exit 1
           fi
        else
           # amcheck extension is not installed on this system
           (echo -e "amcheck extension is not installed");
        fi
        #echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG
        #PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -w -c $amcheck_command 2>>$ENV_ERROR_LOG
}

function amcheck {

amcheck_command=$1

message=$2

## check if exists amcheck extension

if [ -r /opt/vmware/vpostgres/current/share/extension/${extname}.control ]; then

# amcheck extension Exists

echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG

PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -v ON_ERROR_STOP=on -w -c "$amcheck_command" >> /dev/null 2>>$ENV_ERROR_LOG

if [[ $? == 0 ]] ; then

echo -e "$(/bin/date "+%m-%d-%YT%T") Successfully finished the $message" | tee -a $ENV_ERROR_LOG

else

echo -e "The VCDB failed the $message. Please review $ENV_ERROR_LOG file for details.\n"

error=1

exit 1

else

# amcheck extension is not installed on this system

(echo -e "amcheck extension is not installed");

#echo -e "\n$(/bin/date "+%m-%d-%YT%T") Starting the $message " | tee -a $ENV_ERROR_LOG

#PGPASSWORD=$pgpass /opt/vmware/vpostgres/current/bin/psql -U postgres -h $PGHOST -d VCDB -w -c $amcheck_command 2>>$ENV_ERROR_LOG

}

It seems to check for the presence of the extension. A bit further in the script it executes the function. This also puts a pg_extension entry in the vPostgreSQL database, which apparently isn’t there anymore in the newer vPostgreSQL versions since those versions don’t use that library anymore. To check if this extension is present in your vPostgreSQL database execute the following commands:

root@vc01 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

psql.bin (9.6.24 (VMware Postgres 9.6.24.0-19410862 release))

Type "help" for help.

VCDB=# select * from pg_extension;

--------------+----------+--------------+----------------+------------+-----------+--------------

plpgsql | 10 | 11 | f | 1.0 | |

amcheck_next | 10 | 16388 | t | 2 | |

(2 rows)

VCDB=#

Here we can see the second entry which is called ‘amcheck_next’. This was the result for the consistency check mentioned earlier. Now the only thing we have to do is drop the extension by executing the following command (Make sure you have a backup just in case):

1	drop extension amcheck_next;

Once this is done the following should be the result if you execute the steps mentioned earlier again: