VCSA 6.5 U1c – Error 503 – vpxd.log duplicate key value violates unique constraint

While trying to deploy an OVA/OVF today I kept experiencing Error 503’s and VCSA would become unavailable.  If I tried to reload the web client I’d see:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f0ec80aaf10] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

After poking around for a while, I checked vpxd.log (located in /var/log/vmware/vpxd/) and saw several concerning things such as:

–> Panic: Unrecoverable VmRootError. Panic!

and, ultimately:

[VdbStatement] SQLError was thrown: “ODBC error: (23505) – ERROR: duplicate key value violates unique constraint “pk_vpx_dvport_membership”;

Ah!  Interesting.  After some searching I found the solution.  Take a snapshot of your VCSA or a backup of its Postgres DB.  SSH into your VCSA and run service vmware-vpxd stop.  Then, connect to the Postgres DB by issuing the following:

root@kcloud1vcsa2 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Next up, select some data from the VPX_DVS table:

VCDB=# select id,name,port_counter FROM VPX_DVS;
 id |   name    | port_counter
----+-----------+--------------
 25 | iSCSI vDS |            0
  9 | Data vDS  |           40
(2 rows)

Notice we get a vDS with id of 25 and 9.  With that, run the following command but specify the values your output return for DVS_ID :

VCDB=# select * from VPX_DVPORT_MEMBERSHIP where DVS_ID=9;
 dvport_key | dvs_id | dvportgroup_id | host_id | lag_key
------------+--------+----------------+---------+---------
 12         |      9 |             11 |       0 |
 13         |      9 |             11 |       0 |
 14         |      9 |             11 |       0 |
 15         |      9 |             11 |       0 |
 16         |      9 |             11 |       0 |

Notice the column called dvport_key.  Find the highest value, and add 1 for port_counter in the next statement.  For instance, in the output above, we’d be considering a value of 17 (16+1).  Now, run the following:

VCDB=# UPDATE VPX_DVS SET port_counter=17 WHERE id=9;
UPDATE 1

Obviously substitute your values.  You should get the “Update 1” return.  Once done for all vDS, quit the Postgres client by running:

VCDB-# \q

Finally, give your VCSA a reboot and you should be all fixed up!  Remember to delete your snapshot if you took one!

Thanks for reading let me know if you have this issue and whether or not the above fixes it.

Author: Jon

Share This Post On

Submit a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.