VCSA 6.5 -> 6.7 upgrade — Failed to authenticate with guest

Today I was upgrading a VCSA 6.5 instance to 6.7 as I’ve done a dozen times.  It’s a really simple setup – embedded PSC, etc.  However, I just couldn’t get it to work!

I was constantly presented with “Error in getting source info” from the UI and the upgrade logs contained:

2018-07-18T12:47:08.855Z - info: VM with hostName [redacted-vc-fqdn]: { _type: 'VirtualMachine',
  _moid: 'vm-961',
  config: { template: false },
  guest: 
   { hostName: '[redacted-vc-fqdn]',
     ipAddress: '10.20.10.38',
     net: [ [Object] ] },
  name: 'VMware vCenter Server Appliance',
  summary: 
   { storage: 
      { committed: 257543841825,
        uncommitted: 246668061721,
        unshared: 246689054720,
        timestamp: 2018-07-18T12:19:52.857Z } } }
2018-07-18T12:47:08.856Z - info: VM Identifier for Source VC: vm-961
2018-07-18T12:47:09.268Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.
2018-07-18T12:47:09.268Z - debug: Failed to get fileTransferInfo:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.
2018-07-18T12:47:09.269Z - debug: Failed to get url of file in guest vm:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.
2018-07-18T12:47:09.269Z - error: Error in getting fileData for nodeType. Error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.
2018-07-18T12:47:09.270Z - error: Failed to read the nodetype, Error: Failed to authenticate with the guest operating system using the supplied credentials.
2018-07-18T12:47:09.272Z - info: Checking if password expired

I know you’re thinking “well, yeah, it says ‘Checking if password expired’ …” but alas, it says it is not:

2018-07-18T12:47:12.549Z - info: STDOUT: 
    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"


2018-07-18T12:47:12.551Z - info: STDOUT: Command> 
2018-07-18T12:47:12.554Z - info: STDOUT: da
2018-07-18T12:47:12.555Z - info: STDOUT: t
2018-07-18T12:47:12.556Z - info: STDOUT: e

2018-07-18T12:47:12.567Z - info: STDOUT: Unknown command: `date'

2018-07-18T12:47:12.568Z - info: STDOUT: Command> 
2018-07-18T12:47:12.568Z - info: STDOUT: ex
2018-07-18T12:47:12.569Z - info: STDOUT: i
2018-07-18T12:47:12.570Z - info: STDOUT: t

2018-07-18T12:47:12.652Z - info: Stream :: close
2018-07-18T12:47:12.652Z - info: Password not expired
2018-07-18T12:47:12.657Z - error: sourcePrecheck: error in getting source Info: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.

There’s clearly an issue in either an API call or some mechanism checking the password expiration status since we’re seeing malformed STDOUT returns and the password is marked as not expired above.

Strange – the password is not expired… yet when I was in VAMI under Administration the password expiration showed that it had expired in 1970 which is usually an indicator that it has in fact expired.  Oddly enough, though, I could SSH to and access the console of the VCSA with the known expired “not expired” password.

I ended up resetting the root user password using root@vc-fqdn [ ~ ]# passwd New password:  where I reset the password to a temporary password, rebooted, and then reset the password back to the original password, rebooted, and then attempted the upgrade.  At this point everything worked as expected.

Truly weird.  If it were expired I’d have not been able to log in and would have had to use this VMware KB in order to reset the root credentials (I’ve had to do this before on other VCSA’s).

Debating on turning the original VCSA back on and checking the auth logs but…  then again, I’d rather just move forward 🙂

Edit:  I decided that pressing on would not be beneficial to the community.  So, I resurrected the original VCSA and did some searching:

Turns out the root credential was in fact expired.  Not sure why I was still able to login and SSH but that’s life.

Protip:  You need to keep an eye on this (unlike I have!) because if the root account expired then the script (/etc/cron.daily/logrotate) that runs to prune logs will not successfully execute and you will fill up /dev/sda3 – learned this one the hard way many moons ago.

Hope you guys find this useful!

Author: Jon

Share This Post On

14 Comments

  1. Here is the end of January 2020 and the error is still there 🙂
    Thanks for documenting the issue … saved me some time 🙂

    Post a Reply
  2. Years later, this bug still exists. Thanks for your help. Fixed it after just changing the password, no reboot required.

    Post a Reply
  3. Thanks you just solved my problem. All I had to do was run passwd and change the password to the same password I already had. No reboot needed.

    Post a Reply
  4. Thanks man! Really appreciate you taking the time to do this.

    Cheers

    Norman

    Post a Reply
  5. Same issue, same fix. Thanks for the write up!

    Post a Reply
  6. Thank you, that was extremely helpful!

    Post a Reply
  7. Cheers, i was stuck on this 🙂

    Post a Reply
  8. Thanks for posting this; I had exactly the same issue with the expired ‘root’ password when attempting to upgrade to VCSA 6.7 and used your notes above to change the password via the BASH shell in an SSH session.

    Post a Reply
  9. This root password is always such a PITA, given me so many headaches when it expires even though set to not expire! Come on VMware, thanks for the write up sir.

    Post a Reply
  10. Thanks that was very helpful, I didn’t need to reboot.

    Post a Reply
  11. Got the same error.
    Changing the root password was the solution.
    Thank you.

    Post a Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.