Today I was upgrading a VCSA 6.5 instance to 6.7 as I’ve done a dozen times. It’s a really simple setup – embedded PSC, etc. However, I just couldn’t get it to work!
I was constantly presented with “Error in getting source info” from the UI and the upgrade logs contained:
2018-07-18T12:47:08.855Z - info: VM with hostName [redacted-vc-fqdn]: { _type: 'VirtualMachine', _moid: 'vm-961', config: { template: false }, guest: { hostName: '[redacted-vc-fqdn]', ipAddress: '10.20.10.38', net: [ [Object] ] }, name: 'VMware vCenter Server Appliance', summary: { storage: { committed: 257543841825, uncommitted: 246668061721, unshared: 246689054720, timestamp: 2018-07-18T12:19:52.857Z } } } 2018-07-18T12:47:08.856Z - info: VM Identifier for Source VC: vm-961 2018-07-18T12:47:09.268Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.268Z - debug: Failed to get fileTransferInfo:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.269Z - debug: Failed to get url of file in guest vm:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.269Z - error: Error in getting fileData for nodeType. Error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.270Z - error: Failed to read the nodetype, Error: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.272Z - info: Checking if password expired
I know you’re thinking “well, yeah, it says ‘Checking if password expired’ …” but alas, it says it is not:
2018-07-18T12:47:12.549Z - info: STDOUT: * List APIs: "help api list" * List Plugins: "help pi list" * Launch BASH: "shell" 2018-07-18T12:47:12.551Z - info: STDOUT: Command> 2018-07-18T12:47:12.554Z - info: STDOUT: da 2018-07-18T12:47:12.555Z - info: STDOUT: t 2018-07-18T12:47:12.556Z - info: STDOUT: e 2018-07-18T12:47:12.567Z - info: STDOUT: Unknown command: `date' 2018-07-18T12:47:12.568Z - info: STDOUT: Command> 2018-07-18T12:47:12.568Z - info: STDOUT: ex 2018-07-18T12:47:12.569Z - info: STDOUT: i 2018-07-18T12:47:12.570Z - info: STDOUT: t 2018-07-18T12:47:12.652Z - info: Stream :: close 2018-07-18T12:47:12.652Z - info: Password not expired 2018-07-18T12:47:12.657Z - error: sourcePrecheck: error in getting source Info: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials.
There’s clearly an issue in either an API call or some mechanism checking the password expiration status since we’re seeing malformed STDOUT returns and the password is marked as not expired above.
Strange – the password is not expired… yet when I was in VAMI under Administration the password expiration showed that it had expired in 1970 which is usually an indicator that it has in fact expired. Oddly enough, though, I could SSH to and access the console of the VCSA with the known expired “not expired” password.
I ended up resetting the root user password using root@vc-fqdn [ ~ ]# passwd New password: where I reset the password to a temporary password, rebooted, and then reset the password back to the original password, rebooted, and then attempted the upgrade. At this point everything worked as expected.
Truly weird. If it were expired I’d have not been able to log in and would have had to use this VMware KB in order to reset the root credentials (I’ve had to do this before on other VCSA’s).
Debating on turning the original VCSA back on and checking the auth logs but… then again, I’d rather just move forward 🙂
Edit: I decided that pressing on would not be beneficial to the community. So, I resurrected the original VCSA and did some searching:
Turns out the root credential was in fact expired. Not sure why I was still able to login and SSH but that’s life.
Protip: You need to keep an eye on this (unlike I have!) because if the root account expired then the script (/etc/cron.daily/logrotate) that runs to prune logs will not successfully execute and you will fill up /dev/sda3 – learned this one the hard way many moons ago.
Hope you guys find this useful!
January 25, 2020
Here is the end of January 2020 and the error is still there 🙂
Thanks for documenting the issue … saved me some time 🙂
January 23, 2020
Years later, this bug still exists. Thanks for your help. Fixed it after just changing the password, no reboot required.
July 31, 2019
Thanks you just solved my problem. All I had to do was run passwd and change the password to the same password I already had. No reboot needed.
February 26, 2019
Thanks man! Really appreciate you taking the time to do this.
Cheers
Norman
February 13, 2019
Same issue, same fix. Thanks for the write up!
February 10, 2019
Thank you! working very well now
January 29, 2019
Thank you, that was extremely helpful!
January 2, 2019
Cheers, i was stuck on this 🙂
November 12, 2018
thanks!
October 4, 2018
Thanks for posting this; I had exactly the same issue with the expired ‘root’ password when attempting to upgrade to VCSA 6.7 and used your notes above to change the password via the BASH shell in an SSH session.
September 28, 2018
This root password is always such a PITA, given me so many headaches when it expires even though set to not expire! Come on VMware, thanks for the write up sir.
September 5, 2018
Thanks that was very helpful, I didn’t need to reboot.
August 24, 2018
Got the same error.
Changing the root password was the solution.
Thank you.
August 8, 2018
brought your site to the weekly newsletter on the office, I’m sure my colleagues will like it