Upgrade

vCloud Director upgrade from 5.1 to 5.5

/ johannstander / Leave a comment

Current Version:

5.1.2 Build 1068441 vmware-vcloud-director-5.1.2-1968441.bin

New Version:

5.5.0 Build 1323688  vmware-vcloud-director-5.5.0-1323688.bin

I run a RHEL virtual machine with vcloud director installed, so before starting the upgrade I fully patch the RHEL environment.

PRE-CHECKLIST:

vCloud director 5.5 release notes:
https://www.vmware.com/support/vcd/doc/rel_notes_vcloud_director_55.html#sysreqs
Pre upgrade checklist for potential edge gateway problems:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2058977
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2060065

Upgrading vCloud director documentation from VMware which I recommend reading:
http://pubs.vmware.com/vcd-55/index.jsp?topic=%2Fcom.vmware.vcloud.install.doc_55%2FGUID-CEF834DA-1FF5-4819-9D24-88DE6F005C78.html

  1. As always, first off start with BACKUPS
    • Create snapshot of vCloud Director virtual machine.
    • Stop the vcloud services
    • backup the vCloud database.  We have a SQL server were the database resides.
    • Create another 3de party backup.  In my case i made use of Commvault to take a snapshot backup of the VM as well.
  2. Copy the downloaded bin file to vcloud director server.  I place the file in /tmp folder.
  3. Verify the MD5 checksum-value of file
  4. chmod u+x to make it executable.
  5. Stop the vcloud director services on server.
  6. Run the file installation file by typing “./”
  7. Respond to the upgrade prompts.
  8. Once completed DO NO start the vcloud services, firstly upgrade the vcloud database.
  9. Run /opt/vmware/vcloud-director/bin/upgrade
  10. Respond to upgrade prompts.  I did receive an error here which i explain at bottom of this blog!
  11. Once completed vCloud service should now be started.
  12. Upgrade vShield.
  13. From the vShield Manager Inventory panel, click Settings & Reports.
  14. Click the Updates tab.
  15. Click Upload Upgrade Bundle.
  16. Click Browse and select the VMware-vShield-Manager-upgrade-bundle-maintenance-5.1.2-997359.tar.gz file.
  17. Click Open.
  18. Click Upload File.
  19. Click Install to begin the upgrade process.
  20. Click Confirm Install. The upgrade process reboots vShield Manager, so you might lose connectivity to the vShield Manager. None of the other vShield components are rebooted.
  21. Verify the maintenance update has been applied

After you have upgraded vShield manager, you must upgrade all vCenter servers and hosts before you upgrade the vShield Edge appliances that the upgraded vShield Manager manages.

To upgrade ESXi host to 5.5 as well as upgrade the vCloud agent, perform the following steps in conjunction with vCloud director:

  1. From vCloud Director right click host and select disable the host
  2. Right click same host and select “Redeploy all VMs 
  3. On vCenter Server put the ESXi host into maintenance mode
  4. Attach host upgrade and patch baseline to ESXi server.
  5. Remediate host
  6. Once complete from vCloud director right click host and select “Upgrade host agent”
  7. Take host out of maintenance mode and wait for vSphere HA agent install to complete
  8. Within vCloud Director you can enable the host again.
To upgrade the vShield Edges:
  1. Login to vshield
  2. Select the datacenter for which you want to upgrade.
  3. Select network virtualization tab
  4. Select Edges
  5. Select the edge and click on actions -> Upgrade.
  6. I did find that after the edge upgrades the users was not able to get connection through the vCloud Edge gateway.  To resolve this i redeployed the Edge gateway with following steps:
    • Login to vCloud
    • Select organization
    • Select VDC
    • Select Edge Gateway tab
    • Right click edge gateway and select Re-Deploy.  
    • This will recreate the edge gateway but will not loose any settings configured on it.

Upgrade problems experienced:

During step 9 of database upgrade I received the following error message:
Error:   Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run ‘EXEC sp_updatestats’ as the DBO or a member of the sysadmin fixed server role.
Fix:   On the database server (SQL) provide the vCloud user account with sysadmin server role.  Run the command as provided in error against the database.

Error:  http error 500  after upgrade when opening the vCloud director login page
Fix:       Add the text“login.jsp” to the end of the vcloud page URL so you could use a local login.  Then disabled SSO under federation services within vCloud director if you are not using it.  I my case we make use of windows authentication and not SSO.

VUM errror during remediation of ESXi 5.1 to 5.5 host upgade: "Cannot run upgrade script on host" – resolved

/ johannstander / 8 Comments

VUM error during remediation of ESXi host upgrade from 5.1 to 5.5:
“vmware update manager 5.5 Cannot run upgrade script on host“

Debugging the problem:

Troubleshooting this problem led to a few discoveries online of users experiencing the same error message but different log entries regarding problem. This can be read in kb articles below:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007163
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014084

I tried with the VMware rollup ESXi as well as the Dell provided iso and had the exact same problem.

  • VMware-ESXi-5.5-RollupISO.iso
  • VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A01.iso

After the upgrade fails, SSH to your ESXi host and look for the following entries in \var\log\vua.log

2013-12-29T19:10:51.102Z [FFE898C0 info ‘VUA’] VUA exiting
2013-12-29T19:10:51.104Z [FFE898C0 error ‘Default’] Alert:WARNING: This application is not using QuickExit(). The exit code will be set to 0.@ bora/vim/lib/vmacore/main/service.cpp:147
–> Backtrace:
–> backtrace[00] rip 1a8272a3 Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
–> backtrace[01] rip 1a64e6e9 Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref&)
–> backtrace[02] rip 1a5d082f Vmacore::Service::Alert(char const*, char const*, int)
–> backtrace[03] rip 1a60f0e8 Vmacore::Service::AppImpl::Init(Vmacore::Service::Config*)::do_quick_exit::invoke()
–> backtrace[04] rip 1ae68ed9 /lib/libc.so.6(exit+0xe9) [0x1ae68ed9]
–> backtrace[05] rip 1ae52f04 /lib/libc.so.6(__libc_start_main+0xe4) [0x1ae52f04]
–> backtrace[06] rip 0804e5e1 /usr/share/vua/vua [0x804e5e1]

Solution:
I opened a case with VMware to get this resolved so if you experience the same problem I do recommend contacting VMware support to further assist. If you intend to use this fix, please do so at your own risk.

To fix this issue, manually remove the FDM agent on the host, reboot and retry the upgrade.
Note: Removing the host from a vSphere HA cluster also removes the agent. If the agent is not removed, you may have to manually remove the agent.

To manually remove the FDM agent from the host, run these commands:

cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp
chmod +x /tmp/VMware-fdm-uninstall.sh
/tmp/VMware-fdm-uninstall.sh
Reboot the host

After the FDM agent is removed and host has been rebooted you can run the remediation again for host upgrade.

vCenter Server upgrade from 5.1.1 to 5.5.0

/ johannstander / 1 Comment

Current version:

vCenter server:       5.1.0 Build 1063329
ESXi hosts:           5.1.0 Build 1117900

New version:

5.5.0b

ESXi software file:  VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A01   (we have Dell servers so using the latest Dell provided 5.5 installer)
VCenter server:      VMware-VIMSetup-all-5.5.0-1476387-20131201 (5.5.0b)

There are a lot of blogs and post and documentation on upgrading so i am not going to bore you with screenshots and detailed explanation but just give me short point summary of how my upgrade process:

PRE-CHECKLIST:
Firstly I would recommend reading the VMware best practices kb as well as the installation/upgrade guide for VMware:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052329

  1. As always, first off start with BACKUPS:
    • Create snapshot/backup of vCenter Server virtual machine. 
    • If available I recommend a third party backup/snapshot of vCenter for instance I made use of Commvault to a take a snapshot and store.  If vCenter upgrade fails it is not easy to revert/recover the snapshots since cannot start vCenter server.
    • Create SSO backup by selecting the following on server where SSO is installed  “Programs -> VMware -> Generate vCenter Single Sign-On backup bundle”
    • Create backups of databases for VCenter Server (VCDB), SSO (VCSSO) and Update Manager (VCUPDATE).
  2. Check COMPATIBILITY for vCenter plug-ins on new versions:
    • Verify that all third party plugins are compatible with the new version of VMware.
  3. SSO – seems to be a lot of users having issues with SSO upgrade with a few pitfalls.
    1. Warning 250000: Verify that your certificate for SSO is not expired, if so renew before upgrade. http://www.boche.net/blog/index.php/2013/11/12/single-sign-on-warning-25000/  (Very well written article on this warning and how to work through it)   I am still making use of my firstly created self-signed certificates and everything went fine.
    2. The registry key I had to change from IP to FQDN in order to install otherwise you get a notification during the installation. Look at my other blog with images of upgrade to see this error message.  KB 2060511 – Change registry for SSO “HKEY_LOCAL_MACHINE\Software\VMware, Inc.\VMware Infrastructure\SSOServer\FQDNIP” to be FQDN and not an IP address.  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=206051

INSTALLATION

Copy vCenter server install ISO to server and attached with virtual iso application for instance “Daemon tools”.  I do this because if server is rebooted then you have the ISO attached through vCenter server client or web client it will loose the connection.

Simple or custom component install?
My two cents:  I am not a big fan on the simple install because I don’t know what is exactly happening during the upgrade process and if a problem appears it is difficult to figure out which component failed, whereas with custom install you know which component you are installing.  Also if you have components installed on multiple servers you have to use the custom install.

We are going to perform below steps for vCenter Server upgrade – (the visual and text guidance in installer has been much improved by VMware)
1. Upgrade vCenter Single Sign-On
2. Upgrade vSphere Web Client
3. Upgrade vCenter Inventory Service
4. Upgrade vCenter Server
5. Upgrade vCenter Update manager
5. Upgrade vSphere client
6. Upgrade vCenter compatible plug-ins
7. Upgrade Distributed Switches

Upgrade problems experienced:
All of the upgrades went very smoothly without much interaction or problems except for the following –

Error:  Update manager vCenter health status failed.
Fix:      Change the service account for “VMware vSphere Update Manager Service” to same account as used for vCenter server service.

Error:   Storage monitoring service – initialization failed error on health status
Fix:      Change the service account for “VMware vSphere Profile-Driven Storage Service” to same account as used for vCenter server service. Restart service

UPGRADE ESXi: (error “Cannot run upgrade script on host”)

There are multiple methods to upgrade each ESXi host to new release which can be read here –
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2058352
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2032756

We make use of vSphere Update manage to upgrade ESXi hosts, so can only start on this once vCenter and update manager has been upgrade to 5.5:
Firstly import the new iso to ESXi images. (does not work while still on 5.1)
Give the iso a baseline name
Create new Baseline group called for instance “ESX host 5.5 upgrade”
Select “host upgrade” for host baseline.

I did run into errors with upgrading the ESXi host from 5.1 to 5.5 which is addressed and fixed in the follow up blog post.