VMware did not disappoint! Finally NSX supports vSphere 6.5 and then some…

Yesterday VMware released both NSX for vSphere 6.3.o as well as the accompanying  vCenter 6.5a and ESXi 6.5a.

Before upgrading your NSX environment to 6.3 you would need to upgrade your vCenter and ESXi hosts to 6.5a as describe in the KB 2148841

It seems there was a pretty good reasons for VMware holding out on NSX so long since they introduced a lot of great new features with just a fews standouts listed below:

  • NSX kernel modules now independent of ESXi version: This enhancement helps reduce the chance of host upgrades failing due to incorrect kernel module versions since every ESXi upgrade in an NSX environment required at least 2 reboots.
  • Rebootless upgrade and uninstall on hosts: On vSphere 6.0 and later, once you have upgraded to NSX 6.3.0, any subsequent NSX VIB changes will not require a reboot. Instead hosts must enter maintenance mode to complete the VIB change.
  • NSX 6.3.0 also checks for NSX readiness before taking a host out of maintenance mode
  • Controller Disconnected Operation (CDO) mode: A new feature called Controller Disconnected Operation (CDO) mode has been introduced. This mode ensures that data plane connectivity is unaffected when hosts lose connectivity with the controller.
  • Compliance features for FIPS, Common Criteria and ICSA.
  • Improved Layer 2 VPN performance: Performance for Layer 2 VPN has been improved. This allows a single Edge appliance to support up to 1.5 Gb/s throughput, which is an improvement from the previous 750 Mb/s.

  • Linux support for Guest Introspection
  • Better interoperability between vCloud Director 8.20 and NSX 6.3.0 helps service providers offer advanced networking and security services to their tenants. vCD 8.20 with NSX 6.3.0 exposes native NSX capabilities supporting multiple tenants and tenant self-service.  (Very interested to test this and hopefully have a write up in upcoming weeks)








ESXi 6: cannot synchronize host

Today had error pop up on vCenter server for vpxa service in yellow state, also found that ESXi host was showing errors for “Cannot synchronize host” as well as “quick stats on is not up-of-date”


After doing some investigation found that one of the new hosts did not have a DNS entry created.

This should be by now a given but always test forward and reverse DNS lookup for ESXi host before adding to vCenter server!

ESXi reset system configurations

As some point in time you would want to re-use an ESXi host but do not want to go through the hassle of a complete reinstall and rather have quick solution.

I find the “reset system configuration” in DCUI very useful for this task.
There is also a way to perform this task via SSH:

#  /sbin/firmwareConfig.sh –reset   (this will automatically reboot your host)
# /sbin/firmwareConfig.sh –reset-only (this will not reboot host and needs to be done manually)

Mac Pro 6,1 rack environment running VMware ESXi 5.5 (with Fiber connectivity to VNX)

With the recently addition of Mac Pro 6,1 to VMware’s hardware compatibility I was eager to replace our exiting old Power Mac G5 towers in our environment.

  1. Mac Pro bios version MP61.88Z.0116.B05.1402141115
If your Mac Pro has an older boot ROM then just upgrade the Mac Pro to Yosemite (OS X 10.10) which contains the update to be applied to the Mac Pro.
  1. vSphere 5.5 P03 is required
Currently the latest version of ESXi available on VMware download is only 5.5 update 2 so you have to include the required patch version onto the update 2 ISO.  To do this perform the following steps:

  • Download the latest ESXi 5.5 Update 2 Driver rollup
  •  Download the offline bundle for ESXi Update 2 patch 3
  • Next you need to convert the offline bundle zip file into an ISO file to be placed on bootable USB stick.  To do this I used the VMware Image Builder which is available as part of PowerCLI.
    • After you installed PowerCli open the application
    • Change to folder location where zip file resides
    • Run command to add the offline bundle: 
      • ‘add-esxsoftwaredepot .\ESXi550-201412001.zip’
    • Run command to see the image profile: 
      • ‘get-esximageprofile’ 
    • Select the ESXi-5.5.0-20141204001-standard which include VMware tools and security patches.
      • Run Command:  
        • ‘New-EsxImageProfile -CloneProfile “ESXi-5.5.0-20141204001-standard” -name “ESXi55u2-P03-MACPRO” -Vendor MACPRO66’
    • Now you can create the ISO file with running command:
      • ‘Export-EsxImageProfile -ImageProfile “ESXi55u2-p03-MACPRO” -ExportToISO -filepath H:\VMware-ESXi-5.5u2-P03-MACPRO.iso’
    • This file can now be places on a bootable USB.
      • I use Universal-USB-installer or UnetBootin to place the ISO on the USB.
  • Boot ISO from MAC
    • press and hold the “ALT” key on keyboard to boot the USB. 
  • Rest is the same basic installation as with any regular Intel based server
Here is a list of all the hardware items used to in our environment:
  • Sonnet xMac Pro rackmount enclosure.
    • This the most valuable piece of equipment and I highly recommend this if you planning on placing your Mac Pro’s in a server rack. 
    • Comes with 3 x PCIe slots available through thunderbolt which provide 20Gbps throughput and flexibility which is unmatched and can now add extra network and even Fiber connections for storage.
    • Do yourself the favor and check them out:
    • UPDATE:
  • APPLE Mac Pro 6,1  
    • 12GB memory
    • Intel Xeon CPU E5-2697 v2 @ 2.70GHz
    • only purchase small memory size to be replaced with Transcend
  • Transcend 128GB Kit for Mac Pro
  • Intel I350-T4
    • 4 port network card.  We actually have two card installed in Sonnet.
    • This card is VMware compatible but not Sonnet however it works great without issues
  • 1 x Atto Celerity FC-81EN Fiber Channel adapter
  • APC AP7731
      • Since there are no dual power supplies on Mac Pro we purchased this APC switched rack PDU which takes two single-phase 220V drops and can switch power if you have a failure on one of over This provides redundancy even though you only have one cable.  However if the power adapter fails on hardware you are out of luck. 
Some gotchas experienced:
– We tried to run the updates for ESXi through VMware update manager and this caused the onboard NICs on Mac Pro to not be recognized anymore.  Re-installed the old version to resolve this.  Current build is 2302651
– To add storage on VNX a rescan does not seem to work so we had to restart the Mac Pro in order to pick up the LUNs.
– We initially installed all the PCI cards and then installed ESXi.  This cause the network cards numbering to go out of whack.  What we had to do was remove all the cards and power on ESXi and let it complete the startup.  Then shutdown and add a single PCI card  and power on again, Do this one at a time in order you want starting from bottom.    Should fix the network port order.
UPDATE:  created a new blog to show how within vCloud Director to run a MAC OS X and Windows VMs within the same vAPP


Here are some photos of our build:

Started installation

ESXi installed and ready for use

Internal and external casing

Internal casing housing the Mac Pro

work bench

rear of Sonnett chassis. very nice

Mac Pro housing

More Mac Pro housing

Mac Pro ready to be installed

Now for some pictures in the rack:

Rear of Rack

Front of Rack..so nice and clean!

vCloud Director upgrade from 5.1 to 5.5

Current Version:

5.1.2 Build 1068441 vmware-vcloud-director-5.1.2-1968441.bin

New Version:

5.5.0 Build 1323688  vmware-vcloud-director-5.5.0-1323688.bin

I run a RHEL virtual machine with vcloud director installed, so before starting the upgrade I fully patch the RHEL environment.


vCloud director 5.5 release notes:
Pre upgrade checklist for potential edge gateway problems:

Upgrading vCloud director documentation from VMware which I recommend reading:

  1. As always, first off start with BACKUPS
    • Create snapshot of vCloud Director virtual machine.
    • Stop the vcloud services
    • backup the vCloud database.  We have a SQL server were the database resides.
    • Create another 3de party backup.  In my case i made use of Commvault to take a snapshot backup of the VM as well.
  2. Copy the downloaded bin file to vcloud director server.  I place the file in /tmp folder.
  3. Verify the MD5 checksum-value of file
  4. chmod u+x to make it executable.
  5. Stop the vcloud director services on server.
  6. Run the file installation file by typing “./”
  7. Respond to the upgrade prompts.
  8. Once completed DO NO start the vcloud services, firstly upgrade the vcloud database.
  9. Run /opt/vmware/vcloud-director/bin/upgrade
  10. Respond to upgrade prompts.  I did receive an error here which i explain at bottom of this blog!
  11. Once completed vCloud service should now be started.
  12. Upgrade vShield.
  13. From the vShield Manager Inventory panel, click Settings & Reports.
  14. Click the Updates tab.
  15. Click Upload Upgrade Bundle.
  16. Click Browse and select the VMware-vShield-Manager-upgrade-bundle-maintenance-5.1.2-997359.tar.gz file.
  17. Click Open.
  18. Click Upload File.
  19. Click Install to begin the upgrade process.
  20. Click Confirm Install. The upgrade process reboots vShield Manager, so you might lose connectivity to the vShield Manager. None of the other vShield components are rebooted.
  21. Verify the maintenance update has been applied

After you have upgraded vShield manager, you must upgrade all vCenter servers and hosts before you upgrade the vShield Edge appliances that the upgraded vShield Manager manages.

To upgrade ESXi host to 5.5 as well as upgrade the vCloud agent, perform the following steps in conjunction with vCloud director:

  1. From vCloud Director right click host and select disable the host
  2. Right click same host and select “Redeploy all VMs 
  3. On vCenter Server put the ESXi host into maintenance mode
  4. Attach host upgrade and patch baseline to ESXi server.
  5. Remediate host
  6. Once complete from vCloud director right click host and select “Upgrade host agent”
  7. Take host out of maintenance mode and wait for vSphere HA agent install to complete
  8. Within vCloud Director you can enable the host again.
To upgrade the vShield Edges:
  1. Login to vshield
  2. Select the datacenter for which you want to upgrade.
  3. Select network virtualization tab
  4. Select Edges
  5. Select the edge and click on actions -> Upgrade.
  6. I did find that after the edge upgrades the users was not able to get connection through the vCloud Edge gateway.  To resolve this i redeployed the Edge gateway with following steps:
    • Login to vCloud
    • Select organization
    • Select VDC
    • Select Edge Gateway tab
    • Right click edge gateway and select Re-Deploy.  
    • This will recreate the edge gateway but will not loose any settings configured on it.

Upgrade problems experienced:

During step 9 of database upgrade I received the following error message:
Error:   Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run ‘EXEC sp_updatestats’ as the DBO or a member of the sysadmin fixed server role.
Fix:   On the database server (SQL) provide the vCloud user account with sysadmin server role.  Run the command as provided in error against the database.

Error:  http error 500  after upgrade when opening the vCloud director login page
Fix:       Add the text“login.jsp” to the end of the vcloud page URL so you could use a local login.  Then disabled SSO under federation services within vCloud director if you are not using it.  I my case we make use of windows authentication and not SSO.

VUM errror during remediation of ESXi 5.1 to 5.5 host upgade: "Cannot run upgrade script on host" – resolved

VUM error during remediation of ESXi host upgrade from 5.1 to 5.5:
“vmware update manager 5.5 Cannot run upgrade script on host

Debugging the problem:

Troubleshooting this problem led to a few discoveries online of users experiencing the same error message but different log entries regarding problem. This can be read in kb articles below:

I tried with the VMware rollup ESXi as well as the Dell provided iso and had the exact same problem.

  • VMware-ESXi-5.5-RollupISO.iso
  • VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A01.iso

After the upgrade fails, SSH to your ESXi host and look for the following entries in \var\log\vua.log

2013-12-29T19:10:51.102Z [FFE898C0 info ‘VUA’] VUA exiting
2013-12-29T19:10:51.104Z [FFE898C0 error ‘Default’] Alert:WARNING: This application is not using QuickExit(). The exit code will be set to 0.@ bora/vim/lib/vmacore/main/service.cpp:147
–> Backtrace:
–> backtrace[00] rip 1a8272a3 Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
–> backtrace[01] rip 1a64e6e9 Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref&)
–> backtrace[02] rip 1a5d082f Vmacore::Service::Alert(char const*, char const*, int)
–> backtrace[03] rip 1a60f0e8 Vmacore::Service::AppImpl::Init(Vmacore::Service::Config*)::do_quick_exit::invoke()
–> backtrace[04] rip 1ae68ed9 /lib/libc.so.6(exit+0xe9) [0x1ae68ed9]
–> backtrace[05] rip 1ae52f04 /lib/libc.so.6(__libc_start_main+0xe4) [0x1ae52f04]
–> backtrace[06] rip 0804e5e1 /usr/share/vua/vua [0x804e5e1]

I opened a case with VMware to get this resolved so if you experience the same problem I do recommend contacting VMware support to further assist. If you intend to use this fix, please do so at your own risk.

To fix this issue, manually remove the FDM agent on the host, reboot and retry the upgrade.
Note: Removing the host from a vSphere HA cluster also removes the agent. If the agent is not removed, you may have to manually remove the agent.

To manually remove the FDM agent from the host, run these commands:

cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp
chmod +x /tmp/VMware-fdm-uninstall.sh
Reboot the host

After the FDM agent is removed and host has been rebooted you can run the remediation again for host upgrade.

vCenter Server upgrade from 5.1.1 to 5.5.0

Current version:

vCenter server:       5.1.0 Build 1063329
ESXi hosts:           5.1.0 Build 1117900

New version:


ESXi software file:  VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A01   (we have Dell servers so using the latest Dell provided 5.5 installer)
VCenter server:      VMware-VIMSetup-all-5.5.0-1476387-20131201 (5.5.0b)

There are a lot of blogs and post and documentation on upgrading so i am not going to bore you with screenshots and detailed explanation but just give me short point summary of how my upgrade process:

Firstly I would recommend reading the VMware best practices kb as well as the installation/upgrade guide for VMware:

  1. As always, first off start with BACKUPS:
    • Create snapshot/backup of vCenter Server virtual machine. 
    • If available I recommend a third party backup/snapshot of vCenter for instance I made use of Commvault to a take a snapshot and store.  If vCenter upgrade fails it is not easy to revert/recover the snapshots since cannot start vCenter server.
    • Create SSO backup by selecting the following on server where SSO is installed  “Programs -> VMware -> Generate vCenter Single Sign-On backup bundle”
    • Create backups of databases for VCenter Server (VCDB), SSO (VCSSO) and Update Manager (VCUPDATE).
  2. Check COMPATIBILITY for vCenter plug-ins on new versions:
    • Verify that all third party plugins are compatible with the new version of VMware.
  3. SSO – seems to be a lot of users having issues with SSO upgrade with a few pitfalls.
    1. Warning 250000: Verify that your certificate for SSO is not expired, if so renew before upgrade. http://www.boche.net/blog/index.php/2013/11/12/single-sign-on-warning-25000/  (Very well written article on this warning and how to work through it)   I am still making use of my firstly created self-signed certificates and everything went fine.
    2. The registry key I had to change from IP to FQDN in order to install otherwise you get a notification during the installation. Look at my other blog with images of upgrade to see this error message.  KB 2060511 – Change registry for SSO “HKEY_LOCAL_MACHINE\Software\VMware, Inc.\VMware Infrastructure\SSOServer\FQDNIP” to be FQDN and not an IP address.  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=206051


Copy vCenter server install ISO to server and attached with virtual iso application for instance “Daemon tools”.  I do this because if server is rebooted then you have the ISO attached through vCenter server client or web client it will loose the connection.

Simple or custom component install?
My two cents:  I am not a big fan on the simple install because I don’t know what is exactly happening during the upgrade process and if a problem appears it is difficult to figure out which component failed, whereas with custom install you know which component you are installing.  Also if you have components installed on multiple servers you have to use the custom install.

We are going to perform below steps for vCenter Server upgrade – (the visual and text guidance in installer has been much improved by VMware)
1. Upgrade vCenter Single Sign-On
2. Upgrade vSphere Web Client
3. Upgrade vCenter Inventory Service
4. Upgrade vCenter Server
5. Upgrade vCenter Update manager
5. Upgrade vSphere client
6. Upgrade vCenter compatible plug-ins
7. Upgrade Distributed Switches

Upgrade problems experienced:
All of the upgrades went very smoothly without much interaction or problems except for the following –

Error:  Update manager vCenter health status failed.
Fix:      Change the service account for “VMware vSphere Update Manager Service” to same account as used for vCenter server service.

Error:   Storage monitoring service – initialization failed error on health status
Fix:      Change the service account for “VMware vSphere Profile-Driven Storage Service” to same account as used for vCenter server service. Restart service

UPGRADE ESXi: (error “Cannot run upgrade script on host”)

There are multiple methods to upgrade each ESXi host to new release which can be read here –

We make use of vSphere Update manage to upgrade ESXi hosts, so can only start on this once vCenter and update manager has been upgrade to 5.5:
Firstly import the new iso to ESXi images. (does not work while still on 5.1)
Give the iso a baseline name
Create new Baseline group called for instance “ESX host 5.5 upgrade”
Select “host upgrade” for host baseline.

I did run into errors with upgrading the ESXi host from 5.1 to 5.5 which is addressed and fixed in the follow up blog post.

ESXi 5.1 – network port lost connection without notification – Dell Poweredge

I ran into a very serious bug the other day which caused all our production Virtual Servers to loose network connectivity on a specific ESXi host without any notifications or alarms from VMware VCenter server.  Actually VMware was completed unaware of this problem and just continued working as normal with all our servers offline.

Debugging the problem:

Reviewed all the physical ports and found that a particular NIC lost its VLAN’s and VMware did not recognize this so all VM’s was left in disconnected state.  No network errors was detected on the Cisco physical switches as well as from VMware so no fail over took place

Quick fix:

When to teaming and failover within each Port Group where VM’s are affected and removed the problem NIC from Active uplinks.

VMware support case opened:

Dell PowerEdge R720
BIOS Version 1.3.6
Firmware Version 1.23.23 (Build 01)

Broadcom Gigabit Ethernet BCM5719
Family Firmware Version 7.2.20

ESXi 5.1 799733
Problem seems with Broadcom Gigabit network cards and the current firmware version.

Patch the Broadcom tg3 firmware driver with latest Patch ID:  tg3-3.129d.v50.1
Upgraded ESXi hosts to latest 5.1.0 build 1117900 using Update Manager with Baseline group create including Critical and non-critical host patches as well as the Broadcom tg3 baseline.
To work around this issue, disable the NetQueue feature.
If there are 10 Gb NICs on the host in addition to the tg3 NICs, then only disable NetQueue for the tg3 driver.