vCSA & PSC: Update/Patching options available

The update of either vCSA or PSC can be achieved through the VAMI interface which was introduced back in 6.0U1 or through the appliancesh command-line interface.
Method 1: VAMI and URL: 
This method requires internet access from your appliances.
  • Login to VAMI
  • https://vcenterserver.domain.com:5480/
    • U: root / P: password
  • From navigator select Update
  • This will display the current version details
  • Select Check Updates -> Check URL
  • This method will go out to VMware’s repository https://vapp-updates.vmware.com/vai-catalog/valm/… and verify you are on latest.
  • If available updates then select the Install updates -> Install all updates
  • Accept EULA
  • Wait for updates to complete.

Method 2: VAMI and custom URL: 
This method can be used if you do not have internet access from your appliances by setting up a local repository.
  • Setup a webserver on your network (ISS or Apache) that will be accessible to the vCSA and PSC.
  • Create a directory called PSC_updates or VCSA updates.  Names can be changed.
  • On VMware support site now download update but make sure to download the zip update bundle.
  • Extract the zip update bundle to the folder you created earlier.
  • Login to VAMI
  • https://vcenterserver.domain.com:5480
    • U: root / P: password
  • From navigator select Settings
  • Select “Use specified repository”
  • Click OK
  • Select Check for updates
  • If available updates then select the Install updates -> Install all update
  • Accept EULA
  • Wait for updates to complete
Method 3: VAMI and CDROM:

This method is pretty straight forward.

  • On VMware support site download the ISO for latest vCSA and/or PSC.
  • Login to vCenter Web client
  • Select vCSA or PSC appliance VM
  • Launch remote console
  • Select VMRC -> Removable devices -> CD/DVD drive 1 -> Connect to Disk Image File (iso)
  • Mount the ISO downloaded from VMware support site
  • Login to VAMI
  • https://vcenterserver.domain.com:5480/
    • U: root / P: password
  • From navigator select Update
  • Select Check Updates
  • Select Check CDROM
  • If available updates then select the Install updates -> Install all update
  • Accept EULA
  • Wait for updates to complete
Method 4: Using appliancesh command line

This method was the only way to update the appliance when vCenter 6 was released since the VAMI was only introduced in 6.0U1.  You can either mount and ISO or point to a URL for updates.  The URL can be the default VMware or 
  • On VMware support site download the ISO for latest vCSA and/or PSC.
  • Select vCSA or PSC appliance VM
  • Launch remote console
  • Select VMRC -> Removable devices -> CD/DVD drive 1 -> Connect to Disk Image File (iso)
  • Mount the ISO downloaded from VMware support site
  • SSH to VCSA or PSC
  • type # appliancesh
  • enter root password
  • To use CDROM
    • type # software-packages install –iso –acceptEulas
  • To use default VMware URL 
    • type # software-packages –url (This will use the default vmware URL to check)
  • to use local ropository URL

Product patches can be downloaded from this VMware site:
https://my.vmware.com/group/vmware/patch#search
Log files to review for updates:

/var/log/vmware/applmgmt/software-packages.log




vCenter Server SMTP authentication not supported – how to guide on getting alerts

I recently updated a customer from 5.1 to 6.0 and a couple of days later received a question on how to setup a mail server with SMTP authentication.

This of course is not possible as described in the following KB 2063147
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2063147

This got me interested to setup a solution that would provide the functionality to allow outgoing email through SMTP relay service in stead of setting up a full fledged local email server.
Here are the steps I took to setup postfix on CentOS to relay outgoing email to 3de party which requires authentication.

  • Install and configured a linux operating systems.
  • Now we need to install and update the packages required for our configuration which includes postfix as well as cyrus-sasl-plain which is not installed by default on CentOS 6+
    • sudo yum install postfix cyrus-sasl cyrus-sasl-plain
  • To make postfix the default MTA in our system lets remove sendmail
    • sudo yum remove sendmail
  • Postfix setup:
    • vi /etc/postfix/main.cf
    • Configure server FQDN:
      • mydomain =
      • myhostname =
    • Configure relayhost to email provide smtp server.  Verify the port since might not be default 25 to prevent spamming.
      • relayhost =
      • relaydomain =
    • Configure cyrus-sasl-plain:
      • smtpd_sasl_auth_enable = yes
      • smtpd_sasl_path = smtpd
      • stmpd_sasl_password_maps = hash:/etc/postfix/sasl_passwd
      • smtpd_sasl_type = cyrus
      • smtp_sasl_auth_enable = yes
    • Configure receive mail so that communication can be established from all networks.  If you select inet_interfaces = localhost then can only send from local server.
      • inet_interfaces = all
      • inet_protocols = all
    • Configure additional trust and relay control
      • mynetworks_style = subnet  (if you want to specify specific network subnets)
      • mynetworks_style = host  (if you want to specify specific host names)
      • mynetworks = 127.0.0.0/8, 192.168.1.0/24
  • Now since our SMTP server requires authentication we need to setup username and password.
    • vi /etc/postfix/sasl_passwd
      • yourisp.smtp.com:2525 username:password
      • the servername should match exactly what you have entered for relayhost in /etc/postfix/main.cf
  • Generate a postfix lookup table
    • postmap hash:/etc/postfix/sasl_passwd
  • Test lookup table which should return username and password
    • postmap -q yourisp.smtp.com:2525 /etc/postfix/sasl_passwd
  • Verify sasl_passwd and sasl_passwd.db files are read/write enabled for root only to protect the plain text password.
    • chmod 600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
  • Add postfix to be started at boot
    • chkconfig –add postfix
  • start service
    • /etc/init.d/postfix start
  • Send test email.
    • # sendmail -t
    • TO: addressto@test.com
    • From: addressfrom@test.com
    • Subject: Test
    • Did you get this email?
    • .
Troubleshooting:
If you check the status of service and get error:  “Master is Dead ButPid File Exists”, verify that you have removed sendmail successfully.
Connection refused when trying to send from vCenter, verify that port 25 is listening on host with # netstat -nlp | grep 25.  If it shows with 127.0.0.0/8 then it will only allow local connection.  This needs to show 0.0.0.0:25 so make sure you have inet_interfaces = all.
Some useful links:

ESXi 6: cannot synchronize host

Today had error pop up on vCenter server for vpxa service in yellow state, also found that ESXi host was showing errors for “Cannot synchronize host” as well as “quick stats on is not up-of-date”

Troubleshooting:

After doing some investigation found that one of the new hosts did not have a DNS entry created.

Fix:
This should be by now a given but always test forward and reverse DNS lookup for ESXi host before adding to vCenter server!

vCenter Server 6.0U2 errors – "lost access to volume"

Recently upgraded environment from 5.5 to vCenter Server 6.0U2.
Hardware consists of Cisco UCS with B200M3 blades and XtremIO storage.

After the upgrade users complained about slow and dropped connections to their VM’s.

Troubleshooting:

Installed a host with vCenter Server 6.0U1 and did not get the error message which was very strange, so what has changed between 6.0U1 and U2?
After reviewing the logs found that around every 30minute received errors “lost access to volumes”.
Further troubleshooting on logs revealed that this only happens on the XtremeIO datastores.

Also following warning message within vmkernel log file on ESXi host:

WARNING: NMP: nmp_PathDetermineFailure:2872: Cmd (0x85) PDL error (0x5/0x25/0x0) – path vmhba4:C0:T0:L10 device naa.514f0c514ba0000e – triggering path evaluation

Found the following KB from EMC and VMware which relates to this issue:

https://support.emc.com/kb/467750  (need login to view)

vSphere 6 added new VMCP feature with clear distinction between PDL and APD SCSI sense codes.
Good KB from VMware:
This issue relates to the XtremIO firmware (< 4.0.1) that provides a response (illegal request) to the vSphere 6.0 host SMART data request which triggers path evaluation for PDL condition.

Fix:

Upgrade XtremIO firmware to 4.0.1 and above. Latest recommended.
This issue could also affect other storage arrays so please make sure to check with VMware on this and keep the VMware KB as a live bookmark.
At end of day make sure to check the VMware compatibility guide.

vCenter Server 6.0U1 & 6.0U2 – ESXi hosts disappear with installation of Netapp VSC plugin 6.1 or 6.2

Recently performed a vCenter Server upgrade from 5.1 to 6.0U2.

The Netapp VSC vCenter plugin was also being used for backups using the plugin through the VIC.
The greenfield installation went without a hitch for vCenter Server 6 environment and hosts were migrated over.
After I upgraded the hosts to ESXi 6.0U2 I had to perform the upgrade of Netapp VSC plugin.
The plugin upgrade process was pretty straight forward with moving the plugin to a new server and pointing to new vCenter server.
Netapp has a pretty good KB out on how to preserve existing repository information on app migration to new server – https://kb.netapp.com/support/index?page=content&id=1011871
Problem:
After VSC plugin upgrade I ran into a strange issue where ESXi hosts kept randomly disappearing from the vCenter Web client and nothing else getting affected.
Troubleshooting:
Restarted vCenter server and tested with different browsers with no affect.
Disabled the plugin and restarted browser which resolved the issue.
Uninstalled 6.2 and installed 6.1 which seems to have better experience and hosts do no disappear as frequently.
Netapp community has a couple of blog posts related to the issue with no real positive response from Netapp.
Fix:

Open a case with Netapp and was provide BUG ID 986313 related to issue with no estimate on fix.
The bug can be view on Netapp support -> Tools -> Bug Tools.
Workaround:

Workaround at this time seems to be downgrading or install VSC 6.0 which has the least amount of problems. 

PSC : Firstboot script execution error

I installed a test PSC today and right at the end of the installation an error popped up “Firstboot script execution error”.

After looking through the log files I found the following:

VMware Appliance Configuration…\”, \n        \”translatable\”: \”Starting %(0)s…\”\n    }, \n    \”warning\”: [], \n    \”error\”: {\n        \”resolution\”: {\n            \”id\”: \”install.ciscommon.validatePNID.resolution\”, \n            \”localized\”: \”If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\”, \n            \”translatable\”: \”If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\”\n        }, \n        \”detail\”: [\n            {\n                \”args\”: [\n                    \”jpsctest01.sovsystems.com\”\n                ], \n                \”id\”: \”install.ciscommon.validatePNID.error\”, \n                \”localized\”: \”The supplied System Name jpsctest01.sovsystems.com is not valid.\”, \n                \”translatable\”: \”The supplied System Name %(0)s is not valid.\”\n            }\n        ], \n        \”componentKey\”: \”visl-integration\”, \n        \”problemId\”: \”install.ciscommon.validatePNID\”\n    }, \n    \”progress\”:0\n}”,”isFinal”:”true”}
2016-02-22 16:52:16.814728 Progress Controller: [VCSA ERROR] – First Boot error

Solution:

In my haste for testing i forgot to setup the A-records in DNS for the new PSC appliance.
The problem can also be related to providing the wrong DNS name during the installation wizard.

vSphere Web client 6.0 missing license UI

Found that on our recently upgrade vCSA 6.0U1 the license UI was missing.

Found a detailed KB article from VMware on this but they reference this happens when you have a proxy in place, which we don’t so seems this problem can potentially affect more environments.
Solution for vCSA:

  • Stop the vSphere Web Client service by running:
    service vsphere-client stop
  • Remove the contents of the vSphere Web Client work directory by running:
    rm -rf /usr/lib/vmware-vsphere-client/server/work/*
  • Remove the contents of the pickup directory by running:
    rm /usr/lib/vmware-virgo/server/pickup/*
  • Back up the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Remove the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Start the following vCenter service by running:
    service vsphere-client start
  •  
    Links:

    vCSA 6 postgreSQL – connect externally through pgAdmin

    I recently had to query the VCDB database of vCenter Server 6 but had an appliance deployed and since not using a MS SQL database server I had to do some digging to figure out how i can get access.

    My favorite tool to query the database I found to be pgAdmin III but this is installed on my jump server so here are the steps i following to allow pgAdmin to connect to the internal postgresql database on vCSA 6!
    http://www.pgadmin.org/

    SOLUTION:

    1. Enable SSH for vCSA.
    2. Login as root
      1. shell.set –enabled True
      2. shell
    3. View following 2 files for information on database installation
    4. /etc/vmware-vpx/embedded_db.cfg
      1. General server information and password for superuser. 
    5. /etc/vmware-vpx/vcdb.properties
      1. Stores connection information for vCenter server database VCDB (password for vc user
    6. Edit /storage/db/vpostgres/pg_hba.conf
      1. Add following lines to add your own subnets to be able to connect to PG
      2. Host                   all              all                  /           md5
    7. Edit /storage/db/vpostgres/postgresql.conf
      1. Add the line to end of file:   listen_addresses = ‘*’
      2. Restart posgresql /etc/init.d/vmware-vpostgres restart
    8. Open port on vcsa appliance firewall:
      1. iptables -A INPUT -p tcp -m tcp –dport 5432 -j ACCEPT
    This should do it and allow you to connect externally through the nice pgAdmin GUI to your database.

    vCenter Server – Host Profiles error "the option uservars.suppressShellWarning"

    Receive the error “the option uservars.suppressShellWarning”

    This is due to a previous selection to suppress warnings for instance for enable SSH on your ESXi host.

    Solution:

    To fix this you can either change the host profile to enable this suppression

    Advanced Configuration Settings -> Advanced Options -> Advanced configuration option
    Click the green plus “+” sign symbol to create a new option.
    Select advanced option “configured a fixed option”
    The name of the option “UserVars.SuppressShellWarning”
    Set the value to “1”

    If you don’t want to suppress the warnings anymore you can disable it through the advanced settings of the ESXi host using vSphere web client:

    Select ESXi host -> Manage tab -> Settings
    Select Advanced System settings
    easiest way is to search in filter for “suppress”
    This will list the UserVars.SuppressShellWarning.
    Change the settings to 0.

    vSphere 5.5 – Solution for VASA setup errors with VNX

    Over weekend I wanted to review some of the features and functionalities that VASA and VSI provides inside vCenter server for a VNX array(5300).  VASA configuration should be straight forward but ran into some strange issues which took some extensive troubleshooting:

    Troubleshooting:

    VMware compatibility guide for EMC VASA provider specifies that VASA support is part of VNX Block OE 05.32; no additional software is required.  Our VNX 5300 has FLARE 5.32 which should allow for straight connectivity to the VNX…
    http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vasa&productid=20232

    On vCenter server web client select your vCenter server in inventory list on left.  Select Manage tab in right pane -> Storage provider sub-tab. Click the green button.

    URL for direct VNX block:
    https:///vasa/services/vasaService
    URL for direct VNX file:
    https://:5989/vasa/services/vasaService
    URL for SMI-S:
    https://5989/vasa/services/vasaService

    When trying to connect directly to VNX block I received the following error message:

    In vCSA i reviewed the following error messages in sps.log under \var\logs\vmware\vpx\sps\
    “Received fatal alert: unknown_ca”

    The most common issue that seems to be related to the VASA configuration is a expired certificate for the storage monitoring server.
    This can be verified with the following commands on vCSA:
    # cd /etc/vmware-vpx/ssl/
    # keytool -keystore sms.keystore -storepass testpassword -list -v
    Check the valid from and until in the output provided.
    This was NOT my issue.

    To further test this I installed a windows 2012 server with SMI-S. Same connectivity errors appeared.
    So what is going on here!

    Some further troubleshooting was in order:
    – I regenerated the storage management certificates and restarted the vCenter server appliance
    – I restarted the array’s management server.
    – I added the vCenter server certificate to trusted certificates on VNX
    – Verified the user created on VNX has VM administrator role.
    – For user authentication if you create a local user on VNX, then be sure to add local\username for authentication in service provider.
    – Verify that there is an array connected to SMI-S.  this can be verified by running the command “symcfg list” or logging in with testsmiprovider.exe and executing “dv” command.

    Still no luck.
    Looking further at the sps.log file I figured the problem had to be with the vCenter server that is sending the certificate to the array.

    Solution:

    I sticked with SMI-S since it provides more customization in administration console.
    https://:5989/ECOMConfig/
        Username and password the same   (admin / #1Password)
    Under security select “SSL Certification Management.
    Here we want to select option 3 and import CA certificate from file…

    So which certificate do we import?  well that would be the sms.keystore cert, but we need the cert as PEM data.  Do this perform the following commands on vCSA:

    # cd /etc/vmware-vpx/ssl/
    # keytool -importkeystore -srckeystore sms.keystore -destkeystore /tmp/sms.pkcs -srcstoretype JKS -deststoretype PKCS12
    # cd /tmp
    # openssl pkcs12 -in sms.pkcs -out sms.pem

    Now open the sms.pem in and editor and copy the information from —Begin Certificate— to —End Certificate—
    Paste into the textbox to submit the CA to SMI-S.

    Now in vCenter service provider, setup connection to SMI-S again.

    Link:
    https://www.emc.com/collateral/software/white-papers/h10630-vmware-vasa-symmetrix-wp.pdf