vCenter Server 6.0U1 & 6.0U2 – ESXi hosts disappear with installation of Netapp VSC plugin 6.1 or 6.2

Recently performed a vCenter Server upgrade from 5.1 to 6.0U2.

The Netapp VSC vCenter plugin was also being used for backups using the plugin through the VIC.
The greenfield installation went without a hitch for vCenter Server 6 environment and hosts were migrated over.
After I upgraded the hosts to ESXi 6.0U2 I had to perform the upgrade of Netapp VSC plugin.
The plugin upgrade process was pretty straight forward with moving the plugin to a new server and pointing to new vCenter server.
Netapp has a pretty good KB out on how to preserve existing repository information on app migration to new server – https://kb.netapp.com/support/index?page=content&id=1011871
Problem:
After VSC plugin upgrade I ran into a strange issue where ESXi hosts kept randomly disappearing from the vCenter Web client and nothing else getting affected.
Troubleshooting:
Restarted vCenter server and tested with different browsers with no affect.
Disabled the plugin and restarted browser which resolved the issue.
Uninstalled 6.2 and installed 6.1 which seems to have better experience and hosts do no disappear as frequently.
Netapp community has a couple of blog posts related to the issue with no real positive response from Netapp.
Fix:

Open a case with Netapp and was provide BUG ID 986313 related to issue with no estimate on fix.
The bug can be view on Netapp support -> Tools -> Bug Tools.
Workaround:

Workaround at this time seems to be downgrading or install VSC 6.0 which has the least amount of problems. 

PSC : Firstboot script execution error

I installed a test PSC today and right at the end of the installation an error popped up “Firstboot script execution error”.

After looking through the log files I found the following:

VMware Appliance Configuration…\”, \n        \”translatable\”: \”Starting %(0)s…\”\n    }, \n    \”warning\”: [], \n    \”error\”: {\n        \”resolution\”: {\n            \”id\”: \”install.ciscommon.validatePNID.resolution\”, \n            \”localized\”: \”If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\”, \n            \”translatable\”: \”If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\”\n        }, \n        \”detail\”: [\n            {\n                \”args\”: [\n                    \”jpsctest01.sovsystems.com\”\n                ], \n                \”id\”: \”install.ciscommon.validatePNID.error\”, \n                \”localized\”: \”The supplied System Name jpsctest01.sovsystems.com is not valid.\”, \n                \”translatable\”: \”The supplied System Name %(0)s is not valid.\”\n            }\n        ], \n        \”componentKey\”: \”visl-integration\”, \n        \”problemId\”: \”install.ciscommon.validatePNID\”\n    }, \n    \”progress\”:0\n}”,”isFinal”:”true”}
2016-02-22 16:52:16.814728 Progress Controller: [VCSA ERROR] – First Boot error

Solution:

In my haste for testing i forgot to setup the A-records in DNS for the new PSC appliance.
The problem can also be related to providing the wrong DNS name during the installation wizard.

vSphere Web client 6.0 missing license UI

Found that on our recently upgrade vCSA 6.0U1 the license UI was missing.

Found a detailed KB article from VMware on this but they reference this happens when you have a proxy in place, which we don’t so seems this problem can potentially affect more environments.
Solution for vCSA:

  • Stop the vSphere Web Client service by running:
    service vsphere-client stop
  • Remove the contents of the vSphere Web Client work directory by running:
    rm -rf /usr/lib/vmware-vsphere-client/server/work/*
  • Remove the contents of the pickup directory by running:
    rm /usr/lib/vmware-virgo/server/pickup/*
  • Back up the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Remove the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Start the following vCenter service by running:
    service vsphere-client start
  •  
    Links:

    vCSA 6 postgreSQL – connect externally through pgAdmin

    I recently had to query the VCDB database of vCenter Server 6 but had an appliance deployed and since not using a MS SQL database server I had to do some digging to figure out how i can get access.

    My favorite tool to query the database I found to be pgAdmin III but this is installed on my jump server so here are the steps i following to allow pgAdmin to connect to the internal postgresql database on vCSA 6!
    http://www.pgadmin.org/

    SOLUTION:

    1. Enable SSH for vCSA.
    2. Login as root
      1. shell.set –enabled True
      2. shell
    3. View following 2 files for information on database installation
    4. /etc/vmware-vpx/embedded_db.cfg
      1. General server information and password for superuser. 
    5. /etc/vmware-vpx/vcdb.properties
      1. Stores connection information for vCenter server database VCDB (password for vc user
    6. Edit /storage/db/vpostgres/pg_hba.conf
      1. Add following lines to add your own subnets to be able to connect to PG
      2. Host                   all              all                  /           md5
    7. Edit /storage/db/vpostgres/postgresql.conf
      1. Add the line to end of file:   listen_addresses = ‘*’
      2. Restart posgresql /etc/init.d/vmware-vpostgres restart
    8. Open port on vcsa appliance firewall:
      1. iptables -A INPUT -p tcp -m tcp –dport 5432 -j ACCEPT
    This should do it and allow you to connect externally through the nice pgAdmin GUI to your database.

    vCenter Server – Host Profiles error "the option uservars.suppressShellWarning"

    Receive the error “the option uservars.suppressShellWarning”

    This is due to a previous selection to suppress warnings for instance for enable SSH on your ESXi host.

    Solution:

    To fix this you can either change the host profile to enable this suppression

    Advanced Configuration Settings -> Advanced Options -> Advanced configuration option
    Click the green plus “+” sign symbol to create a new option.
    Select advanced option “configured a fixed option”
    The name of the option “UserVars.SuppressShellWarning”
    Set the value to “1”

    If you don’t want to suppress the warnings anymore you can disable it through the advanced settings of the ESXi host using vSphere web client:

    Select ESXi host -> Manage tab -> Settings
    Select Advanced System settings
    easiest way is to search in filter for “suppress”
    This will list the UserVars.SuppressShellWarning.
    Change the settings to 0.

    vSphere 5.5 – Solution for VASA setup errors with VNX

    Over weekend I wanted to review some of the features and functionalities that VASA and VSI provides inside vCenter server for a VNX array(5300).  VASA configuration should be straight forward but ran into some strange issues which took some extensive troubleshooting:

    Troubleshooting:

    VMware compatibility guide for EMC VASA provider specifies that VASA support is part of VNX Block OE 05.32; no additional software is required.  Our VNX 5300 has FLARE 5.32 which should allow for straight connectivity to the VNX…
    http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vasa&productid=20232

    On vCenter server web client select your vCenter server in inventory list on left.  Select Manage tab in right pane -> Storage provider sub-tab. Click the green button.

    URL for direct VNX block:
    https:///vasa/services/vasaService
    URL for direct VNX file:
    https://:5989/vasa/services/vasaService
    URL for SMI-S:
    https://5989/vasa/services/vasaService

    When trying to connect directly to VNX block I received the following error message:

    In vCSA i reviewed the following error messages in sps.log under \var\logs\vmware\vpx\sps\
    “Received fatal alert: unknown_ca”

    The most common issue that seems to be related to the VASA configuration is a expired certificate for the storage monitoring server.
    This can be verified with the following commands on vCSA:
    # cd /etc/vmware-vpx/ssl/
    # keytool -keystore sms.keystore -storepass testpassword -list -v
    Check the valid from and until in the output provided.
    This was NOT my issue.

    To further test this I installed a windows 2012 server with SMI-S. Same connectivity errors appeared.
    So what is going on here!

    Some further troubleshooting was in order:
    – I regenerated the storage management certificates and restarted the vCenter server appliance
    – I restarted the array’s management server.
    – I added the vCenter server certificate to trusted certificates on VNX
    – Verified the user created on VNX has VM administrator role.
    – For user authentication if you create a local user on VNX, then be sure to add local\username for authentication in service provider.
    – Verify that there is an array connected to SMI-S.  this can be verified by running the command “symcfg list” or logging in with testsmiprovider.exe and executing “dv” command.

    Still no luck.
    Looking further at the sps.log file I figured the problem had to be with the vCenter server that is sending the certificate to the array.

    Solution:

    I sticked with SMI-S since it provides more customization in administration console.
    https://:5989/ECOMConfig/
        Username and password the same   (admin / #1Password)
    Under security select “SSL Certification Management.
    Here we want to select option 3 and import CA certificate from file…

    So which certificate do we import?  well that would be the sms.keystore cert, but we need the cert as PEM data.  Do this perform the following commands on vCSA:

    # cd /etc/vmware-vpx/ssl/
    # keytool -importkeystore -srckeystore sms.keystore -destkeystore /tmp/sms.pkcs -srcstoretype JKS -deststoretype PKCS12
    # cd /tmp
    # openssl pkcs12 -in sms.pkcs -out sms.pem

    Now open the sms.pem in and editor and copy the information from —Begin Certificate— to —End Certificate—
    Paste into the textbox to submit the CA to SMI-S.

    Now in vCenter service provider, setup connection to SMI-S again.

    Link:
    https://www.emc.com/collateral/software/white-papers/h10630-vmware-vasa-symmetrix-wp.pdf

    vCenter Infrastructure navigator – re-register vCenter server extension

    Ran into some issue with our vCenter server recently which caused the refresh icon to keep spinning (updating) and would slow down performance and also could not perform any searches.

    VMware support could not figure out the problem but eventioally provided information to investigate our plugins.

    Troubleshooting:

    Firstly we disable all the plugins and restarted vCenter server appliance services.
    This caused the web client to crash whenever we selected the plugins!
    Only way I found to fix this is:
    –  Create copy of file /etc/vmware-vsphere-client/SerenityDB/serenity/ALL/client.package.states
    –  Delete the original file
    –  Close all browsers and login again.
    –  All plugins should now again show enabled.
    ( Disclaimer:  I recommend contacting VMware support since this is not supported by VMware so please take snapshot&backup of VM and be careful)

    Logged into vCenter Server MOB and unregistered all non-system extension.  This did not seem to fix the problem either but we found that Orchestrator connection was never fixed after we regenerated the certificate for vCenter server.
    Not sure this caused our problems or the reset and deletion of plugins but soon after application was working as expected.

    After un-registering plugins we have to of course register vCenter Infrastructure manager again:

    Solution:

    Log into the vSphere client pointing to the vCenter with the issue.
    In top menu select Administration > vCenter Server Settings > Runtime Settings.
    Verify the IP is correct for the vCenter you are wanting to monitor.

    Shut down Navigator.
    – These steps can only be performed from the .net vSphere client.
    – Power down the virtual machine, right click the virtual machine from within the vSphere Desktop Client, select Edit Settings.
    – Choose the vServices tab, click Edit. Under Provider select .
    – Click OK and then OK again.
    – Do not re-add it yet.

    Log into the MOB for vCenter – https://Vc IP/MOB :
    – Open a web browser and navigate to:
    https://virtual_center_address/mob
    – Click Content.
    – Click ExtensionManager.
    – Verify com.vmware.vadm is listed as an Extension.
    – If listed, copy this string and click UnregisterExtension.
    – Paste string into Value box and click Invoke Method.

    If vCenter server on Microsoft OS, log in to the vCenter servers and find a similar path to:
    C:\Program Files\VMware\Infrastructure\VirtualCenterServer\extensions\com.vmware.vadm
    – Delete the entire com.vmware.vadm

    Add the provider back in:
    – Go back to vSphere
    –  Right click the virtual machine from within the vSphere Desktop Client, select Edit Settings.
    – Choose the vServices tab, click Edit. Under Provider select the one below No Provider.
    – Click OK and then OK again.
    – Turn the system back on and verify connection.

    If not coming up then restart the vSphere web client service