vSphere Web client 6.0 missing license UI

Found that on our recently upgrade vCSA 6.0U1 the license UI was missing.

Found a detailed KB article from VMware on this but they reference this happens when you have a proxy in place, which we don’t so seems this problem can potentially affect more environments.
Solution for vCSA:

  • Stop the vSphere Web Client service by running:
    service vsphere-client stop
  • Remove the contents of the vSphere Web Client work directory by running:
    rm -rf /usr/lib/vmware-vsphere-client/server/work/*
  • Remove the contents of the pickup directory by running:
    rm /usr/lib/vmware-virgo/server/pickup/*
  • Back up the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Remove the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:
    • telemetry-service-6.0.0.jar
    • telemetry-ui-war-6.0.0.war
    • phonehome-collector-ui-war-6.0.0.war
    • cis-data-service-cmc-6.0.0.jar
  • Start the following vCenter service by running:
    service vsphere-client start
  •  
    Links:

    vCSA 6 postgreSQL – connect externally through pgAdmin

    I recently had to query the VCDB database of vCenter Server 6 but had an appliance deployed and since not using a MS SQL database server I had to do some digging to figure out how i can get access.

    My favorite tool to query the database I found to be pgAdmin III but this is installed on my jump server so here are the steps i following to allow pgAdmin to connect to the internal postgresql database on vCSA 6!
    http://www.pgadmin.org/

    SOLUTION:

    1. Enable SSH for vCSA.
    2. Login as root
      1. shell.set –enabled True
      2. shell
    3. View following 2 files for information on database installation
    4. /etc/vmware-vpx/embedded_db.cfg
      1. General server information and password for superuser. 
    5. /etc/vmware-vpx/vcdb.properties
      1. Stores connection information for vCenter server database VCDB (password for vc user
    6. Edit /storage/db/vpostgres/pg_hba.conf
      1. Add following lines to add your own subnets to be able to connect to PG
      2. Host                   all              all                  /           md5
    7. Edit /storage/db/vpostgres/postgresql.conf
      1. Add the line to end of file:   listen_addresses = ‘*’
      2. Restart posgresql /etc/init.d/vmware-vpostgres restart
    8. Open port on vcsa appliance firewall:
      1. iptables -A INPUT -p tcp -m tcp –dport 5432 -j ACCEPT
    This should do it and allow you to connect externally through the nice pgAdmin GUI to your database.

    vCenter Server – Host Profiles error "the option uservars.suppressShellWarning"

    Receive the error “the option uservars.suppressShellWarning”

    This is due to a previous selection to suppress warnings for instance for enable SSH on your ESXi host.

    Solution:

    To fix this you can either change the host profile to enable this suppression

    Advanced Configuration Settings -> Advanced Options -> Advanced configuration option
    Click the green plus “+” sign symbol to create a new option.
    Select advanced option “configured a fixed option”
    The name of the option “UserVars.SuppressShellWarning”
    Set the value to “1”

    If you don’t want to suppress the warnings anymore you can disable it through the advanced settings of the ESXi host using vSphere web client:

    Select ESXi host -> Manage tab -> Settings
    Select Advanced System settings
    easiest way is to search in filter for “suppress”
    This will list the UserVars.SuppressShellWarning.
    Change the settings to 0.

    vSphere 5.5 – Solution for VASA setup errors with VNX

    Over weekend I wanted to review some of the features and functionalities that VASA and VSI provides inside vCenter server for a VNX array(5300).  VASA configuration should be straight forward but ran into some strange issues which took some extensive troubleshooting:

    Troubleshooting:

    VMware compatibility guide for EMC VASA provider specifies that VASA support is part of VNX Block OE 05.32; no additional software is required.  Our VNX 5300 has FLARE 5.32 which should allow for straight connectivity to the VNX…
    http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vasa&productid=20232

    On vCenter server web client select your vCenter server in inventory list on left.  Select Manage tab in right pane -> Storage provider sub-tab. Click the green button.

    URL for direct VNX block:
    https:///vasa/services/vasaService
    URL for direct VNX file:
    https://:5989/vasa/services/vasaService
    URL for SMI-S:
    https://5989/vasa/services/vasaService

    When trying to connect directly to VNX block I received the following error message:

    In vCSA i reviewed the following error messages in sps.log under \var\logs\vmware\vpx\sps\
    “Received fatal alert: unknown_ca”

    The most common issue that seems to be related to the VASA configuration is a expired certificate for the storage monitoring server.
    This can be verified with the following commands on vCSA:
    # cd /etc/vmware-vpx/ssl/
    # keytool -keystore sms.keystore -storepass testpassword -list -v
    Check the valid from and until in the output provided.
    This was NOT my issue.

    To further test this I installed a windows 2012 server with SMI-S. Same connectivity errors appeared.
    So what is going on here!

    Some further troubleshooting was in order:
    – I regenerated the storage management certificates and restarted the vCenter server appliance
    – I restarted the array’s management server.
    – I added the vCenter server certificate to trusted certificates on VNX
    – Verified the user created on VNX has VM administrator role.
    – For user authentication if you create a local user on VNX, then be sure to add local\username for authentication in service provider.
    – Verify that there is an array connected to SMI-S.  this can be verified by running the command “symcfg list” or logging in with testsmiprovider.exe and executing “dv” command.

    Still no luck.
    Looking further at the sps.log file I figured the problem had to be with the vCenter server that is sending the certificate to the array.

    Solution:

    I sticked with SMI-S since it provides more customization in administration console.
    https://:5989/ECOMConfig/
        Username and password the same   (admin / #1Password)
    Under security select “SSL Certification Management.
    Here we want to select option 3 and import CA certificate from file…

    So which certificate do we import?  well that would be the sms.keystore cert, but we need the cert as PEM data.  Do this perform the following commands on vCSA:

    # cd /etc/vmware-vpx/ssl/
    # keytool -importkeystore -srckeystore sms.keystore -destkeystore /tmp/sms.pkcs -srcstoretype JKS -deststoretype PKCS12
    # cd /tmp
    # openssl pkcs12 -in sms.pkcs -out sms.pem

    Now open the sms.pem in and editor and copy the information from —Begin Certificate— to —End Certificate—
    Paste into the textbox to submit the CA to SMI-S.

    Now in vCenter service provider, setup connection to SMI-S again.

    Link:
    https://www.emc.com/collateral/software/white-papers/h10630-vmware-vasa-symmetrix-wp.pdf

    vCenter Infrastructure navigator – re-register vCenter server extension

    Ran into some issue with our vCenter server recently which caused the refresh icon to keep spinning (updating) and would slow down performance and also could not perform any searches.

    VMware support could not figure out the problem but eventioally provided information to investigate our plugins.

    Troubleshooting:

    Firstly we disable all the plugins and restarted vCenter server appliance services.
    This caused the web client to crash whenever we selected the plugins!
    Only way I found to fix this is:
    –  Create copy of file /etc/vmware-vsphere-client/SerenityDB/serenity/ALL/client.package.states
    –  Delete the original file
    –  Close all browsers and login again.
    –  All plugins should now again show enabled.
    ( Disclaimer:  I recommend contacting VMware support since this is not supported by VMware so please take snapshot&backup of VM and be careful)

    Logged into vCenter Server MOB and unregistered all non-system extension.  This did not seem to fix the problem either but we found that Orchestrator connection was never fixed after we regenerated the certificate for vCenter server.
    Not sure this caused our problems or the reset and deletion of plugins but soon after application was working as expected.

    After un-registering plugins we have to of course register vCenter Infrastructure manager again:

    Solution:

    Log into the vSphere client pointing to the vCenter with the issue.
    In top menu select Administration > vCenter Server Settings > Runtime Settings.
    Verify the IP is correct for the vCenter you are wanting to monitor.

    Shut down Navigator.
    – These steps can only be performed from the .net vSphere client.
    – Power down the virtual machine, right click the virtual machine from within the vSphere Desktop Client, select Edit Settings.
    – Choose the vServices tab, click Edit. Under Provider select .
    – Click OK and then OK again.
    – Do not re-add it yet.

    Log into the MOB for vCenter – https://Vc IP/MOB :
    – Open a web browser and navigate to:
    https://virtual_center_address/mob
    – Click Content.
    – Click ExtensionManager.
    – Verify com.vmware.vadm is listed as an Extension.
    – If listed, copy this string and click UnregisterExtension.
    – Paste string into Value box and click Invoke Method.

    If vCenter server on Microsoft OS, log in to the vCenter servers and find a similar path to:
    C:\Program Files\VMware\Infrastructure\VirtualCenterServer\extensions\com.vmware.vadm
    – Delete the entire com.vmware.vadm

    Add the provider back in:
    – Go back to vSphere
    –  Right click the virtual machine from within the vSphere Desktop Client, select Edit Settings.
    – Choose the vServices tab, click Edit. Under Provider select the one below No Provider.
    – Click OK and then OK again.
    – Turn the system back on and verify connection.

    If not coming up then restart the vSphere web client service

    vCenter Server – lookup service failed

    Recently did a rack migration for which we had to shutdown SQL cluster and vCenter server including all Virtual machines.

    After powering I was unable to login with my domain account and got the following error:

    “Failed to connect to VMware Lookup Service. SSL certificate verification failed”

    Troubleshooting:

    • Login to vCenter server (SSH or RDP depending on vCenter server platform) and verified that the vmware-sso service is running.
    • Could not login with local admin@vsphere.local so logged into VA configuration https://:5480.  Verified all services was running.
    • Verified SSO server and Active directory domain controllers time is not off by more than 5 minutes.
    • Found KB article on this error.

    Solution:

    Found KB that provides information on this issue which seems to get triggered when you had change the IP or hostname of the vCenter server after its initial creation.

    To fix the issue login to VA configuration: https://:5480.
    Select admin tab
    Check the “yes” radio button for “Certificate regeneration enabled:”
    Click Submit.
    Restart vCenter server

    Link:

    http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2058430&sliceId=1&docTypeID=DT_KB_1_1&dialogID=638751240&stateId=0%200%20638753748

    Unable to power on VM with SCSI bus sharing enabled – error: "Could not open/create change tracking file"

    So recently i was asked to assist with setting up a new SQL 2012 AlwaysOn Cluster for POC.
    During the configuration of the VM’s we had to setup a quorom disk which is shared between both servers for MSCS.
    The shared disk I configured and select virtual device node on newly created SCSI controller, LSI Logic SAS with SCSI bus sharing set to Physical.
    On the 2nd VM i created the same SCSI controller as above and instead of creating new disk the existing disk was select.

    With Primary powered on and trying to power on the 2nd SQL server i receive the following error:
    “Could not open/create change tracking file”

    Troubleshooting:

    The error eludes to problem with CBT on the Virtual server where i cannot access the cbt file due to the primary VM already running and locked the file.

    I could not find much information in documentation regarding setting up clusters or shared vmdk disk and making use of CBT.

    It does however seem that CBT has some restrictions:

    1. The virtual machine version earlier than 7
    2. The virtual machine has a Raw Device Mapping (RDM) in the physical compatibility mode.
    3. The virtual machine has a disk attached to a shared virtual SCSI bus.  Use of CBT only possible when SCSI bus sharing is set to none.

    Following disk formats supported:
    •Thin and Thick virtual disks
    •VMDK and RDM (virtual only)
    Solution:

    Disable CBT for VM: (this can be set for all devices or just specific devices)
    • Edit settings of virtual machine
    • Select options tab
    • Click General under advanced -> configuration parameters
    • Click Add row
    • Add “ctkEnabled and set value “false”  (this will set for all devices)
    • OR
    • Add “scsi0:0.ctkEnabled and set value “false” (this will set for specific devices)

     To prevent any third-party applications from enabling Change Tracking on the virtual machine:
    • SSH to ESXi host
    • browse the datastore where VM resides
    • Open the .vmx file of the virtual machine using a text editor. 
    • Add this entry to the file:
      • ctkDisallowed=”true”