vCenter Server Web Client crash with latest Flash 27.0.0.170

Within the last couple of days Google ran an automatic update of Flash, which is causing havoc in the all-things-flash world.

Since Google manages the upgrade of the flash you are kind of at their mercy on this one, until all applications finally learn to stay as far away from flash as possible! Can only hope!

You can however disable the automatic update feature for flash by setting the registry HKLM\SOFTWARE\Policies\Google\Update\AutoUpdateCheckPeriodMinutes to 0.

My fix was pretty straight forward and I only had to delete the latest release folder from the path “%LocalAppData%\Google\Chrome\User Data\PepperFlash” (windows) or “~/Library/Application Support/Google/Chrome/PepperFlash” (MAC).

In my case the older version was still available in the same folder so I only had to delete the latest and restart google but if do not have the older version you can download it here.

From other users responses it seems that the version 27.0.0.130 and older still works but newer version will crash on both your Google Chrome and Firefox browsers.

Update:  VMware has release a KB here

 

 

Migration of vCenter Server using VMware Migration Assistant

The migration assistant can tackle a couple of different vCenter Server configurations:

  • vCenter Server 5.5 or 6.0 with an embedded vCenter Single Sign-On instance on Windows
  • vCenter Server 5.5 instance on Windows with external SSO
  • vCenter server 6.0 instance on Windows with external PSC

Depending on your current configuration, the migration process will change based using an internal or external SSO/PSC and where VUM is installed.

  • If VUM is installed on a separate Windows server, other that your Windows vCenter Server instance then you MUST to run the migration assistant on this server FIRST!  The VMware Migration Assistant facilitates the migration of the Update Manager server and database to the vCenter Server Appliance 6.5.
  • If embedded SSO/PSC, then you run the Migration Assistant on the source vCenter Server
  • If external SSO/PSC, then you run the Migration Assistant on the source SSO/PSC first and then run the Migration Assistant on the source vCenter Server
    • During the migration process, make sure to leave the migration windows open

You run the VMware Migration Assistant on your source vCenter Server, SSO/PSC or VUM it will perform the following tasks:

  • Discover the source deployment type.
  • Run pre-checks on the source.
  • Report errors that must be addressed before starting the migration.
  • Provide information for the next steps in the migration process.

Here are my step by step instructions for vCenter Server 5.5 with embedded SSO, no VUM:

  1. Download and Mount the vCenter Server Appliance Installer ISO file to the Windows vCenter Server.
  2. Start the migration assistant on the source, depending on your environment configuration
    1. Open the installer folder on CD/DVD drive and copy the  “migration-assistant” folder to your source windows server running the vCenter Server
    2. Open the copied folder on local drive and run “VMware-migration-assistant.exe”
      1. This will run the pre-update checklist and prompt if anything needs to be fixed.
      2. Make sure this window stays open during the whole migration process until completion.
  3. Launch vCenter server appliance UI installer
    1. Click Migrate
    2. Click Next on Introduction
    3. Accept EULA and click Next
    4. Connect to the source server
      1. Enter Windows server FQDN or IP address
      2. Enter migration assistant port number
        • If you are changing networks and have a firewall in place then pick a port that is open and available.
      3. SSO username
        • administrator@vsphere.local
      4. SSO Password
      5. Click Next
    5. Accept Thumbprint
    6. Appliance deployment target
      1. Enter ESXi or vCenter Server name for targer
      2. HTTPS port
      3. Username
      4. Password
      5. Click Next
    7. Accept Thumbprint
    8. Select Folder to place the new vCSA
    9. Select the compute resource
    10. Target appliance VM
      1. Enter VM name
      2. Enter new root password
    11. Select deployment size
    12. Select datastore
    13. Configure the temporary network identity which is used to copy the data, this will be removed after migration is completed since the target vCSA server will get the network identity of the source vCenter server.
      1. Pick network port group
      2. Enter temp IP address
      3. Enter subnet
      4. Enter GW
      5. Enter DNS servers
      6. Click Next
    14. Verify detailed entered are correct!
    15. Click Finish
    16. Wait for the migration to complete.
      • If the migration fails for any reason your can easily roll back by powering off the target vCSA and then just power on the source vCenter Server and its components.
  4. Now for stage two, which copies the data from the source vCenter to new vCSCA.
    1. On Introduction click next
    2. Connect to Source vCenter should complete automatically
    3. Join AD
      • Add username and password for user that can add computer to domain
    4. Select the best migration data options with the following 3 being available. (pretty cool that it provides the size of the migration for each option as well)
      • Just the configuration
      • Configuration, Events and Tasks
      • Configuration, Events, Tasks and Performance metrics
    5. Select if you want to join CEIP and click Next
    6. Review your settings, check the box to verify you have a backup of the vCenter server as well as the database.  Click Next
    7. Click OK on prompt that warns you that the source vCenter Server will be shut down once the network configuration is enabled.
    8. Wait for the data migration to complete.

When successful, your source vCenter Server should now be shut down and brand new shiny vCenter Server appliance should be started in its place. The best part of all you are also upgraded to vCenter Server 6.5! awesome.

 

 

vCenter Server 6.5 error: Cannot start content-library service

I recently discovered a problem with my vCenter Server after an upgrade from 6.0 to 6.5 of the vCenter Server Appliance, which runs in linked mode.

When I tried to take a backup of the vCenter Server through the VAMI I got an error “Invalid vCenter Server Status: All required services are not up! Stopped services: ‘content-library’.

Troubleshooting:

My first attempt was trying to start the service

  • I logged into the vCenter Server web client and went to administration -> Deployment -> System Configuration.
  • Double click on the the vCSA node
  • Select services
  • Select Content Library service, which under summary showed stopped.
  • Click in Actions -> Starts
  • Service failed to start with very cryptic error “An error occurred while starting service ‘%(0)s”

I also ssh into the vCSA and tried to start the service, and see the following error when trying to start the service:

Error executing start on service content-library. Details {
“resolution”: null,
“detail”: [
{
“args”: [
“content-library”
],
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘content-library'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
}
],
“componentKey”: null,
“problemId”: null
}
Service-control failed. Error {
“resolution”: null,
“detail”: [
{
“args”: [
“content-library”
],
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘content-library'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
}
],
“componentKey”: null,
“problemId”: null
}

Since the error message did not help much I went to search on VMware community and found other users having issues with creating content libraries after upgrade and VMware support pointed them to a KB article.

The fix:

The solution comes from KB 2151085 , however I am on version 6.5 not 6.5U1, as well as the OVF stuff was not applicable to my particular situation but the steps provided did resolve my problem and I was able to start the content-libary services.

I compared the files with a vCenter 6.5 environment that is working and found that the ts-config.properties file had the owner and group set to root:root and should be set to content-library:cis.  This is an easy fix and also solved the problem for me.   “chown content-library:cis ts-config.properties”

 

 

 

VMware did not disappoint! Finally NSX supports vSphere 6.5 and then some…

Yesterday VMware released both NSX for vSphere 6.3.o as well as the accompanying  vCenter 6.5a and ESXi 6.5a.

Before upgrading your NSX environment to 6.3 you would need to upgrade your vCenter and ESXi hosts to 6.5a as describe in the KB 2148841

It seems there was a pretty good reasons for VMware holding out on NSX so long since they introduced a lot of great new features with just a fews standouts listed below:

  • NSX kernel modules now independent of ESXi version: This enhancement helps reduce the chance of host upgrades failing due to incorrect kernel module versions since every ESXi upgrade in an NSX environment required at least 2 reboots.
  • Rebootless upgrade and uninstall on hosts: On vSphere 6.0 and later, once you have upgraded to NSX 6.3.0, any subsequent NSX VIB changes will not require a reboot. Instead hosts must enter maintenance mode to complete the VIB change.
  • NSX 6.3.0 also checks for NSX readiness before taking a host out of maintenance mode
  • Controller Disconnected Operation (CDO) mode: A new feature called Controller Disconnected Operation (CDO) mode has been introduced. This mode ensures that data plane connectivity is unaffected when hosts lose connectivity with the controller.
  • Compliance features for FIPS, Common Criteria and ICSA.
  • Improved Layer 2 VPN performance: Performance for Layer 2 VPN has been improved. This allows a single Edge appliance to support up to 1.5 Gb/s throughput, which is an improvement from the previous 750 Mb/s.

  • Linux support for Guest Introspection
  • Better interoperability between vCloud Director 8.20 and NSX 6.3.0 helps service providers offer advanced networking and security services to their tenants. vCD 8.20 with NSX 6.3.0 exposes native NSX capabilities supporting multiple tenants and tenant self-service.  (Very interested to test this and hopefully have a write up in upcoming weeks)

 

Links:

http://pubs.vmware.com/Release_Notes/en/nsx/6.3.0/releasenotes_nsx_vsphere_630.html

https://kb.vmware.com/kb/2148841

http://pubs.vmware.com/Release_Notes/en/vsphere/65/vsphere-vcenter-server-650a-release-notes.html


 

 

vCenter Server 6.5 announcement and detailed new features list

VMware finally pulled the curtains on their new vSphere 6.5 products during the European VMworld 2016 in Barcelona.  No release dates were announced but there are a lot of good stuff here.

  • vCenter 6.5
  • SRM 6.5
  • vRops 6.3
  • vRA 7.2
  • vSAN 6.5
  • VVOLS 2.0

I was fortunate enough to be part of the vCenter 6.5 beta and was impressed with the new features and VMware’s renewed focus on their core application stack.  I also have a couple of JFJ (JumpForJoy) moments which I listed below.

  • Auto-deploy finally got a UI and is now available for configuration in the vSphere Web client.
    • Creation of image profiles
    • Creation and activation of deploy rules
    • Management of deploy rules with ability to check compliance and remediation.
    • Ability to manually match non-deployed ESXi hosts to rules.
  • Enhancements to host profiles
    • Ability to search for a specific setting name, property name, or value by filtering the host profile tree while editing the host profile.
    • Copying setting from one host profile to another profile
    • Mark host profile settings as favorite and filter based on favorites.
  • Current Web client UI and usability improvements
    • Performance improvements
    • Keyboard shortcuts
    • Keyboard support in dialogs, Wizards and Confirmations
    • Recent objects global pane
    • Related objects tab replaced with object category tabs
    • Object details title bar displays the selected object’s icon and name, action icons, and the Actions menu
  • Live refresh, yes live!  JFJ moment!  This feature is awesome and not sure why it it took this long to make this available especially since we now have to use HTML5.   The real time updates are also done across users who are logged into vSphere client at the same time.
    • Live Tasks, Trigger alarms and reset alarms.
    • Navigation tree updates
    • Custom attributes
  • Oh yes and then there is the HTML5 web client.
    • HTML5 (<vcenter>/ui) and vSphere web client (<vcenter>/vsphere-client) are both available.
    • HTML5 web client does not yet have feature parity with vSphere web client and hopefully this will happen soon, but I recommend using the HTML5 as much as possible.
  • New and updated HA features.  JFJ moments all over the place!
    • Enhancements in the way calculations and configuration is done to determine failover capacity within a cluster.
    • Cluster Resource Percentage will be the default admission control moving forward.  The default failover capacity percentage will automatically be re-calculated based on the number of hosts in the cluster
    • Admission control – “VM Resource Reduction Event Threshold” setting
      • In past versions if a cluster did not have enough failover capacity during a hardware failure event, a number of VM’s would not be allowed to restart onto other healthy hosts. This new settings is a new feature that allows admins to specify the amount of resource reduction they are willing to tolerate in the cluster, potentially allowing additional VMs to be restarted even though capacity is not present, in exchange for potential performance degradation of VMs.
      • Setting this value to 0% means that you will not allow any resource reduction of any VM resources in your environment in the event of hardware failure.
    • Configuring orchestrator restarts!
      • Allow admins to specify the order of VM restarts as well as VM dependencies (critical applications, multi-tiered applications, and infrastructure services) at the cluster level.  We finally have similar orchestrated failover capabilities as SRM except SRM allows for injections of scripts which is not availabe with HA.
        • VM restart priority now includes: Lowest, Low, Medium, High, Highest
        • VM dependency restart conditions:
          • Resource allocated – Once resources for a VM are set aside on the host, HA will move to the next VM.
          • Powered On – Occurs when the power-on command is sent to the VM. Does not wait for the VM’s guest OS to be running.
          • Guest Heartbeats detected – Requires VMware Tools. Once vSphere sees that the VMware Tools agent is running, it will proceed.
          • App Heartbeats detected –  Requires scripting with the VMware Tools SDK, however this setting allows for information of a process/application within the VM’s guest OS to be passed shared to notify when an application is up and running in the VM.
  • Enhancements in event logging
    • Improve over 30 existing events for more detailed auditing.
    • Over 20 new events for different inventory operations.
    • Syslog / RELP streams
  • Storage IO Control (SIOC) with Storage policy-based management
    • SIOC was previously enabled per datastore and VM thresholds was set within the VM settings by first configuring the disk share value and then setting the IOPS limit value for each disk .   This was cumbersome to manage.
    • SIOC is now management and configured by using SPBM.
    • For storage policies there are now new rules available for readOPS, writeOPS, readLatency, writeLatency.
  • vCenter Server appliance Backup and Restore capability
    • File based backups/restore of vCenter server appliance through the Appliance Management UI.
    • Backup to a single folder all vCenter server core configuration, inventory and historical data.
    • Backup protocols available are FTP, SFTP, FTPS, HTTP, HTTPS
    • Encryption available for backup data before it is transferred.
    • Optional vCenter data available for backup:  Stats, Events, Alarms, Tasks.
    • To restore you have to use the vCenter installer which will deploy a new vCenter server appliance and restore the backup.  You cannot restore to your existing vCenter server. Make sure your existing vCenter server appliance is powered down before running a restore.
  • Command line deployment of vCenter server appliance
    • Scripted install
    • Installation using JSON formatted template and vcsa-cli-installer
  • vCSA and PSC failover. JFJ moment!
    • I will probably create a separate blog on this topic.
    • Native option to protect a vCenter server deployment from failures in hardware, vCenter and PSC service failures.
  • New Appliance management UI
    • Shows basic health with health badges.
    • CPU and Memory graphs showing utilization trends
    • Backup appliance
    • Create support bundle
    • Perform power operations such as rebooting and shutting down the appliance
  • Migration from a Windows vCenter server 5.5 to vCenter Appliance 6.0U
  • Security enhancements
    • VM level disk encryption.
    • Encrypted vMotion capabilities
    • Secure boot model

 

Please share your thoughts if you feel I am missing any other important features.

Links:

https://blogs.vmware.com/vsphere/2016/10/introducing-vsphere-6-5.html

http://blogs.vmware.com/vsphere/2016/10/whats-new-in-vsphere-6-5-vcenter-server.html

http://blogs.vmware.com/vsphere/2016/10/whats-new-in-vsphere-6-5-security.html

http://blogs.vmware.com/vsphere/2016/10/whats-new-in-vsphere-6-5-host-resource-management-and-operations.html

WinSCP connection to VCSA failed: "Received too large SFTP packet. Max supported packet size is 1024000 B"

The following error might appear when you try to connect with WinSCP to your VCSA.

This is due to login scripts that are printing words and the first 4 characters cast into the number(represents the first 4 bytes read from the server)

To fix the problem you can usually move the command that print the login script text to another proper interactive script or just remove completed, however in VMware the scenario is different and the default shell has change from bash to appliancesh.

VMware’s resolution is to use the SCP file protocol through bash shell.  However after I change to SCP I received the following error: (when default shell not set to bash)


This was fixed after changing the default shell.  I am using a newly created user account that can be used to access the server through WinSCP.  Just remember you would have to modify permissions on your files to copy them if going down this route.  You can use root account to temporary change the shell from bash to appliance to access with WinSCP. Entirely up to you.

>shell.set –enabled True
>shell
>useradd winscp
>passwd winscp
>visudo (add user with root access)
>chsh -s /bin/bash winscp

If you are using root you temporary change to bash shell and then return to appliance shell:
To return:
>chsh -s /bin/appliancesh useraccount

Links:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2107727

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2115983

https://winscp.net/eng/docs/message_large_packet

vCSA & PSC: Update/Patching options available

The update of either vCSA or PSC can be achieved through the VAMI interface which was introduced back in 6.0U1 or through the appliancesh command-line interface.
Method 1: VAMI and URL: 
This method requires internet access from your appliances.
  • Login to VAMI
  • https://vcenterserver.domain.com:5480/
    • U: root / P: password
  • From navigator select Update
  • This will display the current version details
  • Select Check Updates -> Check URL
  • This method will go out to VMware’s repository https://vapp-updates.vmware.com/vai-catalog/valm/… and verify you are on latest.
  • If available updates then select the Install updates -> Install all updates
  • Accept EULA
  • Wait for updates to complete.

Method 2: VAMI and custom URL: 
This method can be used if you do not have internet access from your appliances by setting up a local repository.
  • Setup a webserver on your network (ISS or Apache) that will be accessible to the vCSA and PSC.
  • Create a directory called PSC_updates or VCSA updates.  Names can be changed.
  • On VMware support site now download update but make sure to download the zip update bundle.
  • Extract the zip update bundle to the folder you created earlier.
  • Login to VAMI
  • https://vcenterserver.domain.com:5480
    • U: root / P: password
  • From navigator select Settings
  • Select “Use specified repository”
  • Click OK
  • Select Check for updates
  • If available updates then select the Install updates -> Install all update
  • Accept EULA
  • Wait for updates to complete
Method 3: VAMI and CDROM:

This method is pretty straight forward.

  • On VMware support site download the ISO for latest vCSA and/or PSC.
  • Login to vCenter Web client
  • Select vCSA or PSC appliance VM
  • Launch remote console
  • Select VMRC -> Removable devices -> CD/DVD drive 1 -> Connect to Disk Image File (iso)
  • Mount the ISO downloaded from VMware support site
  • Login to VAMI
  • https://vcenterserver.domain.com:5480/
    • U: root / P: password
  • From navigator select Update
  • Select Check Updates
  • Select Check CDROM
  • If available updates then select the Install updates -> Install all update
  • Accept EULA
  • Wait for updates to complete
Method 4: Using appliancesh command line

This method was the only way to update the appliance when vCenter 6 was released since the VAMI was only introduced in 6.0U1.  You can either mount and ISO or point to a URL for updates.  The URL can be the default VMware or 
  • On VMware support site download the ISO for latest vCSA and/or PSC.
  • Select vCSA or PSC appliance VM
  • Launch remote console
  • Select VMRC -> Removable devices -> CD/DVD drive 1 -> Connect to Disk Image File (iso)
  • Mount the ISO downloaded from VMware support site
  • SSH to VCSA or PSC
  • type # appliancesh
  • enter root password
  • To use CDROM
    • type # software-packages install –iso –acceptEulas
  • To use default VMware URL 
    • type # software-packages –url (This will use the default vmware URL to check)
  • to use local ropository URL

Product patches can be downloaded from this VMware site:
https://my.vmware.com/group/vmware/patch#search
Log files to review for updates:

/var/log/vmware/applmgmt/software-packages.log