vCenter Server Web Client crash with latest Flash 27.0.0.170

Within the last couple of days Google ran an automatic update of Flash, which is causing havoc in the all-things-flash world.

Since Google manages the upgrade of the flash you are kind of at their mercy on this one, until all applications finally learn to stay as far away from flash as possible! Can only hope!

You can however disable the automatic update feature for flash by setting the registry HKLM\SOFTWARE\Policies\Google\Update\AutoUpdateCheckPeriodMinutes to 0.

My fix was pretty straight forward and I only had to delete the latest release folder from the path “%LocalAppData%\Google\Chrome\User Data\PepperFlash” (windows) or “~/Library/Application Support/Google/Chrome/PepperFlash” (MAC).

In my case the older version was still available in the same folder so I only had to delete the latest and restart google but if do not have the older version you can download it here.

From other users responses it seems that the version 27.0.0.130 and older still works but newer version will crash on both your Google Chrome and Firefox browsers.

Update:  VMware has release a KB here

 

 

Migration of vCenter Server using VMware Migration Assistant

The migration assistant can tackle a couple of different vCenter Server configurations:

  • vCenter Server 5.5 or 6.0 with an embedded vCenter Single Sign-On instance on Windows
  • vCenter Server 5.5 instance on Windows with external SSO
  • vCenter server 6.0 instance on Windows with external PSC

Depending on your current configuration, the migration process will change based using an internal or external SSO/PSC and where VUM is installed.

  • If VUM is installed on a separate Windows server, other that your Windows vCenter Server instance then you MUST to run the migration assistant on this server FIRST!  The VMware Migration Assistant facilitates the migration of the Update Manager server and database to the vCenter Server Appliance 6.5.
  • If embedded SSO/PSC, then you run the Migration Assistant on the source vCenter Server
  • If external SSO/PSC, then you run the Migration Assistant on the source SSO/PSC first and then run the Migration Assistant on the source vCenter Server
    • During the migration process, make sure to leave the migration windows open

You run the VMware Migration Assistant on your source vCenter Server, SSO/PSC or VUM it will perform the following tasks:

  • Discover the source deployment type.
  • Run pre-checks on the source.
  • Report errors that must be addressed before starting the migration.
  • Provide information for the next steps in the migration process.

Here are my step by step instructions for vCenter Server 5.5 with embedded SSO, no VUM:

  1. Download and Mount the vCenter Server Appliance Installer ISO file to the Windows vCenter Server.
  2. Start the migration assistant on the source, depending on your environment configuration
    1. Open the installer folder on CD/DVD drive and copy the  “migration-assistant” folder to your source windows server running the vCenter Server
    2. Open the copied folder on local drive and run “VMware-migration-assistant.exe”
      1. This will run the pre-update checklist and prompt if anything needs to be fixed.
      2. Make sure this window stays open during the whole migration process until completion.
  3. Launch vCenter server appliance UI installer
    1. Click Migrate
    2. Click Next on Introduction
    3. Accept EULA and click Next
    4. Connect to the source server
      1. Enter Windows server FQDN or IP address
      2. Enter migration assistant port number
        • If you are changing networks and have a firewall in place then pick a port that is open and available.
      3. SSO username
        • administrator@vsphere.local
      4. SSO Password
      5. Click Next
    5. Accept Thumbprint
    6. Appliance deployment target
      1. Enter ESXi or vCenter Server name for targer
      2. HTTPS port
      3. Username
      4. Password
      5. Click Next
    7. Accept Thumbprint
    8. Select Folder to place the new vCSA
    9. Select the compute resource
    10. Target appliance VM
      1. Enter VM name
      2. Enter new root password
    11. Select deployment size
    12. Select datastore
    13. Configure the temporary network identity which is used to copy the data, this will be removed after migration is completed since the target vCSA server will get the network identity of the source vCenter server.
      1. Pick network port group
      2. Enter temp IP address
      3. Enter subnet
      4. Enter GW
      5. Enter DNS servers
      6. Click Next
    14. Verify detailed entered are correct!
    15. Click Finish
    16. Wait for the migration to complete.
      • If the migration fails for any reason your can easily roll back by powering off the target vCSA and then just power on the source vCenter Server and its components.
  4. Now for stage two, which copies the data from the source vCenter to new vCSCA.
    1. On Introduction click next
    2. Connect to Source vCenter should complete automatically
    3. Join AD
      • Add username and password for user that can add computer to domain
    4. Select the best migration data options with the following 3 being available. (pretty cool that it provides the size of the migration for each option as well)
      • Just the configuration
      • Configuration, Events and Tasks
      • Configuration, Events, Tasks and Performance metrics
    5. Select if you want to join CEIP and click Next
    6. Review your settings, check the box to verify you have a backup of the vCenter server as well as the database.  Click Next
    7. Click OK on prompt that warns you that the source vCenter Server will be shut down once the network configuration is enabled.
    8. Wait for the data migration to complete.

When successful, your source vCenter Server should now be shut down and brand new shiny vCenter Server appliance should be started in its place. The best part of all you are also upgraded to vCenter Server 6.5! awesome.

 

 

vCenter Server 6.5 error: Cannot start content-library service

I recently discovered a problem with my vCenter Server after an upgrade from 6.0 to 6.5 of the vCenter Server Appliance, which runs in linked mode.

When I tried to take a backup of the vCenter Server through the VAMI I got an error “Invalid vCenter Server Status: All required services are not up! Stopped services: ‘content-library’.

Troubleshooting:

My first attempt was trying to start the service

  • I logged into the vCenter Server web client and went to administration -> Deployment -> System Configuration.
  • Double click on the the vCSA node
  • Select services
  • Select Content Library service, which under summary showed stopped.
  • Click in Actions -> Starts
  • Service failed to start with very cryptic error “An error occurred while starting service ‘%(0)s”

I also ssh into the vCSA and tried to start the service, and see the following error when trying to start the service:

Error executing start on service content-library. Details {
“resolution”: null,
“detail”: [
{
“args”: [
“content-library”
],
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘content-library'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
}
],
“componentKey”: null,
“problemId”: null
}
Service-control failed. Error {
“resolution”: null,
“detail”: [
{
“args”: [
“content-library”
],
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘content-library'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
}
],
“componentKey”: null,
“problemId”: null
}

Since the error message did not help much I went to search on VMware community and found other users having issues with creating content libraries after upgrade and VMware support pointed them to a KB article.

The fix:

The solution comes from KB 2151085 , however I am on version 6.5 not 6.5U1, as well as the OVF stuff was not applicable to my particular situation but the steps provided did resolve my problem and I was able to start the content-libary services.

I compared the files with a vCenter 6.5 environment that is working and found that the ts-config.properties file had the owner and group set to root:root and should be set to content-library:cis.  This is an easy fix and also solved the problem for me.   “chown content-library:cis ts-config.properties”

 

 

 

Upgrading vROPS 6.x to 6.6.1

With all the new goodies in 6.6, especially the new HTML5 UI based on the Clarity design System, who can resist the upgrade to vROPS 6.6. Release notes for everything that is new can found here.

From an upgrade standpoint, vROPS has always been an interesting, but simple process with both the OS and application that requires separate updates.  The OS update is required for update RPMs for things like database and gemfire updates that the new vROPS application relies on.  My step by step upgrade guide below:

  1. Download the OS update and Product update files from my.vmware.com
    • OS PAK file:  vRealize_Operations_Manager-VA-OS-xxx.pak
    • Application PAK file:  vRealize_Operations_Manager-VA-xxx.pak
  2. Make sure that all the solutions you have installed has a version available that is compatible with the new vROPS release.
  3. If you customized any default alert definitions, symptom, recommendations, Policy Definitions, Views, Dashboards, Widgets and Reports in the previous version, make sure you clone it first.  When you upgrade vROPS, it is important that you upgrade the current versions of content types that allow you to alert on and monitor the objects in your environment.  It is a good practice to always clone first before customizing content.
  4. Before starting the upgrade, create a snapshot of the each of the nodes in the cluster.
    1. Login to vROPS admin
    2. Under system status click Take Offline
    3. Enter reason and click OK
    4. When Cluster status shows offline for all nodes, go ahead and take a snapshot of each
  5.  Before starting the upgrade, I also recommend taking a backup of all the nodes simultaneously by using your existing backup solution.
  6. First off we will update the Virtual Appliance OS:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the OS update PAK file
      • vRealize_Operations_Manager-VA-OS-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. When completed click Next
    8. Accept EULA click Next
    9. Click Next
    10. Click Install
    11. This will update the OS on the Virtual Appliances and restart them.
  7. Secondly we will perform the vROPS product update:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the application update PAK file
      • vRealize_Operations_Manager-VA-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. This will update the vROPS application on the Virtual Appliances
  8. Lastly, if you have any additional content packs installed, go ahead and upgrade them.

VMware is definitely making awesome improvements in all their products and has come a long way in helping out VMware admins with their daily management tasks.

SovLabs: Upgrading your software

SovLabs isn’t just a vRA plugin, it’s enterprise software that extends the capabilities of your vRealize Automation environment providing you with that end-to-end solution you have been craving for.  As with any other enterprise software they periodically provide new patches and releases and with SoLabs that is no different.

The new 2017.3.x was released in August and provides some awesome new modules:

  • Men & Mice DNS and IPAM
  • SolarWinds DNS
  • Backup as a Service
    • Automate policy-driven backups and provide self-service VM and file-level recovery for –
    • Cohesity
    • Rubrik
    • Veeam
  • SovLabs VM tagging
    • Drive rich metadata using VM tags and categories
  • SovLabs Property Toolkit
    • Manage your existing custom properties on VMs with the SovLabs Template Engine
  • ServiceNow Support for Jakarta
  • Puppet support for 2017.1
  • VMware Tools connection
    • Connect to Windows/Linux servers can now be done through VMware Tools which removes the requirement for WinRM, CygwinSSH or WinSSHD to be installed.  This is huge!
  • As a customer you can sign up under the self-service portal and view the detailed release notes here:

So how do we go about upgrading SovLabs to the latest version?

Step by step guide to upgrading from 2017.2.x to 2017.3.x.  (there are some additional steps if you are upgrading from <= 2017.1.x so please contact SovLabs support) 

  1. First off we want to create a backup of the vRO package
    1. Login to vRO Client
    2. Click Design
    3. Click on the package tab
    4. Click on the package icon on right hand side menu bar
    5. Enter name “com.sovlabs.backup.resources”
    6. Edit the newly create package, click on the pencil icon on the right hand side menu bar
    7. Click the Resources tab
    8. Click the Folder + icon
    9. Expand the Library folder,  select the SovLabs folder
    10. Click on the Select button
    11. Once loaded, click save and close
    12. Right click the saved package and click export package
      1. Create a folder called sovlabs under downloads
      2. leave the rest of settings as default
    13. Save to your local system
    14. Now, lets save the old SovLabs Plugin:
      1. Use WinSCP and login as root to vRO appliance
      2. Go to directory /var/lib/vco/app-server/plugins
      3. Save the o11nplugin-sovlabs.dar to your local file system in same sovlabs folder created earlier under download.s
    15. Done!
  2. We need to update the vRO Heap size
    1. If you have done this before then you can skip this step but this is needed to install the larger sized SovLabs module file into vRO otherwise the appliance might run out of memory during install/upgrade.
    2. Remember if you a vRO cluster, then you have to perform the steps on both server
    3. SSH into vRO appliance with user root
    4. Run # vi /var/lib/vco/configuration/bin/setenv.sh
    5. Find the #MEM_OPTS section
    6. Replace the -Xmx512m \ with -Xmx768m \
    7. Save the file
  3. Delete all SovLabs license keys
    1. Login to vRA tenant
    2. Click on Items tab -> SovLabs vRA Extensibility modules -> SovLabs License
    3. For each SovLabs License item listed
      1. Select Actions -> Delete License
  4. Download the SovLabs plugin
    1. Talk to SovLabs support about getting the software downloaded.
  5. Install the plugin into vRO appliance
    1. Login to controlcenter
      1. https://<vroserver&gt;:8283/vco-controlcenter
    2. Select Plug-Ins -> Manage Plug-ins
    3. Click Browse
    4. Select the plugin
    5. Accept EULA
    6. Click on Install
    7. Accept the EULA
    8. Restart the vRO server
      1. On the Home page, click on the Startup Options icon
      2. Click on Restart
      3. Wait for vRO to restart successfully
    9. Log back in to the vRO configuration page
    10. Click on the Manage Plug-Ins icon
    11. Verify that the installed plugin is listed among the vRO plugins
    12. Now if you have a clustered vRO 7.2 and above, then the plugin should sync but I have seen some problems with 7.2 so follow these steps
      1. Perform a full reboot on primary so that the pending and active config fingerprint ID match.
      2. Then push the config to the other standby node
      3. It will need to rebooted which it often will not do so make sure you perform this step yourself.
      4. Verify that Synchronization state shows synchronized and verify the version of the plugin on both active and standby nodes.
  6. Login to the vRO Client and run the configuration
    1. Click on Design mode
    2. Click on WorkFlow tab
    3. Right click vRO workflow, “SovLabs/Configuration/SovLabs Configuration”
    4. Select Start Workflow
    5. The SovLabs Configuration workflow only needs to be run on one vRO in a clustered environment
      1. Select yes to accept the EULA
      2. Click Next
      3. Select the appropriate tenant and business group
      4. Create SovLabs vRA Catalog Service? = No
      5. Publish License Content? = No
      6. Click Next
      7. Upgrade existing SovLabs vRA content? = Yes
      8. Click Next
      9. Install or Update SovLabs workflow subscriptions (vRA7.x)? = Yes
        1. *Enables vRA to call vRO during machine lifecycles
      10. Click Submit
      11. Verify that the SovLabs Configuration workflow completed successfully
  7. Lastly, let’s verify the SovLabs Plugin in vRA
    1. Select Catalog tab
    2. Verify that Add license -> SovLabs Modules catalog exists
  8. Now lets install the new license key for 2017.3.x
    1. This process has also been drastically simplified with a single license key which will license all modules, where previously this was done one at a time.
    2. Select Catalog tab -> SovLabs vRA Extensibility Modules -> Add license – SovLabs Modules
    3. Copy the text from license file and paste into field
    4. Click Submit
    5. Verify all catalog tab -> SovLabs vRA Extensibility Modules that all catalogs are available.
  9. If you ever need to roll back then follow the steps in the document provided by SovLabs:
    1. https://s3.amazonaws.com/docs.sovlabs.com/vRA7x/guides/SovLabs_BackupRestore-vROPackage.pdf

 

 

 

 

vRealize Automation: Request stuck in progress

I ran into an interesting problem today on my distributed (enterprise) vRA 7.2 environment and wanted to share how I got it resolved.

I have not deployed anything in my environment for a while but when I tried today my request was not completing and status is showing “In Progress”

Troubleshooting:

Review logs:

  • Infrastructure -> Monitoring -> Audit Logs
    • Machine requests shows that is was started
  • Infrastructure -> Monitoring -> Log
    • Found error on my manager services nodes “[EventBrokerService] Failed resuming workflow.. State VMPSMasterWorkflow32.Requested(POST). Event
      Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue ’30da8a16-c532-4e13-bd81-39b09114a887′.”
  • Logged into Service manager nodes and review the logs in Event Viewer
    • Found error “Error occurred while registering the DEM.
      System.Data.Services.Client.DataServiceTransportException: The underlying connection was closed: An unexpected error occurred on a send. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Authentication failed because the remote party has closed the transport stream”
  • Logged into the Web server nodes and review the logs in Event Viewer
    • Found similar error as above
    • Found error messages like “Error occurred writing to the repository tracking log”, “Error occurred while pinging repository”

Review DEM status:

  • Infrastructure -> Monitoring -> DEM status
    • both my DEM worker and Orchestrator shows with Status Active (Green)

Resolution:

I did some investigation and found really 2 problems that I needed to address

  1. If you find errors like “Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue” then you probably have manager service running on both instances (nodes).
  2. If you find errors like “System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it” then the problem is most likely with certificates and found in the vRA documentation that if you have commas in your OU section of the IaaS certificate, that your VM provisioning might fail and the following work around is provided
    1. Remove the commas from the OU section of the IaaS certificate, OR
    2. Change the polling method from WebSocket to HTTP to resolve the issues.
      • Open the Manager Service configuration file in a text editor.
      • C:\:Program Files (x86)\VMware\vCAC\Server\Manager Service.exe.config.
      • Add the following lines to <appSettings>
      • <add key=”Extensibility.Client.RetrievalMethod” value=”Polling”/>
        <add key=”Extensibility.Client.PollingInterval” value=”2000″/>
        <add key=”Extensibility.Client.PollingMaxEvents” value=”128″/>
      • Restart the manager services

Some other things to verify:

  • On Web server Windows OS nodes
      • Verify that the VMware Cloud Automation Center Management agent services is running
  • On Manager service Window OS nodes
    • Verify that the VMware Cloud Automation Center Service is running
      • This should only be running on 1 server if have a load balancer in front.
      • Set the Startup type to Manual on the 2nd server so you don’t have worry about this service starting but remember you have to failover manually by changing the service to automatic and starting the service.
        • In vRA 7.3 the failover process is now automated which is great!
    • Verify that the VMware Cloud Automation Center Management agent services is running on your instances
  • On DEM server Windows OS nodes
    • Verify that your VMware vCloud Automation Center Agent and Management agent services is running
  • Most people do not know this but VMware also has a very cool vRealize production test tool which I will blog about shortly.

Links:

https://docs.vmware.com/en/vRealize-Automation/7.2/com.vmware.vrealize.automation.doc/GUID-71F4F6F1-DBAE-4E0B-83A5-B4B25921B6A7.html

https://docs.vmware.com/en/vRealize-Automation/7.0/com.vmware.vra.extensibility.doc/GUID-7E54F8B9-9F76-4470-9B6F-6DAE5972E740.html

 

 

Adding Microsoft Azure to vRA and vRB Part #2

In part 1 I showed how to add Microsoft Azure to vRA.  In this part 2 I will show how to add Microsoft Azure with Non-EA account to vRealize Business which will provide cost information for your MS Azure account.

I have to apologies for taking so long to publish this but I had the blog written and ready to go but it was created with vRB 7.2, which had a lot of bugs with Azure integration  and the documentation was not very thorough and made use of the old Azure portal interface for configuration.  The problem I ran into can be view in the community post here, but with a lot of views and not responses I decided to wait until vRB 7.3 to review this again.

Prerequisites:

  1. You must have a Microsoft Azure Enterprise Agreement (EA) or non-EA account.
  2. If using MS Azure non-EA you must have one of the following credits offers:
    1. Pay-as-you-go
    2. MSDN
    3. Monetary commitment
    4. Monetary credit

To add a non-EA account you will also need the following information during configuration so please make sure you have this available.  I am also providing the steps on how to configure your non-EA account.

  • Client ID
    • When you register a client app, such as a console app, you receive a Client ID. The Client ID is used by the application to identify themselves to the users that they are requesting permissions from.
  • Location of Purchase
  • Tenant ID
    • Value can be retrieved from the Azure default Active directory when you select manage -> properties in menu.
  • Secret Key
    • Value will be defined during app registration.

Continue reading