vRealize Management 8.1 (vROPS, vRLI, vRA): What’s new 

The big takeaway with the release of vRealize Management 8.1 is the support for vSphere 7 with Kubernetes.

These updates now provides automated delivery, monitoring, troubleshooting and capacity management for both container and VM workloads!  This is a big deal and will allow VI admins to easy provision and manage VMs and containers on products they already know and not having to deal with the complexities around container orchestration with Kubernetes.

Here are some of the key updates VMware mentions for each product as well as the some use cases, but we will dig into each of these in more detail below.

Annotation 2020-03-08 095336.png

Annotation 2020-03-08 095634.png

Before I get started I do just want to mention something that mostly everyone already knows but it is important to reiterate that all these products are available on-premises as well as SaaS based.  This provides lots of flexibility for wherever you workload will run for instance in a hybrid cloud environment or perhaps in the public cloud only.Annotation 2020-03-08 100324.png

Continue reading

Spectre and Meltdown – How to check your VMware environment for vulnerabilities

Updates added to the blog

Unless you have been on a very long vacation without internet access (The BEST type of vacation!) you should know of the Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) vulnerabilities that affect nearly every computer chip manufactured in the last 20 years.

I am not going to provide any specific details on these vulnerabilities since there are more than enough material already available, which you can read here:

I do however want to provide more detailed information related to VMware specifically, as well as different ways on how you can verify what in your VMware environment is vulnerable to these exploits:

VMware responded to the Speculative Execution security issues with KB 52245, which I highly recommend you read and subscribe to.

Intel and AMD released microcode updates that provide hardware support for branch target injection mitigation, for which VMware released KB 52085. The KB provides instructions on how to enable Hypervisor-Assisted Guest Mitigation, which is required in order to use the new hardware feature within VMs.  The KB also provides manual verification instructions for the following:

  • ESXi – Verify that the microcode included in ESXi patch has been applied
  • VM – Verify that the VM is seeing the new microcode ( VM needs to on HWv9 or newer)

ALERT: VMware also released KB 52345, which rollback the recently issued security patch recommendation (ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG). The rollback is due to customers complaining of unexpected reboot after applying Intel’s initial microcode patch on Intel Haswell and Broadwell processors.

UPDATE 01.24.18: VMware updated KB 52345 to include updated list of all Intel CPUs affected by Intel Sightings

  • VMware provides some manual workarounds for these specific processors that have already been patched.
  • For ESXi hosts that have not yet applied one of the patches, VMware recommends not doing so at this time and using the patches listed in VMSA-2018-0002 instead.

That is a lot of information to take in, and the rollbacks just add complexity to IT teams who are trying to secure their customer’s data.

UPDATE 02.15.18: VMware security advisory for VMware Virtual appliance mitigation available here

UPDATE 03.20.18: VMware provided an update to KB 52085 for patching the vSphere vCenter server to latest 6.5U1g, 6.0U3e, 5.5U3h and Hypervisor to ESXi 6.5: ESXi650-201803401-BG* and ESXi650-201803402-BG**, ESXi 6.0: ESXi600-201803401-BG* and ESXi600-201803402-BG**, ESXi 5.5: ESXi550-201803401-BG* and ESXi550-201803402-BG**.

* = Framework to allow guest OSes to utilize the new speculative-execution control mechanisms

** = Applies the microcode updates


Option 1: (The best of the best)

However, to make things a bit easier we have William Lam to the rescue who wrote an excellent script that automates the verification for both the ESXi and Virtual Machines. as well as provide ESXi microcode versions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and performs the following validations

  • Verify that VM’s are running at least HWv9
  • Verify that VM completed a power cycle to see the new CPU features
  • Verify ESXi microcode has been applied
  • Verify that one of the three new CPU features are exposed to the ESXi host.
  • Verify if CPU is affected by Intel Sighting
  • Show the current Microcode version for each ESXi (requires SSH to be enabled)
  • UPDATE 01.24.18: Script was updated to validated the affected CPUs

All the detail regarding the script can be read on virtuallyGhetto here.

Option 2:  (Acceptable, but limited)

Although not nearly as thorough as William’s Script, with RuneCast Analyzer latest 1.6.7 you can detect ESXi hosts that are not protected and patched against these vulnerabilities.

Runecast Analyzer enables you to scan and detect the CPU chip vulnerabilities on your VMware infrastructure.  It detects which ESXi hosts are not protected and advise on how to patch them against such security vulnerabilities.  This solution is continuously updated as new guidance from VMware is released.

Currently only supports VMSA-2018-0002.2

Update 01.26.18: New 1.6.8 release updated to support VMSA-2018-0002.3

Screen Shot 2018-01-18 at 6.34.09 PM.png

Update 01.21.18: Option 3: (Coolest of them all)

This option does not only show what in your VMware environment is impacted but it will also assess the performance impact of both Spectre and Meltdown patches using vRealize Operations Manager (vROPS). We already know the patches will impact the speculative execution capabilities of the processor, which will lead to higher CPU utilization in your cluster due to each OS slower processing times.

The questions that come up then before patching:

  • Will I have enough resources available in my cluster to support these patches?
  • How will my ESXi host resources be impacted?
  • Should I roll out the patches in stages or all at once?

These are hard questions that are not easy to answer, or is it?

If you are using vROPS 6.6.x Advanced or Enterprise, which allows the creation of custom dashboards, then you can download and install the Spectre Meltdown Specific Dashboard kit created by Sunny Dua.  The download is available here.

The Dashboard kit consists of 3 Dashboards:

Screen Shot 2018-01-24 at 10.59.56 AM.png

  • Performance monitoring dashboard
    • Track resources utilization of your environment and will provide valuable information on the impact of patching as it relates to your Clusters, ESXi hosts  VMs.
    • Screen Shot 2018-01-24 at 11.59.56 AM.png
  • VM Patching dashboard
    • Provides views showing which VMs are running idle and can potentially be patched first since it should not have a large overall impact on performance.  Evaluate the resource utilization with the performance monitoring dashboard after the idle VM’s are upgraded, and then make a decision to continue patching or first add additional resources to the cluster.
    • Screen Shot 2018-01-24 at 11.11.41 AM.png
  • vSphere Patching dashboard
    • Shows the ESXi hosts that have been patched and also affected by Intel Sighting.
    • Shows the ESXi hosts that still needs to be patched.
    • Show the Virtual Machines that required Hardware versions upgrade since the recommended version is 9 or higher.
    • I recommend keeping an eye on VMware’s advisory site since this problem is still ongoing and the build numbers will change as new patches are released.  This will then required that you make a manual update in the filters of this dashboard
    • Screen Shot 2018-01-24 at 11.51.08 AM.png

The Performance monitoring dashboard can also be accomplished by just using the default dashboards available in vROPS standard, which means you can download the evaluation version and have that piece of mind that you can track the performance impact while going through these tough times.





Upgrading vROPS 6.x to 6.6.1

With all the new goodies in 6.6, especially the new HTML5 UI based on the Clarity design System, who can resist the upgrade to vROPS 6.6. Release notes for everything that is new can found here.

From an upgrade standpoint, vROPS has always been an interesting, but simple process with both the OS and application that requires separate updates.  The OS update is required for update RPMs for things like database and gemfire updates that the new vROPS application relies on.  My step by step upgrade guide below:

  1. Download the OS update and Product update files from my.vmware.com
    • OS PAK file:  vRealize_Operations_Manager-VA-OS-xxx.pak
    • Application PAK file:  vRealize_Operations_Manager-VA-xxx.pak
  2. Make sure that all the solutions you have installed has a version available that is compatible with the new vROPS release.
  3. If you customized any default alert definitions, symptom, recommendations, Policy Definitions, Views, Dashboards, Widgets and Reports in the previous version, make sure you clone it first.  When you upgrade vROPS, it is important that you upgrade the current versions of content types that allow you to alert on and monitor the objects in your environment.  It is a good practice to always clone first before customizing content.
  4. Before starting the upgrade, create a snapshot of the each of the nodes in the cluster.
    1. Login to vROPS admin
    2. Under system status click Take Offline
    3. Enter reason and click OK
    4. When Cluster status shows offline for all nodes, go ahead and take a snapshot of each
  5.  Before starting the upgrade, I also recommend taking a backup of all the nodes simultaneously by using your existing backup solution.
  6. First off we will update the Virtual Appliance OS:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the OS update PAK file
      • vRealize_Operations_Manager-VA-OS-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. When completed click Next
    8. Accept EULA click Next
    9. Click Next
    10. Click Install
    11. This will update the OS on the Virtual Appliances and restart them.
  7. Secondly we will perform the vROPS product update:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the application update PAK file
      • vRealize_Operations_Manager-VA-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. This will update the vROPS application on the Virtual Appliances
  8. Lastly, if you have any additional content packs installed, go ahead and upgrade them.

VMware is definitely making awesome improvements in all their products and has come a long way in helping out VMware admins with their daily management tasks.

VMware announces general availability for all vRealize Suite Standard products!

VMware has already been teasing us since June 6th with the upcoming releases of the following vRealize Suite products:

Today VMware announced GA for all products mentioned, with what seems to be a unified message to provide one integrated architecture, with greater/deeper integration across SDDC technologies and multiple public clouds.  I like where this is going…

Couple of key take aways for me which are shared amongst some of the products (not all):

  • Redesigned HTML5 UI
    • Log Insight jumped on this long ago.
  • OOTB Integration between the different products
    • We have started seeing this with previous release but not going into full swing
  • Standardizing on authentication with VIDM

Release notes for each product:


Hopefully I can make some time in the upcoming weeks to dive a bit deeper into some of the features, but due to my busy schedule I am not holding my breath 🙂 Happy downloads!

vRealize version releases today: vRA 7.1, vROPS 6.3, vRO 7.1 and vRB 7.1

VMware released new versions today for a couple of vRealize products.  Listed below with new features I think are relevant.  Full list of what’s new features provided in links at bottom of blog.

vRealize Automation 7.1

  • Silent installer
  • Migration tool to migrate data from vRA 6.2.x to fresh vRA 7.1 while preserving the source environment.
  • IPAM integration framework although Sovereign System’s SovLabs modules does a great job with this already.
  • Manual horizontal scale in and out of vRA deployments

vRealize Operations Manager 6.3

  • Enhanced workload placement and DRS integration
  • Improved log insight integration (hopefully write a blog on this soon)
  • Enhanced vSphere monitoring with new hardening policies.
  • Allow for multiple Advanced and Enterprise editions license in same deployment which means you can mix single and suite licenses.  License counting for individual license keys is handled through licensing groups.

vRealize Orchestrator 7.1

  • Extending automation configuration
  • Plugin improvements

vRealize Business for Cloud 7.1

  • Support for newer and latest vRA
  • Allow integration with external VMware Identity manager is probably the biggest one here since this now allows for a standalone installation with its own UI . I tested this earlier and you now have the option to register with either a vRA or vIDM instance.
  • Screen Shot 2016-08-23 at 5.59.57 PM
  • If you register with vIDM you get a new UI which is accessible through the FQDN of your vRB appliance.
  • Screen Shot 2016-08-23 at 6.01.31 PM
  • New version of reference database









vRealize Operations – missing dashboards when sharing

After upgrading from vRO 6.0.0 to 6.0.1 I tried to share some Dashboard tabs and templates with certain account groups.

I have multiple solutions added which creates a lot of dashboards so when viewing the dashboard window under content i see all 69.  However trying to share dashboard only a count of 50 is displayed so cannot select all dashboards.


I opened case with VMware support and they were able to reproduce the issue where the dashboards shared window cannot display more than 50 entries.
Fix will be available with next release.

Sizing vRealize Operation Manager 6.0 using your existing vCOPS 5.8 environment information.

VMware provides a comprehensive KB on sizing your vRealize Operation Manager.  However this document is very much focused on first time installation architecture and does not really provide much information on how you can use your current 5.8 environment details in conjunction with the advanced Sizing guide spreadsheet.


I will just be focusing on the input parameter of this document and how i found the information needed to complete the excel spreadsheet.  Hope this helps but if you have a better solution please let me know since this is just from my own experience.

1. Getting information for vCenter Objects:

Open the custom UI webpage for vCOPS.
Select Admin tab -> Audit report
Select your preferred report type and submit.
At the bottom of the output page you have the section called “Virtual Environment (existing resources)” .  All the details required are available in this section.

2. Custom Adapters resource and metric counts

Open Custom UI website for vCOPS 5.8
Environment tab -> Environment overview

Expand Adapter instances
Select adapter

On right hand side highlight the adapter and select the  “Detail view” button.
There might multiple instance so go through each and add up for the total count of the adapter.

On the metric selector expand the instance generated

Select the number of metrics collected and number of resources collected.

The graphs display the numbers necessary for document.

This should provide all the information needed to get the correct sizing for your new vRealize Operations 6 environment.

vCOPS – Recreate missing vSphere group dashboard

If you are familiar with the custom-ui of vCops you will find that since 5.7 you are providing with some default custom dashboards under the vsphere group.
We recently installed VNX analytics adapter which seems to have removed the vsphere group default dashboards. 
To fix this perform the following tasks:
SSH into UI VM
Browse to
cd /usr/lib/vmware-vcops/user/conf/dashboards/
Run the script  “import-dashboards.sh”
This will only share the dashboards with admin user
Edit the sh file at bottom where states:
    # import imported templates as dashboards for admin user
The xml files listed here are the custom dashboards.
These can also be copied out (for instance with WinSCP) and imported into a  manually created new dashboard.
Add lines for different users but firstly create duplicate of file for backup purposes
Cp import-dashboard.sh import-dashboard.sh.default
Vi /import-dashboards.sh
/usr/lib/vmware-vcops/user/conf/dashboards # ls -l
-rw-r–r– 1 admin admin  61794 Aug 26 04:56 DashboardTemplate-Alerts.xml
-rw-r–r– 1 admin admin 117798 Aug 26 04:56 DashboardTemplate-ClusterUtilization.xml
-rw-r–r– 1 admin admin  90979 Aug 26 04:56 DashboardTemplate-DatastorePerformance.xml
-rw-r–r– 1 admin admin  74401 Aug 26 04:56 DashboardTemplate-DatastoreSpace.xml
-rw-r–r– 1 admin admin 170642 Aug 26 04:56 DashboardTemplate-Heatmaps.xml
-rw-r–r– 1 admin admin  77214 Aug 26 04:56 DashboardTemplate-HostMemory.xml
-rw-r–r– 1 admin admin 117370 Aug 26 04:56 DashboardTemplate-HostUtilization.xml
-rw-r–r– 1 admin admin  76295 Aug 26 04:56 DashboardTemplate-Troubleshooting.xml
-rw-r–r– 1 admin admin 128042 Aug 26 04:56 DashboardTemplate-VMPerformance.xml
-rw-r–r– 1 admin admin 117626 Aug 26 04:56 DashboardTemplate-VMUtilization.xml
  # import imported templates as dashboards for all administrator users
    bash  ./dbcli.sh dashboard import group:Administrators $TD_1 –default true –set 1
    bash  ./dbcli.sh dashboard import group:Administrators $TD_2
    bash  ./dbcli.sh dashboard import group:Administrators $TD_3
    bash  ./dbcli.sh dashboard import group:Administrators $TD_4
    bash  ./dbcli.sh dashboard import group:Administrators $TD_5
    bash  ./dbcli.sh dashboard import group:Administrators $TD_6
    bash  ./dbcli.sh dashboard import group:Administrators $TD_7
    bash  ./dbcli.sh dashboard import group:Administrators $TD_8
    bash  ./dbcli.sh dashboard import group:Administrators $TD_9

    bash  ./dbcli.sh dashboard import group:Administrators $TD_10
From vCOPS 5.7.3  release notes:
  • All and group options for dashboard commands
    For the dashboard import, dashboard delete, and dashboard reorder commands, you can use one of the following options instead of specifying a user name:
    * --all
    * --group:<group_name[,]

    The --all option applies the command to all user accounts. The --group:<group_names[,] option applies the command to all user accounts that belong to the specified user groups. For example, the command dashboard import group:Administrators,Operators "templates\MyDashboard.bin" imports the dashboard MyDashboard.bin to all user accounts that belong to the Administrators and Operators user groups.
    The dashboard import command also has a --set option that enables you to set the dashboard order. For example, the command dashboard import MyUser2 "dashboards/MyDashboard.bin" --set 1 imports the dashboard MyDashboard.bin and makes it the first dashboard in the dashboard list for the user MyUser2.

vCenter Operations Manager (vCops) errors and not collecting stats (/data out of space)

Recently I logged into vCenter Operations Manager (vCOPS) custom UI and found that all objects stopped collecting stats.


Logged into vCenter server and tried health status tab but received connection error.
In Custom UI select Environment tab -> Configuration -> Adapter instances.   Edit a vcenter instance and selected test. Received the following error

 “Test was not successful.  Connection refused to host: Secondvm-internal;nested exception is: java.net.ConnectionException: Connection refused”

I ran the same test on our VNX and Brocade adapter instances and received the exact same error.

In vCops Administration page under registration tab i selected the update bottom on vCenter Server registration and re-entered the password.  After a long while i received an exception error.
I went to status tab and saw that some of the vCenter Operations Manager Statuses was not running.
I tried to start the service but did not work and status did not change nor did i get an error message.

I SSH into the Analytics VM and tried to manually restart the service “vcops-admin restart”.  showed that the database was not coming online on which i though about looking at space.

Ran “df -h” and found that the /data drive was out of space.


Shut down vCops vapp container from vCenter server.
Edit the VM and add new disk.
Power on the vapp container.

vCops will automatically extend the /data drive with the newly added disk.