Spectre and Meltdown – How to check your VMware environment for vulnerabilities

Updates added to the blog

Unless you have been on a very long vacation without internet access (The BEST type of vacation!) you should know of the Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) vulnerabilities that affect nearly every computer chip manufactured in the last 20 years.

I am not going to provide any specific details on these vulnerabilities since there are more than enough material already available, which you can read here:

I do however want to provide more detailed information related to VMware specifically, as well as different ways on how you can verify what in your VMware environment is vulnerable to these exploits:

VMware responded to the Speculative Execution security issues with KB 52245, which I highly recommend you read and subscribe to.

Intel and AMD released microcode updates that provide hardware support for branch target injection mitigation, for which VMware released KB 52085. The KB provides instructions on how to enable Hypervisor-Assisted Guest Mitigation, which is required in order to use the new hardware feature within VMs.  The KB also provides manual verification instructions for the following:

  • ESXi – Verify that the microcode included in ESXi patch has been applied
  • VM – Verify that the VM is seeing the new microcode ( VM needs to on HWv9 or newer)

ALERT: VMware also released KB 52345, which rollback the recently issued security patch recommendation (ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG). The rollback is due to customers complaining of unexpected reboot after applying Intel’s initial microcode patch on Intel Haswell and Broadwell processors.

UPDATE 01.24.18: VMware updated KB 52345 to include updated list of all Intel CPUs affected by Intel Sightings

  • VMware provides some manual workarounds for these specific processors that have already been patched.
  • For ESXi hosts that have not yet applied one of the patches, VMware recommends not doing so at this time and using the patches listed in VMSA-2018-0002 instead.

That is a lot of information to take in, and the rollbacks just add complexity to IT teams who are trying to secure their customer’s data.

Option 1: (The best of the best)

However, to make things a bit easier we have William Lam to the rescue who wrote an excellent script that automates the verification for both the ESXi and Virtual Machines. as well as provide ESXi microcode versions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and performs the following validations

  • Verify that VM’s are running at least HWv9
  • Verify that VM completed a power cycle to see the new CPU features
  • Verify ESXi microcode has been applied
  • Verify that one of the three new CPU features are exposed to the ESXi host.
  • Verify if CPU is affected by Intel Sighting
  • Show the current Microcode version for each ESXi (requires SSH to be enabled)
  • UPDATE 01.24.18: Script was updated to validated the affected CPUs

All the detail regarding the script can be read on virtuallyGhetto here.

Option 2:  (Acceptable, but limited)

Although not nearly as thorough as William’s Script, with RuneCast Analyzer latest 1.6.7 you can detect ESXi hosts that are not protected and patched against these vulnerabilities.

Runecast Analyzer enables you to scan and detect the CPU chip vulnerabilities on your VMware infrastructure.  It detects which ESXi hosts are not protected and advise on how to patch them against such security vulnerabilities.  This solution is continuously updated as new guidance from VMware is released.

Currently only supports VMSA-2018-0002.2

Update 01.26.18: New 1.6.8 release updated to support VMSA-2018-0002.3

Screen Shot 2018-01-18 at 6.34.09 PM.png

Update 01.21.18: Option 3: (Coolest of them all)

This option does not only show what in your VMware environment is impacted but it will also assess the performance impact of both Spectre and Meltdown patches using vRealize Operations Manager (vROPS). We already know the patches will impact the speculative execution capabilities of the processor, which will lead to higher CPU utilization in your cluster due to each OS slower processing times.

The questions that come up then before patching:

  • Will I have enough resources available in my cluster to support these patches?
  • How will my ESXi host resources be impacted?
  • Should I roll out the patches in stages or all at once?

These are hard questions that are not easy to answer, or is it?

If you are using vROPS 6.6.x Advanced or Enterprise, which allows the creation of custom dashboards, then you can download and install the Spectre Meltdown Specific Dashboard kit created by Sunny Dua.  The download is available here.

The Dashboard kit consists of 3 Dashboards:

Screen Shot 2018-01-24 at 10.59.56 AM.png

  • Performance monitoring dashboard
    • Track resources utilization of your environment and will provide valuable information on the impact of patching as it relates to your Clusters, ESXi hosts  VMs.
    • Screen Shot 2018-01-24 at 11.59.56 AM.png
  • VM Patching dashboard
    • Provides views showing which VMs are running idle and can potentially be patched first since it should not have a large overall impact on performance.  Evaluate the resource utilization with the performance monitoring dashboard after the idle VM’s are upgraded, and then make a decision to continue patching or first add additional resources to the cluster.
    • Screen Shot 2018-01-24 at 11.11.41 AM.png
  • vSphere Patching dashboard
    • Shows the ESXi hosts that have been patched and also affected by Intel Sighting.
    • Shows the ESXi hosts that still needs to be patched.
    • Show the Virtual Machines that required Hardware versions upgrade since the recommended version is 9 or higher.
    • I recommend keeping an eye on VMware’s advisory site since this problem is still ongoing and the build numbers will change as new patches are released.  This will then required that you make a manual update in the filters of this dashboard
    • Screen Shot 2018-01-24 at 11.51.08 AM.png

The Performance monitoring dashboard can also be accomplished by just using the default dashboards available in vROPS standard, which means you can download the evaluation version and have that piece of mind that you can track the performance impact while going through these tough times.

Links:

https://communities.vmware.com/message/2738226#2738226

https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html

https://kb.vmware.com/s/article/2143832?r=2&Quarterback.validateRoute=1&KM_Utility.getArticleData=1&KM_Utility.getGUser=1&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1

Upgrading vROPS 6.x to 6.6.1

With all the new goodies in 6.6, especially the new HTML5 UI based on the Clarity design System, who can resist the upgrade to vROPS 6.6. Release notes for everything that is new can found here.

From an upgrade standpoint, vROPS has always been an interesting, but simple process with both the OS and application that requires separate updates.  The OS update is required for update RPMs for things like database and gemfire updates that the new vROPS application relies on.  My step by step upgrade guide below:

  1. Download the OS update and Product update files from my.vmware.com
    • OS PAK file:  vRealize_Operations_Manager-VA-OS-xxx.pak
    • Application PAK file:  vRealize_Operations_Manager-VA-xxx.pak
  2. Make sure that all the solutions you have installed has a version available that is compatible with the new vROPS release.
  3. If you customized any default alert definitions, symptom, recommendations, Policy Definitions, Views, Dashboards, Widgets and Reports in the previous version, make sure you clone it first.  When you upgrade vROPS, it is important that you upgrade the current versions of content types that allow you to alert on and monitor the objects in your environment.  It is a good practice to always clone first before customizing content.
  4. Before starting the upgrade, create a snapshot of the each of the nodes in the cluster.
    1. Login to vROPS admin
    2. Under system status click Take Offline
    3. Enter reason and click OK
    4. When Cluster status shows offline for all nodes, go ahead and take a snapshot of each
  5.  Before starting the upgrade, I also recommend taking a backup of all the nodes simultaneously by using your existing backup solution.
  6. First off we will update the Virtual Appliance OS:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the OS update PAK file
      • vRealize_Operations_Manager-VA-OS-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. When completed click Next
    8. Accept EULA click Next
    9. Click Next
    10. Click Install
    11. This will update the OS on the Virtual Appliances and restart them.
  7. Secondly we will perform the vROPS product update:
    1. Login to the master vROPS node administrator interface
    2. Select Software Update
    3. Click Install a Software Update
    4. Browse the application update PAK file
      • vRealize_Operations_Manager-VA-xxx.pak
    5. Check the box “Reset Default Content”
      • As mentioned above make sure you have cloned all your customized content!
    6. Click Upload
    7. This will update the vROPS application on the Virtual Appliances
  8. Lastly, if you have any additional content packs installed, go ahead and upgrade them.

VMware is definitely making awesome improvements in all their products and has come a long way in helping out VMware admins with their daily management tasks.

VMware announces general availability for all vRealize Suite Standard products!

VMware has already been teasing us since June 6th with the upcoming releases of the following vRealize Suite products:

Today VMware announced GA for all products mentioned, with what seems to be a unified message to provide one integrated architecture, with greater/deeper integration across SDDC technologies and multiple public clouds.  I like where this is going…

Couple of key take aways for me which are shared amongst some of the products (not all):

  • Redesigned HTML5 UI
    • Log Insight jumped on this long ago.
  • OOTB Integration between the different products
    • We have started seeing this with previous release but not going into full swing
  • Standardizing on authentication with VIDM

Release notes for each product:

 

Hopefully I can make some time in the upcoming weeks to dive a bit deeper into some of the features, but due to my busy schedule I am not holding my breath 🙂 Happy downloads!

vRealize version releases today: vRA 7.1, vROPS 6.3, vRO 7.1 and vRB 7.1

VMware released new versions today for a couple of vRealize products.  Listed below with new features I think are relevant.  Full list of what’s new features provided in links at bottom of blog.

vRealize Automation 7.1

  • Silent installer
  • Migration tool to migrate data from vRA 6.2.x to fresh vRA 7.1 while preserving the source environment.
  • IPAM integration framework although Sovereign System’s SovLabs modules does a great job with this already.
  • Manual horizontal scale in and out of vRA deployments

vRealize Operations Manager 6.3

  • Enhanced workload placement and DRS integration
  • Improved log insight integration (hopefully write a blog on this soon)
  • Enhanced vSphere monitoring with new hardening policies.
  • Allow for multiple Advanced and Enterprise editions license in same deployment which means you can mix single and suite licenses.  License counting for individual license keys is handled through licensing groups.

vRealize Orchestrator 7.1

  • Extending automation configuration
  • Plugin improvements

vRealize Business for Cloud 7.1

  • Support for newer and latest vRA
  • Allow integration with external VMware Identity manager is probably the biggest one here since this now allows for a standalone installation with its own UI . I tested this earlier and you now have the option to register with either a vRA or vIDM instance.
  • Screen Shot 2016-08-23 at 5.59.57 PM
  • If you register with vIDM you get a new UI which is accessible through the FQDN of your vRB appliance.
  • Screen Shot 2016-08-23 at 6.01.31 PM
  • New version of reference database

 

Links:

http://pubs.vmware.com/Release_Notes/en/vrops/63/vrops-63-release-notes.html#intro

http://pubs.vmware.com/Release_Notes/en/vra/vrealize-automation-71-release-notes.html#about

http://pubs.vmware.com/Release_Notes/en/orchestrator/vrealize-orchestrator-71-release-notes.html#new

http://pubs.vmware.com/Release_Notes/en/vRBforCloud/71/vRBforCloud-71-release-notes.html#whatsnew

 

 

vRealize Operations – missing dashboards when sharing

After upgrading from vRO 6.0.0 to 6.0.1 I tried to share some Dashboard tabs and templates with certain account groups.

I have multiple solutions added which creates a lot of dashboards so when viewing the dashboard window under content i see all 69.  However trying to share dashboard only a count of 50 is displayed so cannot select all dashboards.

Solution:

I opened case with VMware support and they were able to reproduce the issue where the dashboards shared window cannot display more than 50 entries.
Fix will be available with next release.

Sizing vRealize Operation Manager 6.0 using your existing vCOPS 5.8 environment information.

VMware provides a comprehensive KB on sizing your vRealize Operation Manager.  However this document is very much focused on first time installation architecture and does not really provide much information on how you can use your current 5.8 environment details in conjunction with the advanced Sizing guide spreadsheet.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2093783

I will just be focusing on the input parameter of this document and how i found the information needed to complete the excel spreadsheet.  Hope this helps but if you have a better solution please let me know since this is just from my own experience.

1. Getting information for vCenter Objects:

Open the custom UI webpage for vCOPS.
Select Admin tab -> Audit report
Select your preferred report type and submit.
At the bottom of the output page you have the section called “Virtual Environment (existing resources)” .  All the details required are available in this section.

2. Custom Adapters resource and metric counts

Open Custom UI website for vCOPS 5.8
Environment tab -> Environment overview

Expand Adapter instances
Select adapter

On right hand side highlight the adapter and select the  “Detail view” button.
There might multiple instance so go through each and add up for the total count of the adapter.

On the metric selector expand the instance generated

Select the number of metrics collected and number of resources collected.

The graphs display the numbers necessary for document.

This should provide all the information needed to get the correct sizing for your new vRealize Operations 6 environment.