Spectre and Meltdown – How to check your VMware environment for vulnerabilities

Updates added to the blog

Unless you have been on a very long vacation without internet access (The BEST type of vacation!) you should know of the Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) vulnerabilities that affect nearly every computer chip manufactured in the last 20 years.

I am not going to provide any specific details on these vulnerabilities since there are more than enough material already available, which you can read here:

I do however want to provide more detailed information related to VMware specifically, as well as different ways on how you can verify what in your VMware environment is vulnerable to these exploits:

VMware responded to the Speculative Execution security issues with KB 52245, which I highly recommend you read and subscribe to.

Intel and AMD released microcode updates that provide hardware support for branch target injection mitigation, for which VMware released KB 52085. The KB provides instructions on how to enable Hypervisor-Assisted Guest Mitigation, which is required in order to use the new hardware feature within VMs.  The KB also provides manual verification instructions for the following:

  • ESXi – Verify that the microcode included in ESXi patch has been applied
  • VM – Verify that the VM is seeing the new microcode ( VM needs to on HWv9 or newer)

ALERT: VMware also released KB 52345, which rollback the recently issued security patch recommendation (ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG). The rollback is due to customers complaining of unexpected reboot after applying Intel’s initial microcode patch on Intel Haswell and Broadwell processors.

UPDATE 01.24.18: VMware updated KB 52345 to include updated list of all Intel CPUs affected by Intel Sightings

  • VMware provides some manual workarounds for these specific processors that have already been patched.
  • For ESXi hosts that have not yet applied one of the patches, VMware recommends not doing so at this time and using the patches listed in VMSA-2018-0002 instead.

That is a lot of information to take in, and the rollbacks just add complexity to IT teams who are trying to secure their customer’s data.

Option 1: (The best of the best)

However, to make things a bit easier we have William Lam to the rescue who wrote an excellent script that automates the verification for both the ESXi and Virtual Machines. as well as provide ESXi microcode versions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and performs the following validations

  • Verify that VM’s are running at least HWv9
  • Verify that VM completed a power cycle to see the new CPU features
  • Verify ESXi microcode has been applied
  • Verify that one of the three new CPU features are exposed to the ESXi host.
  • Verify if CPU is affected by Intel Sighting
  • Show the current Microcode version for each ESXi (requires SSH to be enabled)
  • UPDATE 01.24.18: Script was updated to validated the affected CPUs

All the detail regarding the script can be read on virtuallyGhetto here.

Option 2:  (Acceptable, but limited)

Although not nearly as thorough as William’s Script, with RuneCast Analyzer latest 1.6.7 you can detect ESXi hosts that are not protected and patched against these vulnerabilities.

Runecast Analyzer enables you to scan and detect the CPU chip vulnerabilities on your VMware infrastructure.  It detects which ESXi hosts are not protected and advise on how to patch them against such security vulnerabilities.  This solution is continuously updated as new guidance from VMware is released.

Currently only supports VMSA-2018-0002.2

Update 01.26.18: New 1.6.8 release updated to support VMSA-2018-0002.3

Screen Shot 2018-01-18 at 6.34.09 PM.png

Update 01.21.18: Option 3: (Coolest of them all)

This option does not only show what in your VMware environment is impacted but it will also assess the performance impact of both Spectre and Meltdown patches using vRealize Operations Manager (vROPS). We already know the patches will impact the speculative execution capabilities of the processor, which will lead to higher CPU utilization in your cluster due to each OS slower processing times.

The questions that come up then before patching:

  • Will I have enough resources available in my cluster to support these patches?
  • How will my ESXi host resources be impacted?
  • Should I roll out the patches in stages or all at once?

These are hard questions that are not easy to answer, or is it?

If you are using vROPS 6.6.x Advanced or Enterprise, which allows the creation of custom dashboards, then you can download and install the Spectre Meltdown Specific Dashboard kit created by Sunny Dua.  The download is available here.

The Dashboard kit consists of 3 Dashboards:

Screen Shot 2018-01-24 at 10.59.56 AM.png

  • Performance monitoring dashboard
    • Track resources utilization of your environment and will provide valuable information on the impact of patching as it relates to your Clusters, ESXi hosts  VMs.
    • Screen Shot 2018-01-24 at 11.59.56 AM.png
  • VM Patching dashboard
    • Provides views showing which VMs are running idle and can potentially be patched first since it should not have a large overall impact on performance.  Evaluate the resource utilization with the performance monitoring dashboard after the idle VM’s are upgraded, and then make a decision to continue patching or first add additional resources to the cluster.
    • Screen Shot 2018-01-24 at 11.11.41 AM.png
  • vSphere Patching dashboard
    • Shows the ESXi hosts that have been patched and also affected by Intel Sighting.
    • Shows the ESXi hosts that still needs to be patched.
    • Show the Virtual Machines that required Hardware versions upgrade since the recommended version is 9 or higher.
    • I recommend keeping an eye on VMware’s advisory site since this problem is still ongoing and the build numbers will change as new patches are released.  This will then required that you make a manual update in the filters of this dashboard
    • Screen Shot 2018-01-24 at 11.51.08 AM.png

The Performance monitoring dashboard can also be accomplished by just using the default dashboards available in vROPS standard, which means you can download the evaluation version and have that piece of mind that you can track the performance impact while going through these tough times.

Links:

https://communities.vmware.com/message/2738226#2738226

https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html

https://kb.vmware.com/s/article/2143832?r=2&Quarterback.validateRoute=1&KM_Utility.getArticleData=1&KM_Utility.getGUser=1&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1

Free VMware related eBooks available!

Lately, we have seen a lot of excellent guys releasing their valuable books for free! The time and effort they put into these must be crazy, and we thank them for it.  I highly recommend you pick them up and give them a read!

vSphere HA Deep Dive 6.x:

This book was made available some time ago but still relevant, and all the thanks must go to Duncan Epping. You can download it here

vSAN Essentials:

Lots of thanks and praises must go to Cormac Hogan and Duncan Epping for making their Essential Virtual SAN (vSAN) book available for free. You can download it here

vSphere 6.5 Host Resources Deep Dive:

Lots of thanks must go to Frank Denneman and Niels Hagoort for writing this book which was a big hit at VMworld 2017. The book has been made available for free by Rubrik and VMUG, which you can download it here

NSX:

If you ever want to learn more about use cases for NSX then these PDF documents are a must read.

VMware NSX Micro-segmentation Day 1 by Wade Holmes

VMware NSX Micro-segmentation Day 2 by Geoff Wilmington

Operationalizing VMware NSX by Kevin Lees

Automating NSX for vSphere with PowerNSX by Anthony Burke

 

VMware Cloud on AWS

At the recent AWS re:invent conference in Las Vegas, VMware announced a bunch of new features for VMC on AWS.   Here is a complete list of the new features with some already being available and others in preview, which means they might not apply to all customers or AWS regions:

  • VMware site recovery service
    • This new service provides a great use case for an end-to-end DR solution, which simplifies DR operations, faster time-to-protect and removes the requirements for a second data center.
    • Build on top of VMware Site Recovery Manager with vSphere Replication, the service protects workloads between an on-premises data center and VMC on AWS, as well as between different instances of VMC on AWS.
  • 1 and 3-year subscriptions
    • Provide significant cost savings
    • Additional cost savings available base on the number of eligible on-premise product license you own (Hybrid loyalty program)
  • VMware Hybrid Cloud Extension (Preview)
    • In short, this is an add-on SaaS offering which will provide large-scale migration between your on-premise environment running vSphere 5.0+ and VMC on AWS.
    • Provides built-in high-performance layer 2 extensions so you will be able to keep the same networks, IP addresses, and routing policies in place during migration.
  • Layer 2 VPN (Preview)
    • Extending Layer 2 networks from an on-premise data center to VMC on AWS, which allows you to migrate VMs to your cloud SDDC without having to change their IP addresses.
    • Only one Layer 2 VPN is supported per cloud SDDC
    • Hybrid Linked Mode is optional for configuring Layer 2 VPN but is required for cold migration and migration with vMotion between your on-premises data center and cloud SDDC.
    • In your on-premise data center, you can use NSX or configure a Standalone Edge.
  • L3 VPN Generic Download (Preview)
    • This will reduce configuration issues with IPsec deployments since you will be able to download a generic configuration after VPN is configured, which provides all the parameters that need to be set on remote VPN device.
  • AWS Direct Connect
    • High speed, reliable and private network connectivity to an on-premise data center.
    • Single or Multiple DX links option available.
    • While connecting to an SDDC, customers can choose a Private VIF, Public VIF, or both VIF options.
      • Private VIF – carry vMotion and ESXi management traffic
      • Public VIF – optional, and used to establish VPN tunnel and carry management appliance and workload VM traffic.
    • VMC on AWS scale
      • Supports 32 host clusters
      • Multiple SDDC per organization
      • 10 Clusters per SDDC (future)
    • VMC on AWS regions
      • New region US East (N. Virginia)
    • Support for Wavefront by VMware
      • Collects data from application metrics collectors (Java, Ruby, Python, and more) as well as service metrics collectors (MySQL, Pivotal, Kubernetes, AWS, and more)
      • Allows customer to visualize and troubleshooting applications as well as receive alerts.
    • Scripting support
      • API
        • You can use NSX APIs and Power CLI for the Day0 and Day2 automation activities.
      • PowerCLI (preview)
        • A new module has been added since PowerCLI 6.5.4, a which enables the automation and scripting of VMware Cloud on AWS features
      • AWS SDKs (preview)
        • Existing vSphere Automation SDKs for both Python and Java will include functionality for access to VMC on AWS
      • Datacenter CLI (preview)
      • VMC on AWS API is available via a multi-platform simple command line interface
    • AWS service access enhancements
      • You have the choice to access S3 buckets over the internet or over the AWS Connected VPC.
    • VM template support in MVP
      • You can now add VM templates to Content Library, as well as delete and deploy them
    • Live migrations!! (This is a biggy, but still in preview)
      • Live vSphere vMotion will be supported over L2VPN and Direct Connect
      • Need to setup Hybrid Linked Mode (HLM) and L2VPN for this to work
    • vCenter HLM
      • Hybrid link mode sounds similar to enhanced linked mode but differs in requirements, how they work, and what problem each solves. William wrote a great blog describing the differences.
      • Supports vCenter Servers with an embedded or external PSC.
      • Support a single on-premise vCenter Server or multiple on-premise vCenter Servers that are joined to the same SSO domain.
    • External Storage access from inside Guest VM
      • NFS, SMB and iSCSI storage protocols are validated over following networks:
        • AWS Elastic Network Interface (ENI)
        • VMware Cloud on AWS Compute Gateway (CGW)
        • VMware Cloud on AWS Internet Gateway (IGW)

Upgrade your vCenter 6.5 HA environment

As discussed in my previous post here, you can easily setup vCenter HA to provide a decent (not the best and hopefully this will improve) RTO of around 4 minutes for a fail over of your vCenter server.

So now that you have vCenter HA configure, how do you patch or upgrade this environment.  In a single vCenter Server instance the upgrade is really straight forward.

  • Login to the VAMI
  • Before starting the upgrade, take a File based Backup of the vCSA, using the backup utility in the VAMI.
  • Select Update
  • Select Check Updates -> Check Repository (if you have internet access)
    • Otherwise download the software and mount the ISO to the CD/DVD drive.
  • View Available Updates
  • Screen Shot 2017-10-10 at 10.30.25 AM.png
  • Select Install Updates -> Install All Updates

 

For a vCenter HA the steps are bit more complicate since we will use the software-packages utility from the appliance shell which requires us to SSH into the three nodes in a sequence and use manual failover so that we always patch the non-active node.  Below are my quick step by step notes for the upgrade process:

  • There are multiple ways to use the software-package utility:
    • Use the default repository
    • Use a local repository by attaching the ISO to the vCenter Server appliance.
    • Use a remote repository by using a custom repository URL that points to a local webserver in your environment to retrieve the file.
  • In my case I downloaded the vCenter Server Appliance patch ISO from “https://my.vmware.com/group/vmware/patch” and attached the file to the CD/DVD drive of the vCSA.
  • Before I start the upgrade I perform the following tasks:
    • Put the vCenter HA cluster in maintenance mode
    • Make sure SSH is enabled in the vCSA VAMI
    • For each node, I open the console and mount the patch ISO to the CD/DVD drive.
    • Take a File based Backup of the vCSA, using the backup utility in the VAMI.
  • Run the upgrade first on the Witness Node
    • First off SSH into the active vCSA node
      • From the active vCSA node, SSH into the witness node and make sure you are in the appliance shell by running:
        • “appliancesh”
        • Run: “software-packages install –iso”
        • Press Enter way to many times
          • Type yes and press Enter
        • When upgrade is completed, reboot the server
          • “shutdown reboot -r patching”
        • Exit the SSH session
  • Now run the upgrade on the Passive Node
    • First off SSH into the active vCSA node
      • From the active vCSA node, SSH into the passive node and make sure you are in the appliance shell by running:
        • “appliancesh”
        • Run: “software-packages install –iso”
        • Press Enter way to many times
          • Type yes and press Enter
        • When upgrade is completed, reboot the server
          • “shutdown reboot -r patching”
        • Exit the SSH session
  • Log out of the active vCSA node
  • Wait for the nodes to shows status up after reboot.
  • Initiate a vCenter HA failover manually
    • Login to Web client
    • Select the vCenter server -> Configure -> Settings -> vCenter HA
    • Click Initiate failover
    • Click Yes to start the failover
      • Make sure to select performing synchronization first
  • Now lastly run the upgrade on the new Passive Node
    • First off SSH into the new active vCSA node
      • From the active vCSA node, SSH into the passive node and make sure you are in the appliance shell by running:
        • “appliancesh”
        • Run: “software-packages install –iso”
        • Press Enter way to many times
          • Type yes and press Enter
        • When upgrade is completed, reboot the server
          • “shutdown reboot -r patching”
        • Exit the SSH session
  • Optional: Perform another vCenter HA failover manually back to the original vCSA node.
  • Exit vCenter HA maintenance mode
    • Login to Web client
    • Select the vCenter server -> Configure -> Settings -> vCenter HA
    • Click Edit
    • Select “Enable vCenter HA”
    • click OK

Patching of all the vCenter HA nodes should now be completed.

Food for though: This process is quite involved and I wonder, depending on company policy, would it not be easier to just remove vCenter HA, upgrade the single vCSA node through the VAMI and then configure vCenter HA again? It takes way less time and much simpler process. Let me know what you think.

 

 

 

Installing your vCenter Server 6.5 HA environment

vSphere 6.5 finally brought us vCenter High Availability which provides failover/RTO such that users can continue with their work in around 2 minutes through
API clients and in my observations around 5ish minutes through UI clients, awesome right!

vCenter HA is really simple to configure through the Wizard with a basic and advanced option, with the following requirements:

  • vCenter Server 6.5 Appliance with internal or external PSC
  • vCenter Server 6.5 Appliance with no snapshot and CD/DVD drive set to Client Device.
  • Cluster with at least 3 ESXi hosts that this vCSA is both managing and running on will provide Basic installation option
  • If your vCSA is managed by another vCenter Server that is not part of the same SSO domain, or running on a host that is does not manage, you will have to use the Advanced installation option.
  • A port group on ESXi for the private HA network. Optionally, you can have a dedicated vSwitch if network isolation is a requirement.
    • The HA private network must reside on a different subnet than what is used for management.
  • Network latency on the HA network must be less than 10ms.
  • No gateway for the HA network must be specified when configuring the nodes manually during advanced installations.
  • Storage:
    • Datastores for peer and witness cannot resides in a storage cluster.
    • Will get warning if you put nodes in same datastore

When vCenter HA is enabled, a three-node vCenter Server cluster (Active, Passive, and Witness nodes) is deployed, which consists of one IP address for management and three private IP addresses for the HA nodes.

  1. Open vCenter Server 6.5 web client (HTML5 client does not yet support vCenter HA).
  2. Select the vCenter server in Navigator
  3. Select Configuration -> Settings -> vCenter HA
  4. Screen Shot 2017-10-26 at 9.30.46 AM.png
  5. Click Configure

Continue reading

vCenter Server Web Client crash with latest Flash 27.0.0.170

Within the last couple of days Google ran an automatic update of Flash, which is causing havoc in the all-things-flash world.

Since Google manages the upgrade of the flash you are kind of at their mercy on this one, until all applications finally learn to stay as far away from flash as possible! Can only hope!

You can however disable the automatic update feature for flash by setting the registry HKLM\SOFTWARE\Policies\Google\Update\AutoUpdateCheckPeriodMinutes to 0.

My fix was pretty straight forward and I only had to delete the latest release folder from the path “%LocalAppData%\Google\Chrome\User Data\PepperFlash” (windows) or “~/Library/Application Support/Google/Chrome/PepperFlash” (MAC).

In my case the older version was still available in the same folder so I only had to delete the latest and restart google but if do not have the older version you can download it here.

From other users responses it seems that the version 27.0.0.130 and older still works but newer version will crash on both your Google Chrome and Firefox browsers.

Update:  VMware has release a KB here

 

 

Migration of vCenter Server using VMware Migration Assistant

The migration assistant can tackle a couple of different vCenter Server configurations:

  • vCenter Server 5.5 or 6.0 with an embedded vCenter Single Sign-On instance on Windows
  • vCenter Server 5.5 instance on Windows with external SSO
  • vCenter server 6.0 instance on Windows with external PSC

Depending on your current configuration, the migration process will change based using an internal or external SSO/PSC and where VUM is installed.

  • If VUM is installed on a separate Windows server, other that your Windows vCenter Server instance then you MUST to run the migration assistant on this server FIRST!  The VMware Migration Assistant facilitates the migration of the Update Manager server and database to the vCenter Server Appliance 6.5.
  • If embedded SSO/PSC, then you run the Migration Assistant on the source vCenter Server
  • If external SSO/PSC, then you run the Migration Assistant on the source SSO/PSC first and then run the Migration Assistant on the source vCenter Server
    • During the migration process, make sure to leave the migration windows open

You run the VMware Migration Assistant on your source vCenter Server, SSO/PSC or VUM it will perform the following tasks:

  • Discover the source deployment type.
  • Run pre-checks on the source.
  • Report errors that must be addressed before starting the migration.
  • Provide information for the next steps in the migration process.

Here are my step by step instructions for vCenter Server 5.5 with embedded SSO, no VUM:

  1. Download and Mount the vCenter Server Appliance Installer ISO file to the Windows vCenter Server.
  2. Start the migration assistant on the source, depending on your environment configuration
    1. Open the installer folder on CD/DVD drive and copy the  “migration-assistant” folder to your source windows server running the vCenter Server
    2. Open the copied folder on local drive and run “VMware-migration-assistant.exe”
      1. This will run the pre-update checklist and prompt if anything needs to be fixed.
      2. Make sure this window stays open during the whole migration process until completion.
  3. Launch vCenter server appliance UI installer
    1. Click Migrate
    2. Click Next on Introduction
    3. Accept EULA and click Next
    4. Connect to the source server
      1. Enter Windows server FQDN or IP address
      2. Enter migration assistant port number
        • If you are changing networks and have a firewall in place then pick a port that is open and available.
      3. SSO username
        • administrator@vsphere.local
      4. SSO Password
      5. Click Next
    5. Accept Thumbprint
    6. Appliance deployment target
      1. Enter ESXi or vCenter Server name for targer
      2. HTTPS port
      3. Username
      4. Password
      5. Click Next
    7. Accept Thumbprint
    8. Select Folder to place the new vCSA
    9. Select the compute resource
    10. Target appliance VM
      1. Enter VM name
      2. Enter new root password
    11. Select deployment size
    12. Select datastore
    13. Configure the temporary network identity which is used to copy the data, this will be removed after migration is completed since the target vCSA server will get the network identity of the source vCenter server.
      1. Pick network port group
      2. Enter temp IP address
      3. Enter subnet
      4. Enter GW
      5. Enter DNS servers
      6. Click Next
    14. Verify detailed entered are correct!
    15. Click Finish
    16. Wait for the migration to complete.
      • If the migration fails for any reason your can easily roll back by powering off the target vCSA and then just power on the source vCenter Server and its components.
  4. Now for stage two, which copies the data from the source vCenter to new vCSCA.
    1. On Introduction click next
    2. Connect to Source vCenter should complete automatically
    3. Join AD
      • Add username and password for user that can add computer to domain
    4. Select the best migration data options with the following 3 being available. (pretty cool that it provides the size of the migration for each option as well)
      • Just the configuration
      • Configuration, Events and Tasks
      • Configuration, Events, Tasks and Performance metrics
    5. Select if you want to join CEIP and click Next
    6. Review your settings, check the box to verify you have a backup of the vCenter server as well as the database.  Click Next
    7. Click OK on prompt that warns you that the source vCenter Server will be shut down once the network configuration is enabled.
    8. Wait for the data migration to complete.

When successful, your source vCenter Server should now be shut down and brand new shiny vCenter Server appliance should be started in its place. The best part of all you are also upgraded to vCenter Server 6.5! awesome.