Using vRealize Suite LifeCycle Manager to deploy an Enterprise Distributed vRealize Automation environment.

vRealize Suite LifeCycle Manager (vRSLM) has now been around for a while and if you are a vRealize or vCloud Suite license holder this is definitely a product that should be part of our VMware portfolio. I am a bit backward because in my last post is showed how to upgrade your vRA environment using vRSLM and only now will I show how to actually install vRA which actually just comes out of necessity because one of my colleagues accidentally delete all my lab servers ūüôā

For this post, I am using the latest vRSLM 1.3 and will be deploying a distributed vRA 7.4.

  • jvra01 – vRA appliance with embedded vRO (recommended design to use embedded instead of external vRO since 7.3)
  • jvra02 – vRA appliance with embedded vRO
  • jvraweb01 – vRA IaaS Web
  • jvraweb02 –¬†vRA IaaS Web
  • jvramgr01 – vRA IaaS Manager
  • jvramgr02-¬†vRA IaaS Manager

Since vSSLM automates and simplifies the deployment of your VMware SDDC stack, most of your time will be spent on prerequisites, so let’s start with that.

vRA prerequisites:

  • Manually deploy 4 x vRA Iaas Windows Servers in your vCenter Server environment.
    • Make sure they are added to the domain and DNS and NTP is working.
    • Disable UAC on all Windows servers. Make sure to reboot if you have to disable this.
    • Make sure that IPv6 is disabled on all Windows servers
    • Add the windows service account as part of User Rights Assignment under Local Security Policies for Log on as a Service and Log on as a batch job on all windows machines.
    • Verify the minimum resource requirements is set on all Windows servers.¬† Set to at least 8GB.
  • SQL Database
    • Make sure the domain user has added the SQL server to the domain
    • Make sure the domain user is added as part of the SQL DB user Logins list with the sysadmin privilege
  • ¬†Load Balancer
    • Make sure that the second member of each pool in the vRealize Automation load balancer is disabled.

There are also some scripts available to download to verify the prerequisites when you run the precheck for the creation of the vRA environment so this can be done later as well.

vRSLM prerequisites:

  • Ensure that the vRSLCM appliance has correct FQDN configured
    • Command for correcting the hostname is “/opt/vmware/share/vami/vami_set_hostname <hostname>”
    • After setting the correct hostname, verify by using the command “hostname -f” or from 1.3 version of LCM, we can also verify from the settings page.
  • Under vRSLM settings:
    • Register with My VMware to access licenses, download Product Binaries, and consume Marketplace content.
    • Download the vRealize Automation 7.4.0 product
      • If you already have the OVA downloaded then you can import it under the Product binaries tab.
    • Verify that you have vRealize Automation binaries status as completed.
    • If you using a self-signed certificated in your environment (not recommended), then create a self-signed wildcard certificate for vRealize Suite product deployments.
      • Best is to generate a single SAN certificate with all the product or management virtual host names or a wildcard certificate and provide this certificate when you create the environment for the first time. This ensures support for post provisioning actions such as Add Products and Scale Out.
    • Configure NTP Servers for deploying products in environments
  • Under Data Centers
    • Create a Data Center with an associated location.
    • Add the vCenter Server where the vRA environment will be deployed to.
      • Make sure the data collection is successful.

vRealize Automation deployment steps:

  • First off, we need to create an environment for vRA
  • Login to vRSLM
  • Screen Shot 2018-08-03 at 2.31.55 PM.png
  • On the Getting started page select Create Environment
  • Screen Shot 2018-08-03 at 2.34.53 PM.png
    • Select Data Center
    • Select Environment Type
    • Enter an Environment name
    • Enter the Administrator email address
    • Enter a default password which is for both root on the appliance and admin account if applicable on the product being deployed
    • Verify the entered password
    • Disable CEIP if you want
    • Click Next
  • Screen Shot 2018-08-03 at 2.38.42 PM.png
    • In the vRealize Automation Product window, check the selection box in top right-hand corner.
    • Make sure it is set to New install
    • Verify version is 7.4.0
    • Under deployment, select your deployment size¬†with HA
    • Screen Shot 2018-08-06 at 5.20.03 PM.png
      • If you pick Tiny with HA, it will not automatically add your secondary components, and you have to do so manually.¬† Best is to pick Medium with HA.
    • Select Next
  • Scroll to the bottom and accept EULA
  • Click Next
  • License Details
  • Screen Shot 2018-08-03 at 2.44.31 PM.png
    • Enter your license key for vRA, if you are using vRSLM you probably have a vRealize or vCloud Suite license to add.
    • Click Next
  • Infrastructure Details
  • Screen Shot 2018-08-03 at 2.52.27 PM.png
    • Select the vCenter Server enter in the vRSLM data center
    • Select the cluster
    • Select the Network where appliances will be deployed on
    • Select the Datastore¬†where appliances will be deployed on
    • Select disk format
    • Select Applicable Time sync mode with recommendation NTP servers.
      • Select the NTP servers created in settings
    • Click Next
  • Network details
  • Screen Shot 2018-08-03 at 3.00.02 PM.png
    • Enter the default gateway
    • Enter the domain name
    • Enter the domain search path
    • Enter the domain name servers, with comma separated
    • Enter netmask
    • Click Next
  • Certificate details
  • Screen Shot 2018-08-03 at 3.02.38 PM.png
    • Under Manage Certificate select Import Certificate
    • Enter the Passphrase for your wildcard certificate
    • Enter the Private key
    • Enter the Certificate chain
      • This should include the certificate, intermediate and root
  • Product details
  • Screen Shot 2018-08-06 at 5.22.43 PM.png
    • Under Product Properties
      • Enter the Windows service account which will be used for installation and has administrator access on the server.
      • Enter the Windows service account password
      • Select NTP servers
      • Select Yes for Configure Cluster Virtual IPs
      • Select Cluster Virtual IPs
      • Screen Shot 2018-08-03 at 3.36.15 PM.png
        • Enter vRA appliance hostname and IP address
        • Enter IaaS web hostname and IP address
        • Enter Iaas Manager hostname and IP address
    • Under Components
      • Select Primary vRealize Automation Server
        • Enter the Primary vRA VM name
        • Enter the vRA Hostname
        • Enter the vRA IP address
        • If you have some custom configuration for vRA then select the Advanced settings button.
      • Select vRealize Automation Secondary Server
        • Enter the Primary vRA VM name
        • Enter the vRA Hostname
        • Enter the vRA IP address
        • If you have some custom configuration for vRA then select the Advanced settings button.
      • Select Database
        • Enter hostname
        • Enter IP address
        • If you require custom configuration for the database then select the Advanced settings button
      • Select IaaS Web
        • Enter Web hostname
        • Enter IP address
        • If you have some custom configuration for vRA then select the Advanced settings button
      • Select IaaS Manager Active
        • Enter Manager hostname
        • Enter DEM Orchestrator name
        • Enter IP address
        • If you have some custom configuration for vRA then select the Advanced settings button.
      • Select IaaS Manager Passive
        • Enter Manager hostname
        • Enter DEM Orchestrator name
        • Enter IP address
        • If you have some custom configuration for vRA then select the Advanced settings button.
      • Select IaaS Dem Worker
        • Enter Web hostname
        • Enter IP address
        • If you have some custom configuration for vRA then select the Advanced settings button
      • Select vSphere Proxy Agent
        • Enter Web hostname
        • Enter IP address
        • Select Advanced configuration
          • Change the Agent name to be something more descriptive that you will remember because you have to enter the same name when you define the endpoint in vRA
      • If you want to add additional components you can do so at the top of the Product Details.
        • Screen Shot 2018-08-06 at 5.24.50 PM.png
    • Click Next
  • ¬†PreCheck Details
  • Screen Shot 2018-08-03 at 4.07.05 PM.png
    • Click Run PreCheck
    • If error or warning appears, follow the instructions from the required actions. Run the pre-check again to verify fixes.¬† I have performed the installation with the older releases and since v1.3 the preCheck has been greatly improved.
    • Screen Shot 2018-08-06 at 11.27.03 AM.png
    • If the precheck validation comes back successful, Click Next
    • Request Summary:
      • Before clicking submit
        • Create Snapshots of your Windows IaaS servers.
        • Review your configuration and scroll all the way to the bottom.
        • Click on Download Configuration.
          • This will save your configuration settings, which are very useful if for some or other reason you have to reinstall.
    • Click Submit
  • Installation Status
    • Click on requests
    • Screen Shot 2018-08-06 at 10.33.48 AM.png
    • Click on the status “In Progress”
    • Screen Shot 2018-08-06 at 10.36.54 AM.png
    • Here you can follow the process
    • Screen Shot 2018-08-06 at 10.37.43 AM.png
    • If you want to know what the current workflow is that is running, then you can click on the blue dot.
  • Once successfully completed, and this can take up to 2 hours, you can view vRA deployment under Environments
    • Screen Shot 2018-08-06 at 4.55.35 PM.png
    • Click View Details
    • Screen Shot 2018-08-06 at 4.55.51 PM.png
    • Click View Details again
    • Here you will find all the details regarding your existing environment.
  • Now to really make full use of vRSLM, and complete you full SDDC stack environment, you can add additional products to your environment like:
    • vRBC
    • vROPS
    • vRLI (since 1.3)
    • vRNI

Step by Step upgrade of distributed vRealize Automation 7.2 with external vRO to 7.4

As with most of my other blog posts, I am just providing a step by step guide for quick reference.  Please refer to the documentation here for detailed information and please read the vRealize Automation 7.4 Release Notes known issues section which is updated regularly and helps you to be better prepare for the upgrade.

My environment consists of a distributed vRealize Automation running version 7.2 with an¬†external clustered vRealize Orchestrator,¬†which I am upgrading and not migrating to 7.4 Build 8182598.¬† This will be a similar process if you have vRA 7.1 and greater.¬† If you have an older version, refer to VMware’s documentation here.

The in-place upgrade process for the distributed vRA environment happens in 3 stages in the following order:

  1. vRealize Automation appliances
  2. IaaS Web server
  3. vRealize Orchestrator

Pre-requisites before we start:

  1. Make sure all VMware products are compatible with vRA’s current and new release by consulting the Product Interoperability Matrix.
  2. Verify enough storage space on servers
    • At least 5GB on IaaS, SQL and Model Manager
    • At least 5 GB on the root partition of vRA¬†appliance

    • 5 GB on the /storage/db partition for the master vRA appliance

    • 5 GB on the root partition for each replica virtual appliance

  3. Verify that MSDTC is enabled on all vRA and associated SQL servers.
    • Check that the service “Distributed Transaction Coordinator” is running.
  4. The primary IaaS Website node (Model Manager data is installed) must have JAVA SE Runtime Environment 8, 64 bits, update 161 or later installed, and also verify JAVA_HOME environment variable is set correctly after the upgrade.
  5. If using embedded Postgres DB in a distributed vRA environment
    • On master vRA node, navigate to¬†/var/vmware/vpostgres/current/pgdata/
    • Close any opened files in the pgdata directory and remove any files with a .swp suffix
    • Verify the correct ownership of all files in this directories: postgres:users
  6. In a distributed vRA environment, change Postgres synchronous replication to async.
    • Click vRA Settings > Database.
    • Click Async Mode and wait until the action completes.
    • Verify that all nodes in the Sync State column display Async status
    • I have only a master and replica so I am already async but just FYI
  7. In vRA tenants verify the following
    • Make sure that no custom properties have spaces in the names.
    • All saved and in-progress requests have finished successfully

Additional requirements before we start:

Continue reading

Upgrade vRealize Automation 7.2 to 7.4 using vRealize Suite LifeCycle Manager

VMware’s vRealize Suite of Products are great, and each provides a lot of features and capabilities, and VMware has been working hard on integration between the products. However, these products are very much standalone with no cohesion between them from a lifecycle management perspective.¬† This creates a lot of management overhead to install, upgrade, configure and manage all these products, as well the additional solution extensions.

In comes vRealize Suite LifeCycle Manager (vRSLCM) which is a relatively new product and is available to all customers with a vRealize Suite license. It automates the installation, configuration, and upgrading of the following products:

  • vRealize Automation
  • vRealize Operations Managers
  • vRealize Log Insight
  • vRealize Business for Cloud

In this blog, I am going to provide the steps on how to import an existing distributed Enterprise vRA 7.2 environment and perform the upgrade to 7.4 using vRSLCM 1.2.

Let’s start off with the initial creation of the environment, which does require a lot of information up front, but once you create or import products into the environment at a later time, it will make use of this stored environment information.

  1. Log in to your vRSLCM
  2. Screen Shot 2018-05-21 at 3.46.11 PM.png
  3. Select Create Environments
  4. Screen Shot 2018-05-21 at 3.45.43 PM.png
  5. Enter Environment Data
    1. Data Center (this you should have created during the initial configuration of your vRSLCM environment)
    2. Environment Type
    3. Environment Name
    4. Administrator email
    5. Default root password
    6. Click Next
  6. Create Environment
    1. Screen Shot 2018-05-21 at 3.49.10 PM.png
    2. Check the box for vRealize Automation
    3. Since we already have an environment that we need to import, select the import Radio button.
    4. Click Next
  7. EULA
    1. Scroll down to bottom.
    2. Check the box to accept the terms and conditions.
  8. License
    1. Screen Shot 2018-05-21 at 3.52.11 PM.png
    2. Either pick a vRealize Suite license which will populate from your my.vmware.com account, or enter one manually.
    3. Click Next
  9. Infrastructure Details (This information is used if you deploy new products)
    1. Screen Shot 2018-05-21 at 3.55.06 PM.png
    2. Select vCenter Server where your vRealize Suite products reside in.
    3. Select Cluster
    4. Select Network
    5. Select Datastore
    6. Select preferred Disk format for product deployments.
    7. Click Next
  10. Network (This information is used if you deploy new products)
    1. Screen Shot 2018-05-21 at 3.59.39 PM.png
    2. Enter default gateway of the network where your vRealize Suite products are deployed or will be deployed too.
    3. Enter Domain Name
    4. Enter search path
    5. Enter DNS
    6. Enter Netmask
    7. Click Next
  11. Certificates (I import a wildcard certificate or you can use multi-domain certificate would be a good choice to simplify the process)
    1. Screen Shot 2018-05-21 at 4.04.03 PM.png
    2. Click Next to use the self-signed generated certificate or click the import certificate button to add existing wildcard or SAN certificate.
    3. Click Next
  12. Import (Since we selected import we now get ask questions about our existing environment)
    1. Screen Shot 2018-05-21 at 4.06.18 PM.png
    2. Enter vRA root password
    3. Enter vRA Default Administrator password
    4. Enter Tenant User name.
      1. Selecting the “administrator” user works just fine here.
    5. Enter vRA Primary Node FQDN
    6. Enter IaaS Username.
      1. I used the domain service account assigned to all IaaS servers
    7. ¬†Default vRA Tenant name is select “vsphere.local”
    8. Enter vRA Tenant password
    9. Enter IaaS Password for the domain account.
    10. Select vCenter Server from the drop-down where the vRA server is running on.
    11. Click Next
  13. Review summary
    1. Click Download configuration to save the JSON file for later use.
    2. Click Submit
  14. This will run for a while to configure the environment and import vRA
    1. If it fails, you have a couple of options
      1. Review the requests
        1. Screen Shot 2018-05-21 at 4.19.27 PM.png
        2. Under actions select retry and verify the information that you have entered.
      2. Delete the environment and start over (1.2 provides the ability to specify if you also want to delete the VMs when you delete a fully configured environment, definitely not recommended to do so in most cases!)
    2. If you want to pause the import, you can always come back later and resume\
  15. Verify the vRA product environment
    1. Select Environment tab on the left side
    2. Screen Shot 2018-05-21 at 4.25.18 PM.png
    3. Select View details of the newly created environment
    4. Screen Shot 2018-05-21 at 4.24.32 PM.png
    5. Verify that all the information of your distributed vRA environment is accurate. vRSLCM collects all your VIP names, vRA-, IaaS- and Database Servers as well as where each component resides.
    6. Screen Shot 2018-05-21 at 4.24.21 PM.png

Continue reading

VMware drops the motherload, what’s new?

I just recently got back from a productive VMware Partner Empower Conference and since then I have been trying make some time to grind through the installing and upgrades of 10 new product releases. (Sorry I am a bit behind in getting this blog post publish) Yes, you heard correct 10!   I do hope VMware in the future consider staggering the product release as well as align the product compatibility.

Here is a list of new product updates available:

  1.  vRealize Automation 7.4.0 Release Notes Download
  2.  vRealize Orchestrator Appliance 7.4.0 Release Notes Download
  3. vRealize Code Stream 2.4 Release Notes Download
  4. vRealize Business for Cloud 7.4.0 Release Notes Download
  5. vRealize Operations Manager 6.7.0 Release Notes Download
  6. vRealize Suite Lifecycle Manager 1.2 Release Notes Download
  7.  vRealize Log Insight 4.6.0 Release Notes Download
  8.  vCenter Server 6.7 Release Notes Download
  9.  vSphere ESXi 6.7 Release Notes Download
  10.  vSphere Replication 8.1 Release Notes Download

Now let’s see the awesomeness that each of the new products brings.

vRealize Automation 7.4.0

  • Custom Request Forms Designer has been vastly improved!
  • Deployment of Blueprint from an OVF on AVA. sweet!
  • Improvement in the handling items in an in-progress state. This was a big pet peeve¬†of mine which has finally been addressed!
  • Message board portlet security has been enhanced with a whitelist for URLs
  • Multitenancy in vRealize Orchestrator is great since now we do not have to deploy multi vRO environments for different tenants.

vRealize Orchestrator Appliance 7.4.0

  • Multitenancy in vRO, need we say more!¬† Allows for full isolation of content items and execution and no more need to deploy separate¬†vRO environment for tenants.
  • Web-based Clarity UI update with new monitoring dashboards for workflow runs and metrics to monitor and troubleshoot workflow runs including centralized log views.

vRealize Code Stream 2.4

  • General defect fixes and improvements.

vRealize Business for Cloud 7.4.0

  • Note:
    • If you are running vRBC 6.x.x a direct upgrade is not possible and you must first upgrade to 7.3.1!
    • If you upgrade to 7.4.0, any vRealize Automation VMs that are deleted in the current month before the upgrade will be missing in the current months vRA Report.¬† There is currently no fix and the workaround is to download the vRA¬†Report before you upgrade. KB 2151835
  • Enhancements to the VMC on AWS assessment with FTT and Erasure coding.
    • Screen Shot 2018-04-23 at 2.15.29 PM.png
  • Pricing and Chargeback capabilities enhancements
    • For vCloud Director, storage policy driven¬†pricing is now available.
      • Screen Shot 2018-04-23 at 2.32.41 PM.png
      • Screen Shot 2018-04-23 at 2.32.26 PM.png
    • For vCD,¬†you can define T-Shirt sizes for Pay-as-you-go VMs. This is cool and I¬†wish we can get this in vRA as well for Component Profile!
      • Screen Shot 2018-04-23 at 2.51.14 PM.png
    • Apply differential rate for vCPU and Memory, that is beyond the guaranteed resources
      • Screen Shot 2018-04-23 at 6.45.43 PM.png
    • Apply daily pricing of OS license consumption
      • Screen Shot 2018-04-23 at 7.20.16 PM.png
    • Network pricing enhancements
      • Screen Shot 2018-04-23 at 2.49.25 PM.png
    • vCloud Director usage and charge details through report API
    • Security enhancements

Continue reading

How to use vRealize Network Insight (vRNI) for application dependency mappings

vRNI can be a great tool in your networking and security operations arsenal, with loads of features to support your physical, virtual and cloud environments.

There is already a lot of great material available for vRNI but here are just some of the primary use-cases:

  • Plan Application security and migration
    • Micro-segmentation planning with automatic firewall rules recommendations.
  • Manage and Scale NSX
    • Multiple NSX Managers with proactive detection of misconfiguration errors
  • Optimize and Troubleshoot Virtual and Physical networks
    • Optimize application performs by removing bottlenecks
    • Audit network and security changes over time.

Also, the vRNI feature walkthrough page of VMware is excellent for an introduction

So back to why we are here! When you have a hybrid cloud strategy and have to move applications to the cloud you definitely want to know a couple of things:

  • Which VMs are sitting idle or are over-provisioned on resources?
    • vRealize Operations (vROps) should be your go-to tool to identify all these VMs
  • How much will cost to place my VMs into any of Cloud Solutions available out there, including VMC on AWS?
    • vRealize Business for Cloud should be your go-to tool to provide pricing for different cloud-based solutions on a selected VM/application.
  • If you have multi-tiered applications, do you know the dependencies between the VMs and on which port/s they communicate with?
    • There are a lot of tools available that can provide application dependency mappings, but for this exercise, we are just looking at vRealize Network Insight (vRNI).

Let’s look at the steps to create an application dependency mapping, which is very similar to the steps you will use to create your micro-segmentation firewall rules.

  • Step 1: Select the initial VM that you have identified for the application.
    • Using VRNI powerful search capabilities, type the query “VM where name = ‘vmname.’
    • For the duration, if you have collected information for a while, then select maybe the last 7 days as your time frame
    • Click Search
      • The VM can be selected in different ways like:
        • Path and Topology -> VM
        • Entities -> VM
    • Screen Shot 2018-04-24 at 4.26.28 PM.png
    • This will show information about the VM, click on the VM name.
    • Screen Shot 2018-04-24 at 4.29.37 PM.png
    • Click on Flows in the toolbar
    • Screen Shot 2018-04-24 at 4.30.20 PM.png
    • Review the VM Flows – Allowed and VM Flows – Denied
      • This shows all the flows to and from the selected VM
    • Click on the 3 vertical dots and select “Export as CSV.”
      • This exported document provides columns for all source and destination VMs that are connecting to your selected VM.¬† Use this to start your application dependency mapping by creating an application in vRBC.
        • Screen Shot 2018-04-24 at 4.39.50 PM.png
        • Select Entities -> Applications
          • Click Add Application
          • Enter Application Name
          • Enter Tiers and conditions to identify the VM or IP address
            • Add the VMs that you have identified as Source and Destination VMs in the flows.
          • You can also add more conditions to fine tune the VM select and also add additional Tiers.
          • Select Analyze Flows
          • Click Save
  • Step 2: Select the application, and add any additionally identified entities as the first hop.
    • Screen Shot 2018-04-24 at 4.57.18 PM.png
    • Select Security -> Applications
      • Screen Shot 2018-04-24 at 4.57.56 PM.png
      • Under scope drop-down select Application
      • Select Application name created in step 1
      • For Duration you can select anything but 7 days would be good to cover all different connectivity scenarios that might occur.
      • Click Analyze
    • On the Micro-segmentation view
      • Screen Shot 2018-04-24 at 5.01.34 PM
      • Under “Group By” select VM
        • Under “Also show groups for” select All
      • Under Flow, Types select “All allowed flows.”
      • Screen Shot 2018-04-24 at 5.30.36 PM
      • This will provide you with a presentation of how your application VMs are talking with one another
      • However, more importantly, you will see “other entities,” in grey boxes, which is what we are really interested in:Screen Shot 2018-04-24 at 5.38.54 PM
      • You can also filter based the groups to show all the entities associated with the groups below
        • Virtual
          • If you select virtual, you will be presented with a list of all the VMs that communicate to the applications, and have not yet been identified.
          • Again you export the CSV.
          • Review these VM’s and add them to the application.
        • Physical
          • If you select physical, you will be presented with a list of IP addresses for all the physical servers are you connecting too in your environment.
          • Review these VM’s and add the physical IP address to your application.
        • Shared Virtual
          • If you select Shared Virtual, you will be presented with a list of VMs that are connected to all the VMs in your application.
          • Review these VM’s and add them to the application.
        • Internet
          • If you select Internet, you will be presented with a list of public IP addresses that your application is connecting too.
          • Review these public IP addresses and take note of them
  • Step 3:¬† Manually create your application dependency¬†mapping
    • If you really want to see how deep the rabbit hole goes then repeat step 2.
      • This will provide additional virtual, physical, shared and internet entities, based on the updated application.
    • Unfortunately is no way in vRNI to show a network connectivity diagram of the application like you were able to see in VIN so you would have to create your own Visio, making use of the flow diagram or exported CSVs to identify individual connectivity.

 

This is my own method and not sure if right or wrong, but if anyone has figured out a different or better way, please let me know!

Spectre and Meltdown – How to check your VMware environment for vulnerabilities

Updates added to the blog

Unless you have been on a very long vacation without internet access (The BEST type of vacation!) you should know of the Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) vulnerabilities that affect nearly every computer chip manufactured in the last 20 years.

I am not going to provide any specific details on these vulnerabilities since there are more than enough material already available, which you can read here:

I do however want to provide more detailed information related to VMware specifically, as well as different ways on how you can verify what in your VMware environment is vulnerable to these exploits:

VMware responded to the Speculative Execution security issues with KB 52245, which I highly recommend you read and subscribe to.

Intel and AMD released microcode updates that provide hardware support for branch target injection mitigation, for which VMware released KB 52085. The KB provides instructions on how to enable Hypervisor-Assisted Guest Mitigation, which is required in order to use the new hardware feature within VMs.  The KB also provides manual verification instructions for the following:

  • ESXi – Verify that the microcode included in ESXi patch has been applied
  • VM – Verify that the VM is seeing the new microcode ( VM needs to on HWv9 or newer)

ALERT: VMware also released KB 52345, which rollback the recently issued security patch recommendation (ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG). The rollback is due to customers complaining of¬†unexpected reboot after applying Intel’s initial microcode patch on Intel Haswell and Broadwell processors.

UPDATE 01.24.18: VMware updated KB 52345 to include updated list of all Intel CPUs affected by Intel Sightings

  • VMware provides some manual workarounds for these specific processors that have already been patched.
  • For ESXi hosts that have not yet applied one of the patches, VMware recommends not doing so at this time¬†and using the patches listed in¬†VMSA-2018-0002 instead.

That is a lot of information to take in, and the rollbacks just add complexity to IT teams who are trying to secure their customer’s data.

UPDATE 02.15.18: VMware security advisory for VMware Virtual appliance mitigation available here

UPDATE 03.20.18: VMware provided an update to KB 52085 for patching the vSphere vCenter server to latest 6.5U1g, 6.0U3e, 5.5U3h and Hypervisor to ESXi 6.5: ESXi650-201803401-BG* and ESXi650-201803402-BG**, ESXi 6.0: ESXi600-201803401-BG* and ESXi600-201803402-BG**, ESXi 5.5: ESXi550-201803401-BG* and ESXi550-201803402-BG**.

* = Framework to allow guest OSes to utilize the new speculative-execution control mechanisms

** = Applies the microcode updates

 

Option 1: (The best of the best)

However, to make things a bit easier we have William Lam to the rescue who wrote an excellent script that automates the verification for both the ESXi and Virtual Machines. as well as provide ESXi microcode versions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and performs the following validations

  • Verify that VM’s are running at least HWv9
  • Verify that VM completed a power cycle to see the new CPU features
  • Verify¬†ESXi microcode has been applied
  • Verify that one of the three new CPU features are exposed to the ESXi host.
  • Verify if CPU is affected by Intel Sighting
  • Show the current Microcode version for each ESXi (requires SSH to be enabled)
  • UPDATE 01.24.18: Script was updated to validated the affected CPUs

All the detail regarding the script can be read on virtuallyGhetto here.

Option 2:  (Acceptable, but limited)

Although not nearly as thorough as William’s Script, with RuneCast Analyzer latest¬†1.6.7 you can detect ESXi hosts that are not protected and patched against these vulnerabilities.

Runecast Analyzer enables you to scan and detect the CPU chip vulnerabilities on your VMware infrastructure.  It detects which ESXi hosts are not protected and advise on how to patch them against such security vulnerabilities.  This solution is continuously updated as new guidance from VMware is released.

Currently only supports VMSA-2018-0002.2

Update 01.26.18: New 1.6.8 release updated to support VMSA-2018-0002.3

Screen Shot 2018-01-18 at 6.34.09 PM.png

Update 01.21.18: Option 3: (Coolest of them all)

This option does not only show what in your VMware environment is impacted but it will also assess the performance impact of both Spectre and Meltdown patches using vRealize Operations Manager (vROPS). We already know the patches will impact the speculative execution capabilities of the processor, which will lead to higher CPU utilization in your cluster due to each OS slower processing times.

The questions that come up then before patching:

  • Will I have enough resources available in my cluster to support these patches?
  • How will my ESXi host¬†resources be impacted?
  • Should I roll out the patches in stages or all at once?

These are hard questions that are not easy to answer, or is it?

If you are using vROPS 6.6.x Advanced or Enterprise, which allows the creation of custom dashboards, then you can download and install the Spectre Meltdown Specific Dashboard kit created by Sunny Dua.  The download is available here.

The Dashboard kit consists of 3 Dashboards:

Screen Shot 2018-01-24 at 10.59.56 AM.png

  • Performance monitoring dashboard
    • Track resources utilization of your environment and will provide valuable information on the impact of patching as it relates to your Clusters, ESXi hosts¬† VMs.
    • Screen Shot 2018-01-24 at 11.59.56 AM.png
  • VM Patching dashboard
    • Provides views showing which VMs are running idle and can potentially be patched first since it should not have a large overall impact on performance.¬† Evaluate the resource utilization with the performance monitoring dashboard after the idle VM’s are upgraded, and then make a decision to continue patching or first add additional resources to the cluster.
    • Screen Shot 2018-01-24 at 11.11.41 AM.png
  • vSphere Patching dashboard
    • Shows the ESXi hosts that have been patched and also affected by Intel Sighting.
    • Shows the ESXi hosts that still needs to be patched.
    • Show the Virtual Machines that required Hardware versions upgrade since the recommended version is 9 or higher.
    • I recommend keeping an eye on VMware’s advisory site since this problem is still ongoing and the build numbers will change as new patches are released.¬† This will then required that you make a manual update in the filters of this dashboard
    • Screen Shot 2018-01-24 at 11.51.08 AM.png

The Performance monitoring dashboard can also be accomplished by just using the default dashboards available in vROPS standard, which means you can download the evaluation version and have that piece of mind that you can track the performance impact while going through these tough times.

Links:

https://communities.vmware.com/message/2738226#2738226

https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html

https://kb.vmware.com/s/article/2143832?r=2&Quarterback.validateRoute=1&KM_Utility.getArticleData=1&KM_Utility.getGUser=1&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1

vRealize Network Insight (vRNI) 3.5 upgrade process that works

It is have been almost a year ago since my initial post on upgrading vRealize Network insight to 3.2 and since then there has been couple of new versions released. So time for me to upgrade!

The bad part I found out about the upgrade process is that you have to upgrade each version consecutively meaning I had to upgrade my 3.2 environment to 3.3 (which i am currently on right now) and then next step is to upgrade to 3.4 and following that another upgrade to 3.5.  You cannot skip version upgrades all!  Anyways, not going to comment on that but you see where this can be very time consuming so plan accordingly.

As before there are still two upgrade options available with online, which is handled through the GUI and offline, which is handled through the CLI.¬† I am currently running 3.3 and in the GUI under Settings -> Install it states that my Application is up to date. I did verify through CLI command “show-connectivity-status”¬† that my upgrade connectivity status shows passed and I also have no proxy.¬† Not wanting to open a support ticket I am going to go the manual route, and oh yes if you have a cluster configured, your only option is manual upgrade as well. Sorry!

Firstly we must upgrade the vRNI Platform appliances before we upgrade the Proxy appliances. If you have cluster then you have to start with platform1.¬† VMware’s KB on the manual upgrade process to 3.5 does not do such a good job of showing the exact steps to upgrade so here are mine:

  1. Download the upgrade bundle
  2. Extract the bundle from the downloaded zip file.
  3. Snapshot your vRNI Platform and proxy appliances before upgrade. (always have a backup)
  4. Login to Platform CLI with consoleuser
  5. Change password for the support user
    1. (cli) modify-password support
    2. Enter the password
  6. Use a popular tool like WinSCP to copy the bundle file to the all vRNI appliances
    1. Login with the support user
    2. Copy the bundle file in directory /home/support/
  7. Now we need to use the package-installer command to copy the bundle file to the vRNI VM
    1. package-installer copy –host localhost –user support –path /home/support/VMWare-vRNI.3.4.0.1495004044.upgrade.bundle
    2. Enter password
    3. Verify copied completed
    4. Remember one version at a time so first off have to upgrade from 3.3 to 3.4.
  8. Stop the service
    1. (cli) services stop
  9. Run the upgrade
    1. (cli) package-installer upgrade (3.3 -> 3.4)
    2. (cli) package-installer upgrade –name VMware-vRealize-Network-Insight.3.5.0.1502978926.upgrade.bundle (3.4 -> 3.5)
    3. This could take up to 30 minutes to complete so go have a cup of tea or coffee.
    4. Verify upgrade completed by checking the version
      • (cli) show-version
    5. If the service does not start..
      • (cli) services start
  10. Run step 4 through 9 on all appliances
    1. vRNI Platform appliances first
    2. vRNI Proxy appliances last

After the upgrade from 3.3 to 3.4, the upgrade KB states that a reboot is not necessary, but I found that if you do not perform a reboot you are not able to run the upgrade command “package-installer upgrade –name VMware-vRealize-Network-Insight.3.5.0.1502978926.upgrade.bundle”.¬† The –name parameter is not recognizable.

Note:

Do not copy/paste the commands in the KB since the filename is different that what you actually download “VMWare” and this make your upgrade fail.

Links: