vRealize Orchestrator

Upgrade VMware vRA to 7.5 and migrate external vRO to embedded vRO 7.5

/ johannstander / 2 Comments

It has been a while since my last post, (just had too much going on) but I have been putting it off for way too long and I finally upgraded my vRA lab to 7.5.  Here are my notes.

My distributed enterprise vRA 7.4 environment consists of the following components:

  • vRA VIP
    • 2 x vRA appliances
  • vRA IaaS Manager VIP
    • 2 x Windows vRA IaaS Manager Service servers
  • vRA IaaS Web VIP
    • 2 x Windows vRA IaaS Web servers
  • 2 x Windows vRA DEM + Agent servers
  • vRO VIP
    • 2 x external vRO appliances
  • External SQL database for vRA and vRO
  • Running SovLabs extensibility software

There are 2 options available to get to the desired state with either an in-place upgrade of your existing vRA environment or to build out a new greenfield vRA and migrate your data over (VMware calls this a side-by-side upgrade).

If you are currently running 6.2.0 – 6.2.4 or 7.0.x, or have vCloud Director or vCloud Air endpoints you have to migrate!

Always before upgrading, make sure you have successful backups of all your nodes and while you’re at it also take snapshots of all the servers and backup your vRA and vRO database!  You can never be too careful, ever! The upgrade steps for vRA are the same as what I have blogged about here.  For this exercise, I am performing an in-place upgrade of vRA from 7.4 to 7.5, so please review the documentation if you upgrading from 6.2.5.

  • Also, verify that all appliances and servers that are part of your deployment meet the system requirements for vRA 7.5 and also consult the VMware Product Interoperability Matrix about compatibility with other VMware products.
  • I also have SovLabs plugins installed so make sure to upgrade SovLabs to a vRA 7.5 compatible version.  At the time of the post, I upgraded to 2018.3.1. Upgrade steps for SovLabs can be found here.

Continue reading →

Step by Step upgrade of distributed vRealize Automation 7.2 with external vRO to 7.4

/ johannstander / Leave a comment

As with most of my other blog posts, I am just providing a step by step guide for quick reference.  Please refer to the documentation here for detailed information and please read the vRealize Automation 7.4 Release Notes known issues section which is updated regularly and helps you to be better prepare for the upgrade.

My environment consists of a distributed vRealize Automation running version 7.2 with an external clustered vRealize Orchestrator, which I am upgrading and not migrating to 7.4 Build 8182598.  This will be a similar process if you have vRA 7.1 and greater.  If you have an older version, refer to VMware’s documentation here.

The in-place upgrade process for the distributed vRA environment happens in 3 stages in the following order:

  1. vRealize Automation appliances
  2. IaaS Web server
  3. vRealize Orchestrator

Pre-requisites before we start:

  1. Make sure all VMware products are compatible with vRA’s current and new release by consulting the Product Interoperability Matrix.
  2. Verify enough storage space on servers
    • At least 5GB on IaaS, SQL and Model Manager

    • At least 5 GB on the root partition of vRA appliance

    • 5 GB on the /storage/db partition for the master vRA appliance

    • 5 GB on the root partition for each replica virtual appliance

  3. Verify that MSDTC is enabled on all vRA and associated SQL servers.
    • Check that the service “Distributed Transaction Coordinator” is running.
  4. The primary IaaS Website node (Model Manager data is installed) must have JAVA SE Runtime Environment 8, 64 bits, update 161 or later installed, and also verify JAVA_HOME environment variable is set correctly after the upgrade.
  5. If using embedded Postgres DB in a distributed vRA environment
    • On master vRA node, navigate to /var/vmware/vpostgres/current/pgdata/
    • Close any opened files in the pgdata directory and remove any files with a .swp suffix
    • Verify the correct ownership of all files in this directories: postgres:users
  6. In a distributed vRA environment, change Postgres synchronous replication to async.
    • Click vRA Settings > Database.
    • Click Async Mode and wait until the action completes.
    • Verify that all nodes in the Sync State column display Async status
    • I have only a master and replica so I am already async but just FYI
  7. In vRA tenants verify the following
    • Make sure that no custom properties have spaces in the names.
    • All saved and in-progress requests have finished successfully

Additional requirements before we start:

Continue reading →

SovLabs: Upgrading your software

/ johannstander / Leave a comment

SovLabs isn’t just a vRA plugin, it’s enterprise software that extends the capabilities of your vRealize Automation environment providing you with that end-to-end solution you have been craving for.  As with any other enterprise software they periodically provide new patches and releases and with SoLabs that is no different.

The new 2017.3.x was released in August and provides some awesome new modules:

  • Men & Mice DNS and IPAM
  • SolarWinds DNS
  • Backup as a Service
    • Automate policy-driven backups and provide self-service VM and file-level recovery for –
    • Cohesity
    • Rubrik
    • Veeam
  • SovLabs VM tagging
    • Drive rich metadata using VM tags and categories
  • SovLabs Property Toolkit
    • Manage your existing custom properties on VMs with the SovLabs Template Engine
  • ServiceNow Support for Jakarta
  • Puppet support for 2017.1
  • VMware Tools connection
    • Connect to Windows/Linux servers can now be done through VMware Tools which removes the requirement for WinRM, CygwinSSH or WinSSHD to be installed.  This is huge!
  • As a customer you can sign up under the self-service portal and view the detailed release notes here:

So how do we go about upgrading SovLabs to the latest version?

Step by step guide to upgrading from 2017.2.x to 2017.3.x.  (there are some additional steps if you are upgrading from <= 2017.1.x so please contact SovLabs support) 

  1. First off we want to create a backup of the vRO package
    1. Login to vRO Client
    2. Click Design
    3. Click on the package tab
    4. Click on the package icon on right hand side menu bar
    5. Enter name “com.sovlabs.backup.resources”
    6. Edit the newly create package, click on the pencil icon on the right hand side menu bar
    7. Click the Resources tab
    8. Click the Folder + icon
    9. Expand the Library folder,  select the SovLabs folder
    10. Click on the Select button
    11. Once loaded, click save and close
    12. Right click the saved package and click export package
      1. Create a folder called sovlabs under downloads
      2. leave the rest of settings as default
    13. Save to your local system
    14. Now, lets save the old SovLabs Plugin:
      1. Use WinSCP and login as root to vRO appliance
      2. Go to directory /var/lib/vco/app-server/plugins
      3. Save the o11nplugin-sovlabs.dar to your local file system in same sovlabs folder created earlier under download.s
    15. Done!
  2. We need to update the vRO Heap size
    1. If you have done this before then you can skip this step but this is needed to install the larger sized SovLabs module file into vRO otherwise the appliance might run out of memory during install/upgrade.
    2. Remember if you a vRO cluster, then you have to perform the steps on both server
    3. SSH into vRO appliance with user root
    4. Run # vi /var/lib/vco/configuration/bin/setenv.sh
    5. Find the #MEM_OPTS section
    6. Replace the -Xmx512m \ with -Xmx768m \
    7. Save the file
  3. Delete all SovLabs license keys
    1. Login to vRA tenant
    2. Click on Items tab -> SovLabs vRA Extensibility modules -> SovLabs License
    3. For each SovLabs License item listed
      1. Select Actions -> Delete License
  4. Download the SovLabs plugin
    1. Talk to SovLabs support about getting the software downloaded.
  5. Install the plugin into vRO appliance
    1. Login to controlcenter
      1. https://<vroserver&gt;:8283/vco-controlcenter
    2. Select Plug-Ins -> Manage Plug-ins
    3. Click Browse
    4. Select the plugin
    5. Accept EULA
    6. Click on Install
    7. Accept the EULA
    8. Restart the vRO server
      1. On the Home page, click on the Startup Options icon
      2. Click on Restart
      3. Wait for vRO to restart successfully
    9. Log back in to the vRO configuration page
    10. Click on the Manage Plug-Ins icon
    11. Verify that the installed plugin is listed among the vRO plugins
    12. Now if you have a clustered vRO 7.2 and above, then the plugin should sync but I have seen some problems with 7.2 so follow these steps
      1. Perform a full reboot on primary so that the pending and active config fingerprint ID match.
      2. Then push the config to the other standby node
      3. It will need to rebooted which it often will not do so make sure you perform this step yourself.
      4. Verify that Synchronization state shows synchronized and verify the version of the plugin on both active and standby nodes.
  6. Login to the vRO Client and run the configuration
    1. Click on Design mode
    2. Click on WorkFlow tab
    3. Right click vRO workflow, “SovLabs/Configuration/SovLabs Configuration”
    4. Select Start Workflow
    5. The SovLabs Configuration workflow only needs to be run on one vRO in a clustered environment
      1. Select yes to accept the EULA
      2. Click Next
      3. Select the appropriate tenant and business group
      4. Create SovLabs vRA Catalog Service? = No
      5. Publish License Content? = No
      6. Click Next
      7. Upgrade existing SovLabs vRA content? = Yes
      8. Click Next
      9. Install or Update SovLabs workflow subscriptions (vRA7.x)? = Yes
        1. *Enables vRA to call vRO during machine lifecycles
      10. Click Submit
      11. Verify that the SovLabs Configuration workflow completed successfully
  7. Lastly, let’s verify the SovLabs Plugin in vRA
    1. Select Catalog tab
    2. Verify that Add license -> SovLabs Modules catalog exists
  8. Now lets install the new license key for 2017.3.x
    1. This process has also been drastically simplified with a single license key which will license all modules, where previously this was done one at a time.
    2. Select Catalog tab -> SovLabs vRA Extensibility Modules -> Add license – SovLabs Modules
    3. Copy the text from license file and paste into field
    4. Click Submit
    5. Verify all catalog tab -> SovLabs vRA Extensibility Modules that all catalogs are available.
  9. If you ever need to roll back then follow the steps in the document provided by SovLabs:
    1. https://s3.amazonaws.com/docs.sovlabs.com/vRA7x/guides/SovLabs_BackupRestore-vROPackage.pdf

 

 

 

 

vRA & SovLabs: Installing the plugin modules

/ johannstander / Leave a comment

As mentioned in my initial blog post on SovLabs, you would have to create custom code in vRO to support the automation of many of the additional steps like custom naming, IPAM, DNS, AD, Load Balancer, but with SovLabs software modules this is really easy. Below are my notes for the prerequisites and the initial installation of the SovLabs modules.

Some prerequisites needs to be completed before installing the plugin:

  1. Configure the vRO service accounting in vRA
    1. Login to the root vRA tenant
    2. Click Administration -> Users & Groups > Custom Groups
    3. Create a Custom Group
    4. Enter a group name and description.
      1. DO NOT put spaces in the group name.
    5. Select the following roles listed in the Add Roles to this Group box
      1.  Tenant Administrator
      2. XaaS Architect
      3. Screen Shot 2017-04-13 at 2.00.41 PM.png
    6. Click Next
    7. Type in the vRO service account or vRO service account group
      1. If this account does not appear make sure it is sync’d.
    8. Click Add
  2. Configure vRO endpoint in vRA
    1. I have an enterprise install with external vRO so I am assuming you already setup the external vRO server in vRA.
    2. Login to vRA tenant
    3. Click Infrastructure tab > Endpoints > Endpoints
    4. Click on New > Orchestration > vRealize Orchestrator
    5. Screen Shot 2017-04-13 at 2.11.58 PM.png
    6. Enter the information
    7. Click on New Custom Property.
    8. Name: VMware.VCenterOrchestrator.Priority
    9. Value: (number, 1 being highest priority)
    10. Click OK
  3. Configure extensibility message timeout in vRA
    1. Login to vRA tenant
    2. Click Infrastructure tab -> administration -> Global Settings
    3. Click the Extensibility lifecycle message timeout row
    4. Click the Edit button
    5. Screen Shot 2017-04-13 at 2.44.44 PM.png
    6. Input a value that will be greater than the longest event workflow subscription timeout (e.g. 04:00:00)
    7. For the timeout setting to take affect, restart the vCloud Automation Center Service first on the primary manager service server and then on secondary.
  4.  Execution permission in vRO
    1. This is necessary for vRO to execute external applications and perform actions like ping. 
    2. These steps also need to be performed on all vRO nodes.
    3. SSH/Putty vRO server as root
    4. Modify the vmo.properties file:
      1. vi /etc/vco/app-server/vmo.properties
      2. Press the i key on the keyboard
      3. Copy & paste the following line to the end file:
      4. com.vmware.js.allow-local-process=true
      5. Press the esc key on the keyboard
      6. Type in :wq! and press the Enter key
    5. Modify the js-io-rights.conf file:
      1. vi /etc/vco/app-server/js-io-rights.conf
      2. Press the i key on the keyboard
      3. Copy & paste the following line to the end file:
      4. +rwx /tmp
      5. Press the esc key on the keyboard
      6. Type in :wq! and press the Enter key
    6. Ensure that the file has the appropriate permissions:
      1. cd /etc/vco/app-server
      2. chown vco:vco js-io-rights.conf
      3. chmod 640 js-io-rights.conf
    7. Restart the vRO server(s)
      1. service vco-server restart
  5. EMC and Kerberos configuration in vRO
    1. There are some additional steps that you need perform if you are using EMC FEHC 3 and 4, as well as Kerberos.
    2. I am not using these so will skip but documentation provides all the information needed.
    3. http://docs.sovlabs.com/vRA7x/current.html#4.2-first-install
  6. Configure vRA Endpoints in vRO  (use vRO to create workflows in order to interact with vRA)
    1. Perform the following once in vRO for each vRA tenant
    2. Login to vRO Client
    3. Select Design mode
    4. Click workflow tab
    5. Run workflow:  /Library/vRelease Automation/Configuration/Add a vRA host
      1. Screen Shot 2017-04-13 at 2.56.29 PM.png
      2. Enter vRA host name
      3. Host URL
      4. Automatically install Certs = yes
      5. Session mode = shared session
      6. Tenant name
      7. Username and password
        • username@domain.com
      8. Rest of fields not mentioned just leave default
    6. Click Submit
    7. If this fails make sure the service account is searchable in vRA directory users and groups.
  7. Add an IaaS host in vRO
    1. Perform the following once in vRO for each vRA tenant
    2. Login to vRO Client
    3. Select the Design mode
    4. Click Workflow tab
    5. Run workflow:  /Library/vRealize Automation/Infrastructure Administration/Configuration/Add an IaaS host
      1. Screen Shot 2017-04-13 at 3.41.47 PM.png
      2. Enter Host Name (IaaS Host FQDN)
      3. Enter Host URL (https://IaaS Host FQDN)
      4. Automatically install Certs = yes
      5. Use proxy = no
      6. Click Next
      7. Default connection settings = yes
      8. Click Next
      9. Host authentication type = NTLM
        • For the NTLM, is it a local user or an LDAP/AD user?
        • If it’s local, you use user@tenant
        • You can also use SSO
      10. Enter Username and password
        • for Username only specify the username and do not add the domain
      11. Workstation leave blank
      12. Enter domain name for NTLM authentication
    6. Click Submit
  8. Environment setup
    1. Review the documentation for additional setup configurations.
    2. http://docs.sovlabs.com/vRA7x/current.html#4.2-first-install
      1. Firewall configurations provided in documentation
      2. WinRM setup for SovLabs modules utilizing any Windows servers in the environment (for AD, DNS, IPAM, Puppet and etc.)
      3. Configuration of Windows member server when direct access to AD server is not permitted in the environment.

Continue reading →

vRealize Log Insight – vRealize Orchestrator content pack v2 released

/ johannstander / Leave a comment

VMware recently release the new vRealize Orchestrato content pack for vRealize Log Insight.   This content pack fixes some issues with the first version and now supports agent groups as well as the native Log Insight Agent.

The agent group can be used with either VRA 7.0.1 virtual appliance with embedded vRO, or the standalone VRO virtual appliance.  It is mentioned that results may be unpredictable if agent groups are used on an earlier version of vRO.

  • Select Administration
  • Under management select Agents.
  • Verify the agent group for “vRealize Orchestrator 7.0.1” is available from drop down box after installation.
  • Highlight the agent group and select copy template.(double square icon on far right)
  • Provide new name.
  • A prompt will appearing saying no collection agents have sent data to this log insight server .  A link will appear to download the Log Insight Agent version 3.6.0. This is new and a nice touch!
  • Download the agent.
  • Install on either the vRA server with embedded vRO or standalone vRO appliance.
    • This can be accomplished by upload the file using tool like WinSCP.
    • Copy file to /tmp folder.
    • Run “sudo rpm -i VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm”.  You will probably get a conflict error for existing 3.3.0-3516686 version of agent that pre-installed.
    • Run “sudo rpm -Uhv VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm” to update the package.
  • Update the log insight file /var/lib/loginsight-agent/liagent.ini
    • Update hostname=<vrealizeLogInsightserver.domain.com>
    • Some additional parameters are available for configuration like protocol, port, ssl and reconnect.
  • Back on Agent configuration with vRealize log insight..
  • Create filter that limits your specific vCD Cells by either selecting the hostname or IP address to filter by.
    • Verify that you see the agent listed for the vRealize orchestrator server where agent was just installed/update on.
  • Save New Group

Lots of nice dashboards to drool over.

Screen Shot 2016-10-05 at 3.37.56 PM.png

 

vRealize Orchestrator control center : HTTP Status 500 Failed to edit Log insight configuration file

/ johannstander / Leave a comment

With latest vRealize Orchestrator 7.0.1 I was configuring syslog logging integration in control center, to send logs to vRealize Log insight, but ran into error “HTTP Status 500 Failed to edit Log insight configuration file”.

Troubleshooting:

Testing on a fresh install and did no run into the problem so came to the conclusion that this error only appears when you upgrade from 7.0 to 7.0.1

SSH into Orchestrator appliance and reviewed the logs.
/etc/var/log/messages

2016-04-27T17:19:32.013813+00:00 ldvro01 sudo:      vco : a password is required ; TTY=unknown ; PWD=/var/lib/vco/configuration/bin ; USER=root ; COMMAND=/var/lib/vco/app-server                          /../configuration/bin/config_liagent.sh /var/lib/vco/configuration/temp/liagent.tmp /var/lib/loginsight-agent/liagent.ini
2016-04-27T17:20:10.075308+00:00 ldvro01 sshd[20887]: rexec line 79: Unsupported option KerberosAuthentication
2016-04-27T17:20:10.075376+00:00 ldvro01 sshd[20887]: rexec line 85: Unsupported option GSSAPIAuthentication
Found the script that gets executed to be /var/lib/vco/configuration/bin/config_liagent.sh which actually resides on /usr/lib/vco/configuration/bin/config_liagent.sh
Listing the folder shows that vco:vco has rwx permission.
:/usr/lib/vco/configuration/bin # ls -ll
-rwx—— 1 vco vco  218 Feb 19 15:09 config_liagent.sh
-rwx—— 1 vco vco  230 Feb 19 15:09 controlcenter.sh
-rw-r–r– 1 vco vco 6718 Feb 19 15:09 log4j.dtd
-rw-r–r– 1 vco vco 3315 Feb 19 15:09 propagate.sh
-rwx—— 1 vco vco 1321 Feb 19 15:09 setenv.sh
A password is required is throw in the error message which leads me to think the vco user does not have the necessary permissions when trying to execute the command.
Looking in /etc/sudoers file and found the vco missing the path to the config_liagent.sh file.
Resolution:
Add the path to config_liagent.sh for vco user.
# visudo
scroll to bottom of file.
you will see the following:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
update the line as follows:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

Java problems with vCenter Orchestrator

/ johannstander / Leave a comment

All applets and web start java applications has defaulted to high security since Update 11.
The security context that is used by vCO Client is set to high so some changes are needed within the Java control panel.

Resolution:

  • Open the Java Control Panel
  • Go to the Security tab. 
  • At the bottom of the dialog you will see the current Exception Site List. 
  • Click the Edit Site List button.
  • In the exception entry dialog, enter the URL for your vCO Server