VMware announces general availability for all vRealize Suite Standard products!

VMware has already been teasing us since June 6th with the upcoming releases of the following vRealize Suite products:

Today VMware announced GA for all products mentioned, with what seems to be a unified message to provide one integrated architecture, with greater/deeper integration across SDDC technologies and multiple public clouds. ¬†I like where this is going…

Couple of key take aways for me which are shared amongst some of the products (not all):

  • Redesigned HTML5 UI
    • Log Insight jumped on this long ago.
  • OOTB Integration between the different products
    • We have started seeing this with previous release but not going into full swing
  • Standardizing on authentication with VIDM

Release notes for each product:

 

Hopefully I can make some time in the upcoming weeks to dive a bit deeper into some of the features, but due to my busy schedule I am not holding my breath ūüôā Happy downloads!

vRealize Log Insight – vRA 7 content pack configuration

A couple of months ago Kimberly Delgado wrote a detailed blog on the new vRA 7 content pack for vRealize log insight which includes all the enhancements. No need to me to repeat and link provide below.

http://blogs.vmware.com/management/2016/04/introducing-vrealize-automation-7-0-content-pack-log-insight.html

I have had these steps written down quite some time ago but never really got around to putting into a blog, but since my post a couple of days ago on how to setup vRO content pack I though this would be fitting since these 2 content packs really go hand in hand.

vRA consists of multiple components and therefore can be configured with a simple or enterprise installation. This makes the deployment of the vRealize log insight agents interesting since you have install and/or configure the agents on both Windows (IaaS) and linux (virtual appliances).

The vRA content packs makes use of the Log Insight agent for both Windows and Linux which is great since this simplifies the configuration a lot!  The agent already comes preinstalled on vRA virtual appliances (easy) but still needs to be installed on the other component servers.

My configuration is an enterprise installation with following servers.

  • 2 x vRA virtual appliances
  • 2 x IaaS management
  • 2 x IaaS web
  • 2 x DEM/Agent servers

Here are the steps:

  • Login to vRealize Log insight
  • Select Content Packs
  • Select Marketplace
  • Select VMware – vRA 7 and install.
  • Select Administration
  • Under management select Agents.
  • Verify the agent groups ‚ÄúvRealize Automation 7 – Linux‚ÄĚ and ‚ÄúvRealize Automation 7 – Windows” are available from drop down box.
  • Scroll to the bottom of Agent page and select “Download Log Insight Agent Version 3.x.x”
    • Download the Windows MSI.¬† Remember no need to download the linux agent since the agent is already preinstalled on vRA appliance.
  • Install windows agents on all IaaS web servers.
    • Run the msi on windows server.
    • Enter/verify the hostname of vRealize log insight <loginsightname.domain.com> during configuration setup.¬† Since we downloaded the agent from the vRLI server management UI the hostname gets populated in installer. cool!
    • Press Install
  • SSH to the vRA VA and update liagent.ini to point to the LI server.
    • Update the log insight file /var/lib/loginsight-agent/liagent.ini
      • Update hostname=<vrealizeLogInsightserver.domain.com>
      • Some additional parameters are available for configuration like protocol, port, ssl and reconnect.
  • Login to vRealize log insight to verify the agents are communicating with vRI server.
    • Select Administration
    • Select Agents
    • Verify agents are showing up in list.¬† If they do not verify that the service is running on the server or review the log files for agent on windows servers C:\ProgramData\VMware\Log Insight Agent\log
  • Now lets create the new agents groups
    • Highlight each of agent groups mentioned earlier from the drop down box and select copy template.(double square icon on far right)
    • Create filters to limit to the required servers.¬† Since we have 6 IaaS windows servers I create a easy filter with hostname starts with vraweb,vradem,vramgr.¬† Click refresh and you should now see all 6 agents.
    • Some further configuration is required here and we need to update and/or add to the agent configuration file logs for Windows agents.
      • Have a look at Vra-dem, vra-dem-metrics, vra-deo, vra-deo2
      • For instance lets review vra-deo where the directory is normally <hostname>-DEO after Distributed Execution Manager folder.
      • By default the directory is set to “C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEO\Logs\”. This is incorrect…
      • Screen Shot 2016-10-05 at 5.56.13 PM.png
      • Directory SHOULD BE C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\FQDN-DEO\Logs\
      • screen-shot-2016-10-05-at-5-57-12-pm
      • Also in my case I have multiple DEM and management servers so we have to create additional file log entries for each of the servers since the DEM folder name changes.
      • The easiest method to duplicate File Logs is to select the Edit tab and copy/paste original filelog and then just update the directory.
        • Screen Shot 2016-10-07 at 10.04.54 AM.png
      • The other method is to hover over File Logs and you will see a green “+ New” button appear, select it.¬† Lets called it “vra-dem2” to add the 2nd server file location.
        • I hope they release a feature to clone the file logs but currently I just go back and forth between original and new file log and copy paste the items since only change required is the directory, rest of entries stay the same.
        • Add the same tags manually
        • Do not forget to update the parse fields from drop down box as well.
        • screen-shot-2016-10-05-at-6-16-15-pm
      • Select “Save Agent Group”
      • This can be confusing and a bit tricky but I hope my explanation makes sense to update the file logs for Agent Configuration build.

Complimentary content packs which are highly recommend:

  • VMware Orchestrator 7.0
  • VMware vSphere
  • VMware NSX-vSphere (if implemented)

You should start seeing the event count increase for each of the agents as well as dashboards populating.  Go geek out on stats!  But wait, the Catalog dashboard also provides you with valuable pre-build queries to perform analysis on specific errors and failure scenarios.  The queries are different per dashboard so I do encourage you to get familiar with each.  Example of Alert Queries below for General РProblems dashboard:

Screen Shot 2016-10-07 at 6.40.14 PM.png

 

 

 

vRealize Log Insight – vRealize Orchestrator content pack v2 released

VMware recently release the new vRealize Orchestrato content pack for vRealize Log Insight.   This content pack fixes some issues with the first version and now supports agent groups as well as the native Log Insight Agent.

The agent group can be used with either VRA 7.0.1 virtual appliance with embedded vRO, or the standalone VRO virtual appliance.  It is mentioned that results may be unpredictable if agent groups are used on an earlier version of vRO.

  • Select Administration
  • Under management select Agents.
  • Verify the agent group for “vRealize Orchestrator 7.0.1” is available from drop down box after installation.
  • Highlight the agent group and select copy template.(double square icon on far right)
  • Provide new name.
  • A prompt will appearing saying no collection agents have sent data to this log insight server .¬† A link will appear to download the Log Insight Agent version 3.6.0. This is new and a nice touch!
  • Download the agent.
  • Install on either the vRA server with embedded vRO or standalone vRO appliance.
    • This can be accomplished by upload the file using tool like WinSCP.
    • Copy file to /tmp folder.
    • Run “sudo rpm -i VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm”.¬† You will probably get a conflict error for existing 3.3.0-3516686 version of agent that pre-installed.
    • Run “sudo rpm -Uhv VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm” to update the package.
  • Update the log insight file /var/lib/loginsight-agent/liagent.ini
    • Update hostname=<vrealizeLogInsightserver.domain.com>
    • Some additional parameters are available for configuration like protocol, port, ssl and reconnect.
  • Back on Agent configuration with vRealize log insight..
  • Create filter that limits your specific vCD Cells by either selecting the hostname or IP address to filter¬†by.
    • Verify that you see the agent listed for the vRealize orchestrator server where agent was just installed/update on.
  • Save New Group

Lots of nice dashboards to drool over.

Screen Shot 2016-10-05 at 3.37.56 PM.png

 

vCloud Director 8: Configure logging to vRealize Log insight

With the recent release of the vCloud Director content pack (v8.4) for vRealize log insight I thought I would put the steps here for how to get this configured.

There are 2 methods to get the logs forwarded to your vRLI server.

  1. log4j. Setting an additional logger in VCD log4j.properties file.
  2. Loginsight Agent installed on

Steps to configure Log4j:

  • Logging is normally handled by log4j with configuration file /$VCLOUD_HOME/etc/log4j.properties.
  • Login to Cell with SSH.
  • Change to directory /$VCLOUD_HOME/etc/
  • Make backup copy of log4j.properties
    • cp log4j.properties log4j.properties.orig
  • Open the file log4j.properties in a text editor and add the following lines, where syslog-host-fqdn¬†is the FQDN¬†of your syslog host and port is an optional port number. If no port number specified then will default to 514.
    • log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
    • log4j.appender.vcloud.system.syslog.syslogHost=syslog-host-fqdn:port
  • Modify this line to add the vCloud Director syslog appenders:
    log4j.appender.vcloud.system.syslog.facility=LOCAL1

    • log4j.appender.vcloud.system.syslog.facility=LOCAL1, log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog
  • Specify the logger pattern
    • #log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.layout.CustomPatternLayout
    • log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
    • log4j.appender.vcloud.system.syslog.threshold=INFO
  • Save file
  • Restart Cell ¬†(yes not ideal, so therefore my reason to recommended log inisght agent)
    • service vmware-vcd restart
  • Repeat on each cell

Steps to configured Log Insight agent: (recommended)

  • Install the content pack which is pretty straight forward through the marketplace.
  • Verify the agent group for vCloud Director is available after installation.
  • Select the agent group and select copy template.
  • Provide new name.
  • Create filter that limits your specific vCD Cells by either selecting the hostname or IP address to filter¬†by.
  • Save
  • Install the LI agent on each vCD cell
    • Download the agent
      • Administration -> Management -> Agents
      • At bottom on page you can download the agent
    • Copy the Linux RPM file to tmp folder on vCD cell. Good tool to use is WinSCP.
    • Install agent
      • rpm – U VMware-Log-Insight-Agent-3.3.1-3636434.noarch_10.10.30.74.rpm
    • Since we downloaded the agent directly from our log insight server the liagent.ini should already be populated with your server IP Address. ¬†This can be verified by reviewing the ini file and looking for hostname entry. cat /etc/liagent.ini
  • You will now see the agent in log insight server. Verify that the agent is filtered correctly for you vCD Active group.

 

Links:

https://kb.vmware.com/kb/2004564

vRealize Log Insight: Configuring agents

The vSphere content pack provides powerful insight into your vSphere logs, allowing you to make informed and proactive decisions within your environment. ¬†For the exercise I am just reviewing some of the VMware products and providing notes I took during installation. Sorry if they seems a bit all over the place ūüôā

Log Insight agent now gets pre-installed on some of the appliances which is great and means no need to install agents manually.  Some of the VMware products that has agent pre-installed:
vRealize Business
vRealize Operations Manager (beginning from 6.1)
vRealize Orchestrator (beginning from 7.0.1)
vRealize Automation (beginning from 7.0.1)

vRealize Log Insight

Here are some basic functions which will help a lot for instructions on content packs:

Install Content Packs:

  1. Login to vRealize Log insight.
  2. Select the stack menu button in top right hand corner
  3. Select Content Packs
  4. Installation has been simplified a lot since you do not have to go to VMware solution exchange anymore to download and manually install the content packs, it is available straight from Marketplace window.  Super awesome!
  5. Just click on Install for which ever content pack you want to install.

How to view setup instructions?

  1. Select the stack menu button in top right hand corner
  2. Select Content Packs 
  3. Select Installed content pack
  4. Click the cog wheel -> Setup instructions

To verify if agent configuration from Log Insight was pushed successful to server:

Check the affective file to see if the correct agent configuration file logs has been pushed to the liagentd.

Linux:

\etc\liagent-affective

Windows:

C:\ProgramDATA\Vmware\Log insight agent\liagent-affective

View the agent configuration settings:

 

  1. Login to vRealize Log insight.
  2. Select the stack menu button in top right hand corner
  3. Select Content Packs 
  4. Select Installed content pack
  5. Select Agent Groups tab
  6. Find group name and review the Notes and Configuration

Agent Groups

Agent Groups comes as part of the content packs you installed. This is required for dashboard to work correctly. If you use syslog-ng, you will still receive the events but the vSphere content pack dashboards will not work.

  • I would always recommend making a copy of the original
  • Provide a new name
  • Save it
  • Provide a filtered list of hosts which could be by name, IP address or wildcards. ¬†These hosts should already have been already registered to Log Insight via their Agent configuration.
  • Save the Agent Group.

The configuration is automatically pushed out to the selected hosts and log messages will begin flowing in.

Install agents on linux:

This is of course not part of VMware products but providing the steps to manually install the agent on a linux box which you still need to do sometimes.

http://pubs.vmware.com/log-insight-30/index.jsp#com.vmware.log-insight.agent.admin.doc/GUID-83976956-C16C-42BD-9950-C6EDDF983086.html

  1. Make sure the hostname is set under /etc/hosts, /etc/HOSTNAMES, hostname   (otherwise server will show up with localhost hostname)
  2. Copy the bin file to appliance (this is SUSE so have to copy the bin)
  3. Chmod +x .bin
  4. ./.bin
  5. Vi /etc/liagent.ini
  6. http://pubs.vmware.com/log-insight-30/index.jsp?topic=%2Fcom.vmware.log-insight.agent.admin.doc%2FGUID-D245F706-BC99-46D0-87E3-584D9D250529.html
  7. (/etc/init.d/liagentd status/stop/restart)

 

In order to download the agent from server and install the agent I use following commands:

# curl -o /tmp/liagent-current.rpm http://LOGINSIGHT-SERVER:9000/api/v1/agent/packages/types/rpm ; rpm -Uvh /tmp/liagent-current.rpm

NSX:

NSX Manager

Sends all audit logs and system events from NSX Manager to the syslog server.

Steps

  1. Log in to the NSX Manager virtual appliance.
  2. Under Appliance Management, click Manage Appliance Settings.
  3. From the Settings panel, click General.
  4. Click Edit next to Syslog Server.
  5. Type the IP address of the syslog server.
  6. Required Type the port and protocol for the syslog server.  If you do not specify a port, the default UDP port for the IP address/host name of the syslog server is used.
  7. Click OK.

 

NSX Edge

NSX Edge events and logs related to firewall events that flow from NSX Edge appliances are sent to the syslog servers.

Steps

  1. Log in to the vSphere Web Client.
  2. Click Networking & Security and then click NSX Edges.
  3. Double-click a NSX Edge.
  4. Click the Manage tab and then click the Settings tab.
  5. In the Details panel, click Change next to Syslog servers.
  6. Type the IP address of both remote syslog servers and select the protocol.
  7. Click OK to save the configuration.

NSX Controllers:

The only supported method on configuring the syslog server on the NSX controllers is through the NSX API which is described in the KB below:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2092228

https://jreypo.wordpress.com/2015/09/30/how-to-configure-a-remote-syslog-server-for-nsx-controller/

I did however found another way to perform this through SSH but use at own risk and I still recommend using the NSX API!

  1. SSH into NSX controller:
  2. Change Controller cluster password
  3. vCenter server -> Networking & security -> Installation -> Management
  4. NSX controller select
  5. Actions -> Change controller cluster password
  6. 12 character min
  7. Login
  8. # show syslog-exporters
  9. add syslog-exporter
    1. This will add a syslog exporter
  10. add syslog-exporter-facility
    1. This will add a facility to a syslog exporter

Example:  # add syslog-exporter nsx-controller-syslog INFO kern,user,mail,deamon,auth,syslog,lpr,news,uucp,cron,security,ftp,ntp,logaudit,logalert,clock,local0,local1,local2,local3,local4,local5,local6,local7,api,api_request,api_request_content,api_request_header,logical_net,system,transport_net

  • 514 UDP

 

vRA 7:

  1. Install content packs:
    1. Vra7
    2. Vrealize orchestrator
    3. Apache
  2. Download windows agents from administration -> Management -> Agents -> Right at bottom of screen!
  3. Install agents on windows servers  (management, DEM, Web)
  4. From drop-down agents select vRealize 7 – Windows and create filter for only the windows server for instance hostname = wdvra*.domain.com

Update:

Vra-dem, vra-dem-metrics, vra-deo, vra-deo2

Under agent configuration update the paths where necessary like for instance vra-deo where the directory is normally -DEO after Distributed Execution Manager folder¬† “C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEO\Logs\”¬† BUT SHOULD BE C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\FQDN-DEO\Logs\

If you have multiple DEM servers or management servers then create  another file log called vra-dem2 to add the 2nd server file location.

For vRA appliances:

  1. Just update the \etc\liagent.ini with the hostname for vrealizeloginsight server.
  2. Restart service \etc\init.d\liagentd restart

vRealize Orchestrator:

Some good information from VMware blog on Orchestrator for vRealize Log insight.

http://blogs.vmware.com/management/2016/04/vrealize-orchestrator-7-0-content-pack-log-insight.html

 

  1. Login to vrealize orchestrator control center.
  2. https://10.10.30.133:8283/vco-controlcenter/
  3. Select Log -> Logging Integration
  4. Check box for “Enable logging to a remote log server”
  5. Currently only Log4j is supported but upcoming release after 7.0.1 should support Log Insight Agent
  6. Enter Host, Port and protocol.
  7. Test Connection
  8. Save

 

Problems experienced:

This did not work and got an error “HTTP Status 500 – Failed to edit Log Insight Agent configuration file!”

I create another blog to show how to fix this problem:

http://virtualrealization.blogspot.com/2016/05/vrealize-orchestrator-control-center.html

Agents group template does not show up and had to uninstall and reinstall the agent.

 

vRealize Orchestrator control center : HTTP Status 500 Failed to edit Log insight configuration file

With latest vRealize Orchestrator 7.0.1 I was configuring syslog logging integration in control center, to send logs to vRealize Log insight, but ran into error “HTTP Status 500 Failed to edit Log insight configuration file”.

Troubleshooting:

Testing on a fresh install and did no run into the problem so came to the conclusion that this error only appears when you upgrade from 7.0 to 7.0.1

SSH into Orchestrator appliance and reviewed the logs.
/etc/var/log/messages

2016-04-27T17:19:32.013813+00:00 ldvro01 sudo:      vco : a password is required ; TTY=unknown ; PWD=/var/lib/vco/configuration/bin ; USER=root ; COMMAND=/var/lib/vco/app-server                          /../configuration/bin/config_liagent.sh /var/lib/vco/configuration/temp/liagent.tmp /var/lib/loginsight-agent/liagent.ini
2016-04-27T17:20:10.075308+00:00 ldvro01 sshd[20887]: rexec line 79: Unsupported option KerberosAuthentication
2016-04-27T17:20:10.075376+00:00 ldvro01 sshd[20887]: rexec line 85: Unsupported option GSSAPIAuthentication
Found the script that gets executed to be /var/lib/vco/configuration/bin/config_liagent.sh which actually resides on /usr/lib/vco/configuration/bin/config_liagent.sh
Listing the folder shows that vco:vco has rwx permission.
:/usr/lib/vco/configuration/bin # ls -ll
-rwx—— 1 vco vco ¬†218 Feb 19 15:09 config_liagent.sh
-rwx—— 1 vco vco ¬†230 Feb 19 15:09 controlcenter.sh
-rw-r–r– 1 vco vco 6718 Feb 19 15:09 log4j.dtd
-rw-r–r– 1 vco vco 3315 Feb 19 15:09 propagate.sh
-rwx—— 1 vco vco 1321 Feb 19 15:09 setenv.sh
A password is required is throw in the error message which leads me to think the vco user does not have the necessary permissions when trying to execute the command.
Looking in /etc/sudoers file and found the vco missing the path to the config_liagent.sh file.
Resolution:
Add the path to config_liagent.sh for vco user.
# visudo
scroll to bottom of file.
you will see the following:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
update the line as follows:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

vCenter Log Insight – move NFS archive location

Had to recently change the vCenter log insight archive to a new NFS location.   Here is how you change this 
Solution:

SSH to appliance.
Stop the service

  • /etc/init.d/loginsight stop¬†
Copy the archive files over the new NFS location.

Change log insight configuration file to new NFS location

  • /storage/core/loginsight/config/¬†
  • Vi loginsight-config.xml#
Start the service 
  • /etc/init.d/loginsight start