vRealize Log Insight

vRealize Management 8.1 (vROPS, vRLI, vRA): What’s new 

/ johannstander / 2 Comments

The big takeaway with the release of vRealize Management 8.1 is the support for vSphere 7 with Kubernetes.

These updates now provides automated delivery, monitoring, troubleshooting and capacity management for both container and VM workloads!  This is a big deal and will allow VI admins to easy provision and manage VMs and containers on products they already know and not having to deal with the complexities around container orchestration with Kubernetes.

Here are some of the key updates VMware mentions for each product as well as the some use cases, but we will dig into each of these in more detail below.

Annotation 2020-03-08 095336.png

Annotation 2020-03-08 095634.png

Before I get started I do just want to mention something that mostly everyone already knows but it is important to reiterate that all these products are available on-premises as well as SaaS based.  This provides lots of flexibility for wherever you workload will run for instance in a hybrid cloud environment or perhaps in the public cloud only.Annotation 2020-03-08 100324.png

Continue reading →

VMware announces general availability for all vRealize Suite Standard products!

/ johannstander / Leave a comment

VMware has already been teasing us since June 6th with the upcoming releases of the following vRealize Suite products:

Today VMware announced GA for all products mentioned, with what seems to be a unified message to provide one integrated architecture, with greater/deeper integration across SDDC technologies and multiple public clouds.  I like where this is going…

Couple of key take aways for me which are shared amongst some of the products (not all):

  • Redesigned HTML5 UI
    • Log Insight jumped on this long ago.
  • OOTB Integration between the different products
    • We have started seeing this with previous release but not going into full swing
  • Standardizing on authentication with VIDM

Release notes for each product:

 

Hopefully I can make some time in the upcoming weeks to dive a bit deeper into some of the features, but due to my busy schedule I am not holding my breath 🙂 Happy downloads!

vRealize Log Insight – vRA 7 content pack configuration

/ johannstander / Leave a comment

A couple of months ago Kimberly Delgado wrote a detailed blog on the new vRA 7 content pack for vRealize log insight which includes all the enhancements. No need to me to repeat and link provide below.

http://blogs.vmware.com/management/2016/04/introducing-vrealize-automation-7-0-content-pack-log-insight.html

I have had these steps written down quite some time ago but never really got around to putting into a blog, but since my post a couple of days ago on how to setup vRO content pack I though this would be fitting since these 2 content packs really go hand in hand.

vRA consists of multiple components and therefore can be configured with a simple or enterprise installation. This makes the deployment of the vRealize log insight agents interesting since you have install and/or configure the agents on both Windows (IaaS) and linux (virtual appliances).

The vRA content packs makes use of the Log Insight agent for both Windows and Linux which is great since this simplifies the configuration a lot!  The agent already comes preinstalled on vRA virtual appliances (easy) but still needs to be installed on the other component servers.

My configuration is an enterprise installation with following servers.

  • 2 x vRA virtual appliances
  • 2 x IaaS management
  • 2 x IaaS web
  • 2 x DEM/Agent servers

Here are the steps:

  • Login to vRealize Log insight
  • Select Content Packs
  • Select Marketplace
  • Select VMware – vRA 7 and install.
  • Select Administration
  • Under management select Agents.
  • Verify the agent groups “vRealize Automation 7 – Linux” and “vRealize Automation 7 – Windows” are available from drop down box.
  • Scroll to the bottom of Agent page and select “Download Log Insight Agent Version 3.x.x”
    • Download the Windows MSI.  Remember no need to download the linux agent since the agent is already preinstalled on vRA appliance.
  • Install windows agents on all IaaS web servers.
    • Run the msi on windows server.
    • Enter/verify the hostname of vRealize log insight <loginsightname.domain.com> during configuration setup.  Since we downloaded the agent from the vRLI server management UI the hostname gets populated in installer. cool!
    • Press Install
  • SSH to the vRA VA and update liagent.ini to point to the LI server.
    • Update the log insight file /var/lib/loginsight-agent/liagent.ini
      • Update hostname=<vrealizeLogInsightserver.domain.com>
      • Some additional parameters are available for configuration like protocol, port, ssl and reconnect.
  • Login to vRealize log insight to verify the agents are communicating with vRI server.
    • Select Administration
    • Select Agents
    • Verify agents are showing up in list.  If they do not verify that the service is running on the server or review the log files for agent on windows servers C:\ProgramData\VMware\Log Insight Agent\log
  • Now lets create the new agents groups
    • Highlight each of agent groups mentioned earlier from the drop down box and select copy template.(double square icon on far right)
    • Create filters to limit to the required servers.  Since we have 6 IaaS windows servers I create a easy filter with hostname starts with vraweb,vradem,vramgr.  Click refresh and you should now see all 6 agents.
    • Some further configuration is required here and we need to update and/or add to the agent configuration file logs for Windows agents.
      • Have a look at Vra-dem, vra-dem-metrics, vra-deo, vra-deo2
      • For instance lets review vra-deo where the directory is normally <hostname>-DEO after Distributed Execution Manager folder.
      • By default the directory is set to “C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEO\Logs\”. This is incorrect…
      • Screen Shot 2016-10-05 at 5.56.13 PM.png
      • Directory SHOULD BE C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\FQDN-DEO\Logs\
      • screen-shot-2016-10-05-at-5-57-12-pm
      • Also in my case I have multiple DEM and management servers so we have to create additional file log entries for each of the servers since the DEM folder name changes.
      • The easiest method to duplicate File Logs is to select the Edit tab and copy/paste original filelog and then just update the directory.
        • Screen Shot 2016-10-07 at 10.04.54 AM.png
      • The other method is to hover over File Logs and you will see a green “+ New” button appear, select it.  Lets called it “vra-dem2” to add the 2nd server file location.
        • I hope they release a feature to clone the file logs but currently I just go back and forth between original and new file log and copy paste the items since only change required is the directory, rest of entries stay the same.
        • Add the same tags manually
        • Do not forget to update the parse fields from drop down box as well.
        • screen-shot-2016-10-05-at-6-16-15-pm
      • Select “Save Agent Group”
      • This can be confusing and a bit tricky but I hope my explanation makes sense to update the file logs for Agent Configuration build.

Complimentary content packs which are highly recommend:

  • VMware Orchestrator 7.0
  • VMware vSphere
  • VMware NSX-vSphere (if implemented)

You should start seeing the event count increase for each of the agents as well as dashboards populating.  Go geek out on stats!  But wait, the Catalog dashboard also provides you with valuable pre-build queries to perform analysis on specific errors and failure scenarios.  The queries are different per dashboard so I do encourage you to get familiar with each.  Example of Alert Queries below for General – Problems dashboard:

Screen Shot 2016-10-07 at 6.40.14 PM.png

 

 

 

vRealize Log Insight – vRealize Orchestrator content pack v2 released

/ johannstander / Leave a comment

VMware recently release the new vRealize Orchestrato content pack for vRealize Log Insight.   This content pack fixes some issues with the first version and now supports agent groups as well as the native Log Insight Agent.

The agent group can be used with either VRA 7.0.1 virtual appliance with embedded vRO, or the standalone VRO virtual appliance.  It is mentioned that results may be unpredictable if agent groups are used on an earlier version of vRO.

  • Select Administration
  • Under management select Agents.
  • Verify the agent group for “vRealize Orchestrator 7.0.1” is available from drop down box after installation.
  • Highlight the agent group and select copy template.(double square icon on far right)
  • Provide new name.
  • A prompt will appearing saying no collection agents have sent data to this log insight server .  A link will appear to download the Log Insight Agent version 3.6.0. This is new and a nice touch!
  • Download the agent.
  • Install on either the vRA server with embedded vRO or standalone vRO appliance.
    • This can be accomplished by upload the file using tool like WinSCP.
    • Copy file to /tmp folder.
    • Run “sudo rpm -i VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm”.  You will probably get a conflict error for existing 3.3.0-3516686 version of agent that pre-installed.
    • Run “sudo rpm -Uhv VMware-Log-Insight-Agent-3.6.0-4148343.noarch_10.10.40.55.rpm” to update the package.
  • Update the log insight file /var/lib/loginsight-agent/liagent.ini
    • Update hostname=<vrealizeLogInsightserver.domain.com>
    • Some additional parameters are available for configuration like protocol, port, ssl and reconnect.
  • Back on Agent configuration with vRealize log insight..
  • Create filter that limits your specific vCD Cells by either selecting the hostname or IP address to filter by.
    • Verify that you see the agent listed for the vRealize orchestrator server where agent was just installed/update on.
  • Save New Group

Lots of nice dashboards to drool over.

Screen Shot 2016-10-05 at 3.37.56 PM.png

 

vCloud Director 8: Configure logging to vRealize Log insight

/ johannstander / Leave a comment

With the recent release of the vCloud Director content pack (v8.4) for vRealize log insight I thought I would put the steps here for how to get this configured.

There are 2 methods to get the logs forwarded to your vRLI server.

  1. log4j. Setting an additional logger in VCD log4j.properties file.
  2. Loginsight Agent installed on

Steps to configure Log4j:

  • Logging is normally handled by log4j with configuration file /$VCLOUD_HOME/etc/log4j.properties.
  • Login to Cell with SSH.
  • Change to directory /$VCLOUD_HOME/etc/
  • Make backup copy of log4j.properties
    • cp log4j.properties log4j.properties.orig
  • Open the file log4j.properties in a text editor and add the following lines, where syslog-host-fqdn is the FQDN of your syslog host and port is an optional port number. If no port number specified then will default to 514.
    • log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
    • log4j.appender.vcloud.system.syslog.syslogHost=syslog-host-fqdn:port
  • Modify this line to add the vCloud Director syslog appenders:
    log4j.appender.vcloud.system.syslog.facility=LOCAL1

    • log4j.appender.vcloud.system.syslog.facility=LOCAL1, log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog
  • Specify the logger pattern
    • #log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.layout.CustomPatternLayout
    • log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
    • log4j.appender.vcloud.system.syslog.threshold=INFO
  • Save file
  • Restart Cell  (yes not ideal, so therefore my reason to recommended log inisght agent)
    • service vmware-vcd restart
  • Repeat on each cell

Steps to configured Log Insight agent: (recommended)

  • Install the content pack which is pretty straight forward through the marketplace.
  • Verify the agent group for vCloud Director is available after installation.
  • Select the agent group and select copy template.
  • Provide new name.
  • Create filter that limits your specific vCD Cells by either selecting the hostname or IP address to filter by.
  • Save
  • Install the LI agent on each vCD cell
    • Download the agent
      • Administration -> Management -> Agents
      • At bottom on page you can download the agent
    • Copy the Linux RPM file to tmp folder on vCD cell. Good tool to use is WinSCP.
    • Install agent
      • rpm – U VMware-Log-Insight-Agent-3.3.1-3636434.noarch_10.10.30.74.rpm
    • Since we downloaded the agent directly from our log insight server the liagent.ini should already be populated with your server IP Address.  This can be verified by reviewing the ini file and looking for hostname entry. cat /etc/liagent.ini
  • You will now see the agent in log insight server. Verify that the agent is filtered correctly for you vCD Active group.

 

Links:

https://kb.vmware.com/kb/2004564

vRealize Log Insight: Configuring agents

/ johannstander / Leave a comment

The vSphere content pack provides powerful insight into your vSphere logs, allowing you to make informed and proactive decisions within your environment.  For the exercise I am just reviewing some of the VMware products and providing notes I took during installation. Sorry if they seems a bit all over the place 🙂

Log Insight agent now gets pre-installed on some of the appliances which is great and means no need to install agents manually.  Some of the VMware products that has agent pre-installed:
vRealize Business
vRealize Operations Manager (beginning from 6.1)
vRealize Orchestrator (beginning from 7.0.1)
vRealize Automation (beginning from 7.0.1)

vRealize Log Insight

Here are some basic functions which will help a lot for instructions on content packs:

Install Content Packs:

  1. Login to vRealize Log insight.
  2. Select the stack menu button in top right hand corner
  3. Select Content Packs
  4. Installation has been simplified a lot since you do not have to go to VMware solution exchange anymore to download and manually install the content packs, it is available straight from Marketplace window.  Super awesome!
  5. Just click on Install for which ever content pack you want to install.

How to view setup instructions?

  1. Select the stack menu button in top right hand corner
  2. Select Content Packs 
  3. Select Installed content pack
  4. Click the cog wheel -> Setup instructions

To verify if agent configuration from Log Insight was pushed successful to server:

Check the affective file to see if the correct agent configuration file logs has been pushed to the liagentd.

Linux:

\etc\liagent-affective

Windows:

C:\ProgramDATA\Vmware\Log insight agent\liagent-affective

View the agent configuration settings:

 

  1. Login to vRealize Log insight.
  2. Select the stack menu button in top right hand corner
  3. Select Content Packs 
  4. Select Installed content pack
  5. Select Agent Groups tab
  6. Find group name and review the Notes and Configuration

Agent Groups

Agent Groups comes as part of the content packs you installed. This is required for dashboard to work correctly. If you use syslog-ng, you will still receive the events but the vSphere content pack dashboards will not work.

  • I would always recommend making a copy of the original
  • Provide a new name
  • Save it
  • Provide a filtered list of hosts which could be by name, IP address or wildcards.  These hosts should already have been already registered to Log Insight via their Agent configuration.
  • Save the Agent Group.

The configuration is automatically pushed out to the selected hosts and log messages will begin flowing in.

Install agents on linux:

This is of course not part of VMware products but providing the steps to manually install the agent on a linux box which you still need to do sometimes.

http://pubs.vmware.com/log-insight-30/index.jsp#com.vmware.log-insight.agent.admin.doc/GUID-83976956-C16C-42BD-9950-C6EDDF983086.html

  1. Make sure the hostname is set under /etc/hosts, /etc/HOSTNAMES, hostname   (otherwise server will show up with localhost hostname)
  2. Copy the bin file to appliance (this is SUSE so have to copy the bin)
  3. Chmod +x .bin
  4. ./.bin
  5. Vi /etc/liagent.ini
  6. http://pubs.vmware.com/log-insight-30/index.jsp?topic=%2Fcom.vmware.log-insight.agent.admin.doc%2FGUID-D245F706-BC99-46D0-87E3-584D9D250529.html
  7. (/etc/init.d/liagentd status/stop/restart)

 

In order to download the agent from server and install the agent I use following commands:

# curl -o /tmp/liagent-current.rpm http://LOGINSIGHT-SERVER:9000/api/v1/agent/packages/types/rpm ; rpm -Uvh /tmp/liagent-current.rpm

NSX:

NSX Manager

Sends all audit logs and system events from NSX Manager to the syslog server.

Steps

  1. Log in to the NSX Manager virtual appliance.
  2. Under Appliance Management, click Manage Appliance Settings.
  3. From the Settings panel, click General.
  4. Click Edit next to Syslog Server.
  5. Type the IP address of the syslog server.
  6. Required Type the port and protocol for the syslog server.  If you do not specify a port, the default UDP port for the IP address/host name of the syslog server is used.
  7. Click OK.

 

NSX Edge

NSX Edge events and logs related to firewall events that flow from NSX Edge appliances are sent to the syslog servers.

Steps

  1. Log in to the vSphere Web Client.
  2. Click Networking & Security and then click NSX Edges.
  3. Double-click a NSX Edge.
  4. Click the Manage tab and then click the Settings tab.
  5. In the Details panel, click Change next to Syslog servers.
  6. Type the IP address of both remote syslog servers and select the protocol.
  7. Click OK to save the configuration.

NSX Controllers:

The only supported method on configuring the syslog server on the NSX controllers is through the NSX API which is described in the KB below:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2092228

https://jreypo.wordpress.com/2015/09/30/how-to-configure-a-remote-syslog-server-for-nsx-controller/

I did however found another way to perform this through SSH but use at own risk and I still recommend using the NSX API!

  1. SSH into NSX controller:
  2. Change Controller cluster password
  3. vCenter server -> Networking & security -> Installation -> Management
  4. NSX controller select
  5. Actions -> Change controller cluster password
  6. 12 character min
  7. Login
  8. # show syslog-exporters
  9. add syslog-exporter
    1. This will add a syslog exporter
  10. add syslog-exporter-facility
    1. This will add a facility to a syslog exporter

Example:  # add syslog-exporter nsx-controller-syslog INFO kern,user,mail,deamon,auth,syslog,lpr,news,uucp,cron,security,ftp,ntp,logaudit,logalert,clock,local0,local1,local2,local3,local4,local5,local6,local7,api,api_request,api_request_content,api_request_header,logical_net,system,transport_net

  • 514 UDP

 

vRA 7:

  1. Install content packs:
    1. Vra7
    2. Vrealize orchestrator
    3. Apache
  2. Download windows agents from administration -> Management -> Agents -> Right at bottom of screen!
  3. Install agents on windows servers  (management, DEM, Web)
  4. From drop-down agents select vRealize 7 – Windows and create filter for only the windows server for instance hostname = wdvra*.domain.com

Update:

Vra-dem, vra-dem-metrics, vra-deo, vra-deo2

Under agent configuration update the paths where necessary like for instance vra-deo where the directory is normally -DEO after Distributed Execution Manager folder  “C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEO\Logs\”  BUT SHOULD BE C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\FQDN-DEO\Logs\

If you have multiple DEM servers or management servers then create  another file log called vra-dem2 to add the 2nd server file location.

For vRA appliances:

  1. Just update the \etc\liagent.ini with the hostname for vrealizeloginsight server.
  2. Restart service \etc\init.d\liagentd restart

vRealize Orchestrator:

Some good information from VMware blog on Orchestrator for vRealize Log insight.

http://blogs.vmware.com/management/2016/04/vrealize-orchestrator-7-0-content-pack-log-insight.html

 

  1. Login to vrealize orchestrator control center.
  2. https://10.10.30.133:8283/vco-controlcenter/
  3. Select Log -> Logging Integration
  4. Check box for “Enable logging to a remote log server”
  5. Currently only Log4j is supported but upcoming release after 7.0.1 should support Log Insight Agent
  6. Enter Host, Port and protocol.
  7. Test Connection
  8. Save

 

Problems experienced:

This did not work and got an error “HTTP Status 500 – Failed to edit Log Insight Agent configuration file!”

I create another blog to show how to fix this problem:

http://virtualrealization.blogspot.com/2016/05/vrealize-orchestrator-control-center.html

Agents group template does not show up and had to uninstall and reinstall the agent.

 

vRealize Orchestrator control center : HTTP Status 500 Failed to edit Log insight configuration file

/ johannstander / Leave a comment

With latest vRealize Orchestrator 7.0.1 I was configuring syslog logging integration in control center, to send logs to vRealize Log insight, but ran into error “HTTP Status 500 Failed to edit Log insight configuration file”.

Troubleshooting:

Testing on a fresh install and did no run into the problem so came to the conclusion that this error only appears when you upgrade from 7.0 to 7.0.1

SSH into Orchestrator appliance and reviewed the logs.
/etc/var/log/messages

2016-04-27T17:19:32.013813+00:00 ldvro01 sudo:      vco : a password is required ; TTY=unknown ; PWD=/var/lib/vco/configuration/bin ; USER=root ; COMMAND=/var/lib/vco/app-server                          /../configuration/bin/config_liagent.sh /var/lib/vco/configuration/temp/liagent.tmp /var/lib/loginsight-agent/liagent.ini
2016-04-27T17:20:10.075308+00:00 ldvro01 sshd[20887]: rexec line 79: Unsupported option KerberosAuthentication
2016-04-27T17:20:10.075376+00:00 ldvro01 sshd[20887]: rexec line 85: Unsupported option GSSAPIAuthentication
Found the script that gets executed to be /var/lib/vco/configuration/bin/config_liagent.sh which actually resides on /usr/lib/vco/configuration/bin/config_liagent.sh
Listing the folder shows that vco:vco has rwx permission.
:/usr/lib/vco/configuration/bin # ls -ll
-rwx—— 1 vco vco  218 Feb 19 15:09 config_liagent.sh
-rwx—— 1 vco vco  230 Feb 19 15:09 controlcenter.sh
-rw-r–r– 1 vco vco 6718 Feb 19 15:09 log4j.dtd
-rw-r–r– 1 vco vco 3315 Feb 19 15:09 propagate.sh
-rwx—— 1 vco vco 1321 Feb 19 15:09 setenv.sh
A password is required is throw in the error message which leads me to think the vco user does not have the necessary permissions when trying to execute the command.
Looking in /etc/sudoers file and found the vco missing the path to the config_liagent.sh file.
Resolution:
Add the path to config_liagent.sh for vco user.
# visudo
scroll to bottom of file.
you will see the following:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
update the line as follows:
vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

vCenter Log Insight – move NFS archive location

/ johannstander / Leave a comment
Had to recently change the vCenter log insight archive to a new NFS location.   Here is how you change this 
Solution:

SSH to appliance.
Stop the service

  • /etc/init.d/loginsight stop 
Copy the archive files over the new NFS location.

Change log insight configuration file to new NFS location

  • /storage/core/loginsight/config/ 
  • Vi loginsight-config.xml#
Start the service 
  • /etc/init.d/loginsight start

vCenter Log Insight – VMware – vSphere content pack v 2.0 reinstall

/ johannstander / Leave a comment

After upgrading to vCenter Log Insight v 2.0 I wanted to install the latest content packs so I downloaded the VMware – vSphere content pack from VMware Solution Exchange.

I imported the content pack and I was prompted if i wanted to upgrade.
Not paying attention I actually downgraded the content pack to v1.5 which is the latest which seems to be available on Solution Exchange.

Here are the steps to revert back to latest v 2.0 content pack for VMware – vSphere:

  1. Download 7zip.
  2. Open the archive PAK file which is downloaded for the upgrade of Log Insight.
  3. Browse to \usr\lib\loginsight\application\etc\content-packs\vsphere\
  4. Extract the file content.json.
  5. Open Log Insight.
  6. Select Content Packs
  7. Select “Import Content Pack”
  8. Browse for the content.json file.
  9. Upload
This will now again install the latest version 2.0.

vCenter Log Insight 1.0.4 upgrade to v2.0.3? where to now?

/ johannstander / Leave a comment

So I had vCenter Log Insight v 1.0.4 installed in our environment and just never had the time to upgrade this until recently.

The problem that exists in that in the old releases the rpm file was used to upgrade through CLI.
In the new releases since 1.5G they make use of PAK files for upgrades through the web interface which is must more user friendly.

Well since I only had v1 installed and uses RPM it caused a bit of issues since only the v2 PAK file is available on download site.

To fix this i did the following:

  1. Download the v1.5 file from following location:
  2. Backups / snapshot your VM appliance.
  3. Copy the rpm to the VM, i used the \tmp folder.
  4. Upgrade Log Insight through the CLI by running the following commands:
    1. service loginsight stop
    2. rpm -Uvh loginsight-cloudvm-1.5.0-1435442.x86_64.rpm
    3. service loginsight start
  5. Now login to the appliance.
    1. Go to administration
    2. Under management select Appliance
    3. upload the new PAK file for v2
    4. Click upgrade.
This should be it and you should have the new version ready.

Links:

Upgrade log insight installed versions:
http://pubs.vmware.com/log-insight-15/index.jsp?topic=%2Fcom.vmware.log-insight.administration.doc%2FGUID-5933433F-4A17-4DF0-A05F-49DF2D104974.html&__utma=207178772.1598018981.1404147381.1404147381.1404147381.1&__utmb=207178772.0.10.1404147381&__utmc=207178772&__utmx=-&__utmz=207178772.1404147381.1.1.utmcsr=bing|utmccn=(organic)|utmcmd=organic|utmctr=vcenter%20log%20insight%20upgrade&__utmv=-&__utmk=81845013