Posted By Jon on Sep 26, 2017 | 2 comments

Hey everyone – welcome back to Part 3 of my vRealize LogInsight 4 series.  I apologize that I’ve taken so long to get back on track with this series.  Check the links below for previous installments:

vRealize LogInsight 4: The best syslog solution you didn’t know you had

vRealize LogInsight 4 – Part 1: Security and Planning Considerations

vRealize LogInsight 4 – Part 2: Deploying – Single or First Node

Today we’ll be talking about situations where we will use agents, where we won’t, and why or why not.

What is a syslog?

Let’s talk about syslog in general.  When you setup a device to send its log entries to a logging server… what happens?  Well, good question!  Many things could happen.  Depending on the device type and implementation of its syslog mechanism it could mean that all entries that are logged are no longer stored on the device and are solely shipped to the logging server.  Or, it could mean that every single entry anywhere in /var/log/ is sent to the syslog server and also retained locally (possibly growing /var/log out of control).  Maybe something gets appended to the beginning or end of each log entry.  Maybe nothing gets appended and all you have is the IP address of the device as an identifier.  Sometimes you can configure this behavior with packages like rsyslog.  Either way, when you send logs via syslog you are requiring the native log shipping mechanisms to do their thing and hoping that the output is useful and structured.

Sometimes this works out fine:

As you can see, this “Switched Rack PDU” identified only by 192.168.50.6 had some events kick out.  It’s totally usable and totally legible in this case.  But, sometimes you’re not so lucky and the messages look like this:

Both examples of these logs above are taken from the Administration | Hosts menu within LogInsight.  In both cases it looks like single-line logs were shipped and things entered cleanly.  However, this is not always the case.  This is where you’ll find all of your ordinary syslog “hosts” stored.  Because these are syslog events, you might be lucky and get an actual hostname or you might just get an IP address for the “host”.  But, either way, things that show up here need to have a set of rules built around them (which VMware does provide via various Content Packs/Dashboards) in order to make use of the actual information.  Or, you could just use LogInsight to store syslogs for longer duration than could an endpoint, but you probably want to leverage other vRealize LogInsight clever features.  Enter the vRealize LogInsight Agent.

What is a vRealize LogInsight agent?

When talking vRealize LogInsight, an agent is a piece of software provided by VMware that you install on an endpoint in order for it to communicate to the LogInsight server.  Because it’s an extra piece of software to be installed on a guest you can bet there are a few features included:

• Single or organized group deployment (via transform file and .msi for GPO installation)
• Leverages the vRealize LogInsight ingestion API for compression, encryption, and more.
• Upgrading vRealize LogInsight agents from the LogInsight interface
• Support for parsing/storing of multi-line syslog events
• Log rotation schemes
• Takes unstructured log entries and parses them into useful format

The awesome part about agents is that they are inherently more “intelligent” when reporting into LogInsight.  By this, I mean that the data from the client comes with some more metadata right off the bat:

But, not only do you see more data from the agents view, you can also filter and group these agents into applicable groups.  For instance, if I want to be able to view all Windows machines under the provided Microsoft – Windows Content Pack, I can filter by certain attributes and apply those filters to an included group for the Content Pack.  So, simply filter on OS:

You can see I cloned the Microsoft – Windows Content Pack group to “Microsoft – Windows – KensyCloud” so as to not lose the original configuration.  When you installed the Microsoft – Windows Content Pack you would have seen the instructions on what needs to be performed from within the guest in order to get the right data out.  In case you missed it, or ignored it, you can review the information by going to Content Pack admin menu and selecting the installed pack, then clicking the cog near the top of the screen and choosing Setup Instructions.  Some Content Packs do not require anything to be performed in the guest.

Once you’ve got the default Microsoft – Windows Content Pack template cloned and filters added, you should soon see your applicable hosts showing up in the dashboard:

That’s it!  Of course you can tweak the dashboards and or view the actual source syslog messages, but you get the idea.

The benefit of the agent is that you can collect a lot more than syslog might be able to send by default.  For instance, Microsoft – SQL Server Content Pack extracts fields out from the SQL Server logging location (default is C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Log).  The cool thing is that this does not mess with the actual log file for SQL Server.  Instead, there is an operational log kept by the vRealize LogInsight agent itself (default is C:\ProgramData\VMware\Log Insight Agent\logs) which rolls over every 10MB with a total combined 50MB total aggregated capacity across LogInsight agent logs.  So, you can see how this might be super useful for building your own fields and dashboards for applications you run.

Another benefit is appeasing your security guy(s).  For instance, some NetSec people shudder when you say you’re shipping syslogs inside or outside of a network via UDP/TCP 514 – “OMG – there could be credentials in plaintext!”  They’re not wrong.  Sure, there’s encrypted syslog but if you want to ensure encryption (and compression) is in place simply use the cfapi with ssl method for connection to the LogInsight server within the agent config (this is the default) and REST easy (I love puns).

By the way – this is not to say that you can only use cool dashboards with agents.  If the logs kicked out of a device have some pattern or distinguishing aspect to them then you can surely apply them to a dashboard.  For instance, the Cisco – ASA Content Pack comes with dashboards that just filters on cisco_asa_ tags within conventional syslog output from Cisco ASA devices (obviously you can’t install an agent on these hardware devices):

Installing the vRealize LogInsight agent

Installing the agent is super easy.  I mean – it couldn’t be any easier.  For singular install options you simply visit your Administration | Agents page and at the bottom there is a link to download the pre-built agent in the binary format matching your OS:

Simply get the MSI, RPM, DEB, or BIN file over to your guest and run it.  The nice part is that when you run the installer you will notice that it is setup to ship logs to the hostname of your vRealize LogInsight server:

The same holds true with the Linux binaries.  Once installed on Linux you can check out the configuration by looking in /etc/liagent.ini:

~$ sudo cat /etc/liagent.ini
; VMware Log Insight Agent configuration. Please save as UTF-8 if you use non-ASCII names / values !
; Actual configuration is this file joined with settings from server to form liagent-effective.ini
; Note: Restarting the agent is not required after making a configuration change
; Note: It may be more efficient to configure from server's Agents page !

[server]
hostname=kcloud1logs1.ad.kensycloud.com.

; Hostname or IP address of your Log Insight server / cluster load balancer. Default:
;hostname=LOGINSIGHT

; Protocol can be cfapi (Log Insight REST API), syslog. Default:
;proto=cfapi

; Log Insight server port to connect to. Default ports for protocols (all TCP):
; syslog: 514; syslog with ssl: 6514; cfapi: 9000; cfapi with ssl: 9543. Default:
;port=9543

; SSL usage. Default:
;ssl=yes
; Example of configuration with trusted CA:
;ssl=yes
;ssl_ca_path=/etc/pki/tls/certs/ca.pem

; Time in minutes to force reconnection to the server.
; This option mitigates imbalances caused by long-lived TCP connections. Default:
;reconnect=30

[logging]
; Logging verbosity: 0 (no debug messages), 1 (essentials), 2 (verbose with more impact on performance).
; This option should always be 0 under normal operating conditions. Default:
;debug_level=0

[storage]
; Max local storage usage limit (data + logs) in MBs. Valid range: 100-2000 MB.
;max_disk_buffer=200

; Uncomment the appropriate section to collect system logs
; The recommended way is to enable the Linux content pack from LI server
;[filelog|syslog]
;directory=/var/log
;include=messages;messages.?;syslog;syslog.?

[update]
; Do not change this parameter
package_type=deb


; Enable automatic update of the agent. If enabled:
; the agent will silently check for updates from the server and
; if available will automatically download and apply the update.
;auto_update=yes

If you want to change any of the default options you can do so.  Both the Windows and Linux agents run as a service/daemon.  There are many installation customization options available so be sure to check the documentation for all options available.

One other really neat feature of deploying the vRealize LogInsight agent is that you can do it via GPO for Windows as well.  This relies on the creation of a .mst transformation file along with the .msi installer file and the options within the transform file are applied to the Windows OS receiving the GPO.  I encourage you to check out the documentation on this as it’s a little more complex than I think is worth getting into for this blog entry but you can probably see how powerful this might be.  Put an AD Computer Object into an OU, GPO applies, boom logs come in via agent.  You get it.

You can also put the vRealize LogInsight agent in your templates – this is what I prefer for labs.  The benefit is that every VM will report into LogInsight the moment it boots up (and the service or daemon starts).  The negative side is you’ll likely have a machine named “RHEL 7.3 Template” that you obviously installed the agent on in preparation that has checked into LogInsight once or twice but never again (until the template is booted up):

You can see I have a bunch of stuff that checked in 20 – 30 days ago when I patched the templates.  However, once the machine is deployed and has a unique hostname everything reports correctly.

One final mention is the ability to automatically update the agents.  I know we glanced over that in the features list of agents, but check this out:

By sliding the toggle to “on” all agents will be upgraded in the future to the version of the agent being served up by your vRealize LogInsight server.  Pretty neat, right?

Hopefully this part of my LogInsight series has helped you understand the benefit of using the vRealize LogInsight agent while also showing you brief examples of what you can do with and without the agent.  Let me know if you have any other tips or tricks!  Thanks all!