This post will kick off a series I am writing on LogInsight 4. Each part of this series can be navigated by following the links below:
Today we’ll be talking about something often overlooked. Specifically, in this post, we’ll be talking about syslog servers Ew. I know.
Actually, I find log servers pretty fascinating especially those created within the last few years. I’ve dabbled with Kiwi Syslog, Graylog, PRTG, and the (unfortunately) industry standard, Splunk. I also gave VMware vRealize LogInsight a try back when it was version 2.5 and again in the 3.0-3.3 era, but ran with Graylog for a long time. However, VMware has released vRealize LogInsight 4 and it’s time for a revisit which turns out to be a really worthwhile exercise! This post will serve as a kick-off to my vRealize Log Insight series of blog posts. Being that this is the first of the series, I’ll give an overview of features and why you should use it and then move on to deploying, configuring agents, and interpreting the data in later blog posts.
Why you should have a consolidated logging solution
To some people, deploying a logging solution is dreadful and is a topic they avoid at all costs. Let it be someone else’s problem; I totally understand. In fact, once you set up a logging solution and have data pouring into it, you then need to go sift through things and figure out what’s important and what’s not, setup triggers, alerts, and after all of that you forget why you even need it because half the time you never get alerts or are getting alerted all the time.
Here’s why you need it:
- A modern consolidated logging solution provides a single pane of glass allowing you to parse and discover log events from all of your hosts/devices
- Some hosts/devices do not retain logging history when restarted and so troubleshooting is next to impossible if a device crashes
- Some hosts/devices do not retain more than a few megabytes of logging data and rolls over frequently
- Some hosts/devices do not even offer to store historical logging data on the host/device itself and require a server to point to
- When troubleshooting an application or service that spans many hosts/devices, it is not effective to have to pull logs from each individual host/device
- Auditing/compliance (yuck!)
Whether you’re a small business with a couple ESXi hosts and 10 guests or if you have 50 ESXi hosts and 2,000 guests the conversation is the same. If something breaks or is compromised how quickly, easily, and precisely can you determine what happened? If your ESXi hosts run from removable media and you have the syslog directory mapped to a SAN volume, then you need to go hunt down where each host is saving its logs, grab what’s there, move it to a PC, search, etc. This is the same with a Microsoft Windows system – you need to hunt down where an issue occurred and start combing the event viewer. Half the time the things you’re looking for on a Microsoft Windows machine are not even logged by default!
So, if a consolidated logging solution is so glorious what keeps us from implementing one immediately? Price and complexity. These two factors are what keep most mid-level management from rushing into a logging solution. Let’s touch on these two factors.
Licensing: You probably don’t know you already own it
One of the biggest caveats with syslog servers is that you have two options: free or really, really expensive. Isn’t that the case with most enterprise solutions? Open source packages like Graylog are really good if you get down and dirty with their lingo – grok, extractors, etc. If you’re less DevOps and more PleaseJustWorkOps, then Graylog and other open source solutions may not be for you unless you just want a place to store logs. Splunk, on the other hand, does ship with some templates, but I have found that more often than not Splunk reps want to not only sell you the product but also hours to develop reports/view/alerts/escalations for you. The biggest drawback to Splunk and other enterprise-class solutions, in my opinion, is that you’re licensed by gigabytes of ingested data. This is a very, very difficult way to estimate costs. If you have 200 nodes to monitor and some are firewalls and some are domain controllers you will have no idea what your licensing fees will be until you start logging. After you get your GB/day out-of-license email from Splunk and have to pay up, you need to either cut logging levels down or cut the number of nodes down. Event-based licensing is just unfair to the person buying the product unless they’re really tuned in to what they’re trying to accomplish.
With Splunk, they really know what they’re doing with the pricing model:
A perpetual license is a single payment that allows you to ingest syslogs at the given rate. You’ll notice that the typical SMB rate of 1 – 10GB/day of logging is the most expensive – that’s no coincidence. I believe there’s an additional 20% maintenance fee on the perpetual license due annually, as well. It’s extremely hard to estimate the license level needed because device output varies so much. For instance, you could probably monitor 12 – 24 edge switches and notice only 10% of the amount of logs generated when compared to a single Cisco ASA. This license model really caters to Splunk more than the consumer. A node-based model tends to be more fair to the consumer since you do not have to worry about how much or how little data is being ingested.
So what is a VMware shop to do?
You want to consolidate your vCenter, ESXi, network devices, and guest logs into a single pane of glass, but you are already paying a lot for your VMware licensing and you can’t really justify the cost of more licensing… enter vRealize LogInsight 4.
VMware has done a really cool thing – they’ve made vRealize LogInsight 4 (and 3.3, previously) available to anyone with vCenter Standard. In fact, you get 25-OSI (operating system instances) licenses per vCenter Standard license. That means you can monitor syslogs for a vCenter server, 5 ESXi hosts, and 19 other nodes included with your vCenter license! You can’t extend the 25-OSI license, but you could add additional OSI licenses (you can see retail prices around $1,000 for 25-OSIs). This is awesome, especially since vRealize LogInsight 4 is so… awesome. Sorry, I had to use the word awesome twice!
But really, once you break it down, while LogInsight may sound expensive you have to remember if you’re running vCenter Standard you already have 25-OSI licenses waiting. There’s no limit on GB/day or event/second for logging, either. So, considering the brief example of 12 edge switches and a few Cisco ASA logging to a Splunk server could cost around $4,500 + 20% (900)/year (or $1,800/year annual license), or you can use your existing vCenter license and buy 25-OSI packs as needed. Sounds like a good deal to me! And, you don’t need to buy any dashboards or pay any consultants to build you views,triggers, or alerts that should, in my opinion, ship with the product.
In addition to the included 25-OSI licenses you also get access to the LogInsight Marketplace which is chock full of really nice “Content Packs” for systems ranging from Cisco Nexus, F5 Load Balancers, NetApp Ontap, Nginx, Linux, Microsoft Active Directory, and the list goes on. That kind of offering will cost you a bundle in Splunk or take you a ton of time in Graylog.
What do you get with vRealize LogInsight 4?
In case I was too subtle in the text above, you get a lot of really cool stuff in vRealize LogInsight 4. I know, this sounds like a sales campaign but I don’t get anything from VMware for this – I am an end-user just like anyone else. Here’s my list of top features/benefits:
- Native integration with vSphere environment (including vRealize Operations!) – your vCenter server and all of your ESXi hosts get configured, automatically, to report into the LogInsight server
- Native clustering is supported with built-in load balancing featuring a virtual IP making scaling out simple
- Traditional syslog shipping to UDP 514 for network devices or any other type of device that supports syslog shipping
- Agent-based logging for guests including pre-built packages for Windows (MSI), and Linux (deb, rpm, bin) guests
- Active Directory sign-on integration and granular access control using AD users/groups or local users, etc.
- Pre-configured dashboards/templates for many Content Pack add-ons (vSphere, Microsoft Windows, etc.)
- Caching of queries with periodic roll-off to keep the most frequently used queries performance as best as possible
- Detailed statistics involing cluster details, ingestions per second, cache hits/misses, what type of even (API or syslog) is coming in, dropped events, etc.
- Alerting configuration based on queries – create a query or use an already configured one as an alert definition
- A simple and convenient way to filter hosts into categorized groups
So, if you’ve read through the list above, there’s really nothing I can think of that LogInsight 4 fails to provide. If you can think of anything that it is lacking please let me know, but as far as I can tell it’s hard to beat the value you get with VMware vRealize LogInsight 4 from a logging and security perspective.
In the next blog post I’ll talk about deploying a standalone and clustered environment, then move on to configuration and alerting. Stay tuned!