Installing the Elastic Stack (ELK 5.4) on Windows Server 2016

Elasticsearch, Logstash, and Kibana from Elastic are the three major products that make up the Elastic Stack (what used to be called ELK Stack). It represents a hugely versatile set of tools that can be used to collect and analyze data from just about source. There are tons of products in this space, so why bother with Elastic Stack? Logging and event management solutions are often expensive, and generally not where SMBs want to spend their limited IT budget. Elastic Stack is an open-source solution, providing a huge amount of configurability and customization, creating quite a lot of bang for your buck - if you can invest the time to install and configure it. And whether you're operating in an all-Windows environment or simply not interested in working with Linux, there are plenty good reasons to install your Elastic Stack on Windows Server. Let's take a look.

This guide will also work with Windows Server 2012 R2. The process is exactly the same.

Architecture

Before diving in to the installation portion I wanted to take a second to review the architecture of the Elastic Stack that we'll be building. If you're a visual learner like me, it may aid in understanding how these components fit together and interact with one another.

The Elastic Stack allows you to visualize data from myriad sources. For our simplified example, consider Windows Event Logs.

1. An agent program installed on our server captures Event Log data and ships it to Logstash. In this guide we're using Elastic Beats as our agents. Specifically for Windows Event Logs we will use Winlogbeat. (Note that agent programs are not required for all data sources; network appliance syslogs, for example.)
2. Logstash receives the input from Winlogbeat, considers any filters and performs any transforms that we've defined, and ships the data to Elasticsearch.
3. Elasticsearch indexes and centrally stores the data from Logstash, and makes it available for searching and analytics.
4. Kibana connects to Elasticsearch to provide a friendly user interface for filtering and visualizing your data.

For this guide we'll be installing all three applications on one Windows Server 2016 VM, however they do support distributed installation.

Download Installation Files

Start by downloading Elasticsearch, Logstash, and Kibana from the Elastic website. While we're there, let's also download Filebeat, Packetbeat, Winlogbeat, and Topbeat (or Metricbeat now, as Topbeat has been deprecated). We'll need those later. Choose the option for Windows x64, Windows, or ZIP as appropriate.

https://www.elastic.co/downloads

Then, download the Java Development Kit (JDK) for Windows x64.

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

You’ll also need WinPcap if you want to use Packetbeat to send network data to your Elastic Stack.

https://www.winpcap.org/install/

Lastly, download the Non-Sucking Service Manager (NSSM).

https://nssm.cc/download

Place everything into a folder on your server's C drive. I called mine ELK-Stack and that's what you'll see referenced throughout this guide.

Install Java Development Kit (JDK)

Once everything is downloaded, we'll start by installing the JDK with the default options. Make note of the installation directory as we’re going to create an environment variable with that path in the next step.

With the JDK installed, we need to create an environment variable that points to the program directory. Browse to System Properties > Advanced Tab, and select Environment Variables…

Click New... at the bottom to add a new System Variable. Name it JAVA_HOME and provide the path of the JDK installation that we just completed.

Click OK to finish, and OK again to close System Properties.

Installing Elastic Stack

With the JDK installed, let’s go ahead and extract all of our zip packages to the ELK-Stack folder. I’ve removed the version info from my folder names to neaten it up a bit, but it isn’t necessary. Just make note of the file paths when following this guide if yours are named differently.

With everything extracted, let's get started installing each application as a service, so we can control them like other services and have them launch at Windows startup.

Elasticsearch

Of the three applications in the Elastic Stack, Elasticsearch is the only one that is able to install itself as a service out of the box. In order to achieve that, we want to run the elasticsearch-service.bat file with the install option. That can be done at a command prompt as in the screenshot below, or by using the Invoke-Expression cmdlet within Powershell. Elastic uses the Powershell cmdlets in their Windows documentation so I’ll use those as well for the remainder of this post.

elasticsearch-service install

Performing the Elasticsearch service installation with Powershell:

Invoke-Expression -Command "C:\ELK-Stack\elasticsearch\bin\elasticsearch-service install"

After running the install command you should see a response indicating that the service has been installed successfully. Next we need to tweak the properties for the service by launching the service manager. Use the following Powershell command.

Invoke-Expression -Command "C:\ELK-Stack\elasticsearch\bin\elasticsearch-service manager"




On the Elasticsearch Properties dialogue, change the Startup Type to Automatic and start the service.

This is also where you can adjust the Java memory settings, which will be useful when we have more devices logging to our Elastic Stack. For now, we can leave the Java settings alone.

Once you've made your changes, click OK to close the Elasticsearch properties.

With the Elasticsearch service started, go ahead and open a web browser on your server and point it to http://127.0.0.1:9200. In Chrome you should see results similar to the following image. Depending on your settings, Internet Explorer may just prompt you to download a json file (the contents of the file will be the same as what’s seen in Chrome). This is fine and there's no need to keep the file. The purpose of the test is just to validate that Elasticsearch is reachable on port 9200.

With Elasticsearch installed, it’s time to move on to Logstash.

Logstash

Logstash doesn’t provide means to install it as a Windows service like Elasticsearch did, so we’ll use the Non-Sucking Service Manager (NSSM) to help us with that. First, we’re going to need a config file for Logstash that we'll point to when we’re setting up the service. For that we’ll use a text editor like Notepad to create a new file in the \logstash\bin directory and name it config.json. We need to give it a simple configuration to start with or Logstash won't start properly, so let's put the following into our new config.json file:

input {
  beats {
   port => 5044
   type => "log"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

This config file sets us up to use the Beats plugin for Logstash, so we'll be able to use the Elastic Beats shippers to send data to our Elastic Stack later on. Note that if you wanted to host Logstash on a separate server from Elasticsearch, this config file is where you would point the output to somewhere other than localhost. For now let's move on and use Powershell to launch NSSM and install our new service named Logstash.

Invoke-Expression -Command "C:\ELK-Stack\nssm\win64\nssm install Logstash"

In the NSSM service installer window > Application tab we’ll configure the path to the logstash.bat file and the config.json file as shown.

On the Details tab, give the new service an appropriate name and description.

Lastly, on the Dependencies tab add elasticsearch-service-x64, and click Install Service.

You should see a message that the service was installed successfully.

Before we move on, let's install the Beats input plugin for Logstash, as we'll be using Beats to ship data into our Elastic Stack. Note that there are dozens of input plugins available for Logstash. Use Powershell to install the Beats input plugin now.

Invoke-Expression -Command "C:\ELK-Stack\logstash\bin\logstash-plugin.bat install logstash-input-beats"

It’s time to move on to Kibana.

Kibana

For Kibana, we’ll do the same as above with NSSM.

Invoke-Expression -Command "C:\ELK-Stack\nssm\win64\nssm install Kibana"

We don’t need to pass any arguments to Kibana, so we’ll leave that field blank this time.

Provide a name and description.

Add both elasticsearch-service-x64 and logstash as dependencies, and click Install Service.

Kibana's default options can be changed by modifying the \kibana\config\kibana.yml file. This is where you could update the value for elasticsearch.url to something other than localhost, if Kibana is on a different server from Elasticsearch. Since we're keeping the defaults for now, there's no need to edit this file currently.

With Kibana installed successfully, go ahead and make sure our three new services are started. Due to the dependencies we setup, we’ll need to start Elasticsearch first, then Logstash, and finally Kibana. If all has gone according to plan, you should now be able to open a browser and browse to http://127.0.0.1:5601 and see Kibana’s initial setup page. While I was testing I found that Kibana can take a minute or two to load up after the service is started, before the website is accessible. If it doesn't come up right away just give it a second.

And that's it for the initial setup of the Elastic Stack on Windows Server 2016. Not very useful yet, is it? Let's talk about how to setup Beats for shipping data to your Elastic Stack.

Elastic Beats

As discussed above, Elastic Beats are the agent programs that we'll use to ship data into Logstash. We downloaded four of them at the beginning of this article so we'll go ahead and install those on our server now. Elastic provides Powershell scripts for installing each Beat as a Windows service, so we just need to execute each script in a Powershell window.

Filebeat - for monitoring log files such as IIS logs.

PowerShell.exe -ExecutionPolicy Bypass -File C:\ELK-Stack\filebeat\ .\install-service-filebeat.ps1

Packetbeat* - for monitoring network traffic.

PowerShell.exe -ExecutionPolicy Bypass -File C:\ELK-Stack\packetbeat\ .\install-service-packetbeat.ps1

Topbeat -for monitoring resource usage.

PowerShell.exe -ExecutionPolicy Bypass -File C:\ELK-Stack\topbeat\ .\install-service-topbeat.ps1

Winlogbeat - for monitoring Windows Event Logs

PowerShell.exe -ExecutionPolicy Bypass -File C:\ELK-Stack\winlogbeat\ .\install-service-winlogbeat.ps1

*For Packetbeat, you'll need to install WinPcap as well on the host(s) that you'll be monitoring network traffic for. If you don't want to use Packetbeat on your Elastic Stack server, skip that install and also skip WinPcap.

Install WinPcap

Now we'll install WinPcap so we can send network data to our Elastic Stack with Packetbeat. We'll select the default options here as well.

There's no additional configuration needed for WinPcap.

Beats Configuration

Within the program directory for each of the Beats we just installed you'll notice a yml configuration file. If you take a look inside, you'll probably notice that the default Beats configuration points to Elasticsearch on port 9200. What we want to do instead is point the Beats to the input plugin that we installed for Logstash earlier on. To do so, we simply comment out the hosts configuration for output.elasticsearch and uncomment the hosts line for output.logstash. The position of these configurations is slightly different in each Beat configuration file but the necessary change is the same.

Here's an example after I've commented the Elasticsearch host and uncommented the same for Logstash. Be sure to comment/uncomment both the output. line and the hosts. line in each config file.

That's all the configuration we need at this stage. Go ahead and start each of your Beats Windows services.

Bringing it all together

Back to Kibana now, it's time to configure our indexes so we can visualize data from each of the Beats that we just turned on. At the Configure an index pattern screen we're going to add an index pattern for each of our Beats. Clear out logstash-* which populates by default, and add each of the following:

• packetbeat-*
• topbeat-*
• winlogbeat-*
• filebeat-*

Select @timestamp for the Time-field name, then click Create to create the index pattern.

After you've added the first index pattern, you'll need to use the plus icon to add additional indexes.

When you get to Filebeat (there's a reason I listed it last) you'll notice that adding it is unsuccessful. This is expected. Why?

Filebeat expects a log file (or files) as input. Since we're on Windows Server 2016 and we haven't modified this part of the config, it's fairly unlikely there's anything located at /var/log to ingest and push to Logstash. For now, we're OK with this and we'll move on using the other Beats as examples.

Click on the Discover tab in Kibana and let's look at what we have. By default, based on the Beats that we added, we should see entries from Packetbeat. If we click the drop down over packetbeat-* we can also select topbeat-* and winlogbeat-* to view that data.

Now we have the foundation in place for our Elastic Stack on Windows Server 2016. We can install the Beats agents on other servers and point them back to Logstash to aggregate data from our entire infrastructure, if we choose.