Overview
In order to configure basic ELK stack configuration, as we already explained, we have to configure:
- Elasicsearch
- Kibana
- Logstash
Let's start with the configuration of Elastic & Kibana, since these 2 can exist by themselves.
Elasticsearch
Elasticsearch is the hearth of the ELK stack. The job of the elasticsearch is to index the documents passed to it by the logstash OR by a beat agent: filebeat, packetbeat, etc. The installation of the Elasticsearch is very easy via YUM. We will use YUM since it is the most easy, however on system on which you don't have access to the internet, feel free to download the package manually: In order to install it via YUM, we have to use the following repository:
[root@mykibana elasticsearch]# cd /etc/yum.repos.d/ [root@mykibana yum.repos.d]# ls -alrt total 44 -rw-r--r--. 1 root root 226 May 28 05:20 elasticsearch.repo drwxr-xr-x. 2 root root 213 May 28 05:20 . drwxr-xr-x. 82 root root 8192 May 30 08:26 .. [root@mykibana yum.repos.d]# cat elasticsearch.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md [root@mykibana yum.repos.d]#
Installation
Installation of the Elasticsearch is done easily after the yum repository creation.
[root@mykibana /]# yum install elasticsearch Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: ftp.hosteurope.de * extras: ftp.hosteurope.de * updates: ftp.hosteurope.de Resolving Dependencies --> Running transaction check ---> Package elasticsearch.noarch 0:6.2.4-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================================================================== Installing: elasticsearch noarch 6.2.4-1 elasticsearch-6.x 28 M Transaction Summary ====================================================================================================================================================================================================================== Install 1 Package Total download size: 28 M Installed size: 31 M Is this ok [y/d/N]: y Downloading packages: elasticsearch-6.2.4.rpm | 28 MB 00:00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Creating elasticsearch group... OK Creating elasticsearch user... OK Installing : elasticsearch-6.2.4-1.noarch 1/1 ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Verifying : elasticsearch-6.2.4-1.noarch 1/1 Installed: elasticsearch.noarch 0:6.2.4-1 Complete! [root@mykibana /]#
The configuration files of elasticsearch is at:
- YML File - /etc/elasticsearch/elasticsearch.yml
Define a host
By default the elasticsearch will open on hos: 127.0.0.1:9200 and the Kibana will be available on: 127.0.0.1:5601 which sometimes MIGHT no but convenient, you know, if you want to connect from outside via HTTP/HTTPS client :) That's why, feel free to change the hostname to a externally visible one, for example: 192.168.0.13 or any visible to you:
[root@mykibana elasticsearch]# grep "host" elasticsearch.yml network.host: 192.168.0.13 <-That will try to bind the elasticsearch to this IP and port: 9200
After the change is done, start the service:
===Service start=== [root@mykibana bin]# service elasticsearch start Starting elasticsearch (via systemctl): [ OK ] [root@mykibana bin]#
Install X-PACK on Elasticsearch
We have to install X-PACK on both Elasticsearch and Kibana. The installation is also rather simple:
[root@mykibana bin]# /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack -> Downloading x-pack from elastic [=================================================] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission \\.\pipe\* read,write * java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission setFactory * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission createPolicy.JavaPolicy * java.security.SecurityPermission getPolicy * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setPolicy * java.util.PropertyPermission * read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin forks a native controller @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This plugin launches a native controller that is not subject to the Java security manager nor to system call filters. Continue with installation? [y/N]y Elasticsearch keystore is required by plugin [x-pack-security], creating... -> Installed x-pack with: x-pack-core,x-pack-deprecation,x-pack-graph,x-pack-logstash,x-pack-ml,x-pack-monitoring,x-pack-security,x-pack-upgrade,x-pack-watcher [root@mykibana bin]#
Password generation
Once the X-PACK is installed we have to configure passwords for all users: elastic, kibana and logstash_system. That can easily be done using the x-pack utility:
[root@mykibana x-pack]# /usr/share/elasticsearch/bin/x-pack/setup-passwords auto Changed password for user kibana PASSWORD kibana = password1 Changed password for user logstash_system PASSWORD logstash_system = password2 Changed password for user elastic PASSWORD elastic = password3
Kibana
The installation of Kibana is as simple as elasticsearch maybe:
[root@mykibana log]# yum install kibana Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: ftp.hosteurope.de * extras: ftp.hosteurope.de * updates: ftp.hosteurope.de Resolving Dependencies --> Running transaction check ---> Package kibana.x86_64 0:6.2.4-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================================================================== Installing: kibana x86_64 6.2.4-1 elasticsearch-6.x 83 M Transaction Summary ====================================================================================================================================================================================================================== Install 1 Package Total download size: 83 M Installed size: 292 M Is this ok [y/d/N]: y Downloading packages: kibana-6.2.4-x86_64.rpm | 83 MB 00:00:16 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : kibana-6.2.4-1.x86_64 1/1 Verifying : kibana-6.2.4-1.x86_64 1/1 Installed: kibana.x86_64 0:6.2.4-1 Complete! [root@mykibana log]#
Install X-PACK on Kibana
We have to install X-PACK to Kibana, the same way as in elasticsearch:
[root@mykibana bin]# /usr/share/kibana/bin/kibana-plugin install x-pack -> Downloading x-pack from elastic [=================================================] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission \\.\pipe\* read,write * java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission setFactory * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission createPolicy.JavaPolicy * java.security.SecurityPermission getPolicy * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setPolicy * java.util.PropertyPermission * read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin forks a native controller @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This plugin launches a native controller that is not subject to the Java security manager nor to system call filters. Continue with installation? [y/N]y Elasticsearch keystore is required by plugin [x-pack-security], creating... -> Installed x-pack with: x-pack-core,x-pack-deprecation,x-pack-graph,x-pack-logstash,x-pack-ml,x-pack-monitoring,x-pack-security,x-pack-upgrade,x-pack-watcher [root@mykibana bin]#
Configuration
In order to link Kibana and Elasticsearch we have to edit the kibana configuration file, by providing the password for the kibana in the kibana.yml file:
elasticsearch.username: "kibana" elasticsearch.password: "password1"
With all that, we can finally start Kibana and verify the port is open:
[root@mykibana elasticsearch]# service kibana restart kibana is started [root@mykibana elasticsearch]# netstat -na | grep 5601 tcp 0 0 192.168.0.13:5601 0.0.0.0:* LISTEN tcp 0 0 192.168.0.13:5601 192.168.0.4:47879 ESTABLISHED [root@mykibana elasticsearch]# cd /etc/elasticsearch/
Congrats, you have just installed Elasticsearch + Kibana, on their own, you can do basic monitoring of the host server :) without any specific log processing. In order to configure log processing we have to configure logstash.