======Overview======
Beats are acting as agents in Client -> Server configuration. One such agent is called filebeat processes log files and can upload them either to logstash or to elasticsearch directly.
The installation of a filebeat can be done either file YUM or via downloading a package.
{{ :beats.jpg?600 |}}
======Installation======
Let's install filebeat again using YUM as follows:
[root@psqlxlb yum.repos.d]# yum install filebeat
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
elasticsearch-6.x | 1.3 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/2): extras/7/x86_64/primary_db | 147 kB 00:00:00
(2/2): elasticsearch-6.x/primary | 67 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: ftp.hosteurope.de
* extras: ftp.hosteurope.de
* updates: ftp.hosteurope.de
elasticsearch-6.x 180/180
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
======================================================================================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================================================================================
Installing:
filebeat x86_64 6.2.4-1 elasticsearch-6.x 12 M
Transaction Summary
======================================================================================================================================================================================================================
Install 1 Package
Total download size: 12 M
Installed size: 49 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/elasticsearch-6.x/packages/filebeat-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY========================= ] 2.8 MB/s | 10 MB 00:00:00 ETA
Public key for filebeat-6.2.4-x86_64.rpm is not installed
filebeat-6.2.4-x86_64.rpm | 12 MB 00:00:03
Retrieving key from https://artifacts.elastic.co/GPG-KEY-elasticsearch
Importing GPG key 0xD88E42B4:
Userid : "Elasticsearch (Elasticsearch Signing Key) "
Fingerprint: 4609 5acc 8548 582c 1a26 99a9 d27d 666c d88e 42b4
From : https://artifacts.elastic.co/GPG-KEY-elasticsearch
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-6.2.4-1.x86_64 1/1
Verifying : filebeat-6.2.4-1.x86_64 1/1
Installed:
filebeat.x86_64 0:6.2.4-1
Complete!
After the agent is installed we can configure it to connect to the elasticsearch directly by editing the configuration file (/etc/filebeat/filebeat.yml)
**Please remember that the configuration file is in YML format, therefore each space " " counts :) **
======Configuration======
There are couple values we have to change:
=====Upload System Data to Elasticsearch=====
In case we want to upload to Elasticsearch directly for indexing, we have to edit the elasticsearch paragraph:
output.elasticsearch:
* hosts: ["IP_OF_THE_ELASTICSEARCH_SERVER:9200"]
* username: "elastic"
* password: "elasticPassword"
After the configurations are changed, don't forget to disable the firewalld (or edit the configuration of the firewall)
[root@psqlxlb filebeat]# service filebeat start
Starting filebeat (via systemctl): [ OK ]
[root@psqlxlb filebeat]#
This will **ONLY** enable the agent but wont upload any metrics, in order to monitor the host we have to edit 2 variables in the filebeat.yml configuration file:
Please ssure that the xpack.monitoring.enabled is set to true.
[root@mysqlmaster filebeat]# cat filebeat.yml | grep monitor
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# Set to true to enable the monitoring reporter.
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch:
[root@mysqlmaster filebeat]#
You should be able to see this in Kibana interface now:
=====Upload MySQL Data to Elasticsearch=====
After the filebeat is configured we have enable the mysql plugin as follows:
[root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules enable mysqld
Module mysqld doesn't exists!
[root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules enable mysql
Enabled mysql
[root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules list
Enabled:
mysql
Disabled:
apache2
auditd
icinga
kafka
logstash
nginx
osquery
postgresql
redis
system
traefik
Once the plugin is enabled we can load the elasticsearch data:
[root@mysqlmaster filebeat]# /usr/bin/filebeat setup -e
2018-06-05T10:12:52.874-0400 INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-06-05T10:12:52.874-0400 INFO instance/beat.go:475 Beat UUID: ddba7bc3-7bc5-4450-bebf-740353ae0603
2018-06-05T10:12:52.874-0400 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-06-05T10:12:52.874-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200
2018-06-05T10:12:52.875-0400 INFO pipeline/module.go:76 Beat name: mysqlmaster
2018-06-05T10:12:52.875-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200
2018-06-05T10:12:52.877-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-06-05T10:12:53.019-0400 INFO template/load.go:73 Template already exists and will not be overwritten.
Loaded index template
2018-06-05T10:12:53.019-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200
2018-06-05T10:12:53.020-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-06-05T10:12:53.020-0400 INFO kibana/client.go:69 Kibana url: http://192.168.0.13:5601
2018-06-05T10:13:45.286-0400 INFO instance/beat.go:583 Kibana dashboards successfully loaded.
Loaded dashboards
2018-06-05T10:13:45.286-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200
2018-06-05T10:13:45.288-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
Loaded machine learning job configurations
[root@mysqlmaster filebeat]#
===Set up the mysql configuration===
Configure the configure for mysql in: /etc/fileabeat/modules.d/mysql.yml
[root@mysqlmaster modules.d]# cat mysql.yml
- module: mysql
error:
enabled: true
var.paths: ["/var/log/mysqld.log*"]
slowlog:
enabled: false
var.paths: ["/var/log/mysql/mysql-slow.log*"]
[root@mysqlmaster modules.d]#
Restart the filebeat and you should be able to see the metrics in Kibana
[root@mysqlmaster filebeat]# service filebeat restart
Restarting filebeat (via systemctl): [ OK ]
[root@mysqlmaster filebeat]#
=====Upload to Logstash=====