Overview
Beats are acting as agents in Client → Server configuration. One such agent is called filebeat processes log files and can upload them either to logstash or to elasticsearch directly. The installation of a filebeat can be done either file YUM or via downloading a package.
Installation
Let's install filebeat again using YUM as follows:
[root@psqlxlb yum.repos.d]# yum install filebeat Loaded plugins: fastestmirror base | 3.6 kB 00:00:00 elasticsearch-6.x | 1.3 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/2): extras/7/x86_64/primary_db | 147 kB 00:00:00 (2/2): elasticsearch-6.x/primary | 67 kB 00:00:00 Loading mirror speeds from cached hostfile * base: ftp.hosteurope.de * extras: ftp.hosteurope.de * updates: ftp.hosteurope.de elasticsearch-6.x 180/180 Resolving Dependencies --> Running transaction check ---> Package filebeat.x86_64 0:6.2.4-1 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================================================================== Installing: filebeat x86_64 6.2.4-1 elasticsearch-6.x 12 M Transaction Summary ====================================================================================================================================================================================================================== Install 1 Package Total download size: 12 M Installed size: 49 M Is this ok [y/d/N]: y Downloading packages: warning: /var/cache/yum/x86_64/7/elasticsearch-6.x/packages/filebeat-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY========================= ] 2.8 MB/s | 10 MB 00:00:00 ETA Public key for filebeat-6.2.4-x86_64.rpm is not installed filebeat-6.2.4-x86_64.rpm | 12 MB 00:00:03 Retrieving key from https://artifacts.elastic.co/GPG-KEY-elasticsearch Importing GPG key 0xD88E42B4: Userid : "Elasticsearch (Elasticsearch Signing Key) <[email protected]>" Fingerprint: 4609 5acc 8548 582c 1a26 99a9 d27d 666c d88e 42b4 From : https://artifacts.elastic.co/GPG-KEY-elasticsearch Is this ok [y/N]: y Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : filebeat-6.2.4-1.x86_64 1/1 Verifying : filebeat-6.2.4-1.x86_64 1/1 Installed: filebeat.x86_64 0:6.2.4-1 Complete!
After the agent is installed we can configure it to connect to the elasticsearch directly by editing the configuration file (/etc/filebeat/filebeat.yml)
Please remember that the configuration file is in YML format, therefore each space “ ” counts :)
Configuration
There are couple values we have to change:
Upload System Data to Elasticsearch
In case we want to upload to Elasticsearch directly for indexing, we have to edit the elasticsearch paragraph:
output.elasticsearch:
- hosts: [“IP_OF_THE_ELASTICSEARCH_SERVER:9200”]
- username: “elastic”
- password: “elasticPassword”
After the configurations are changed, don't forget to disable the firewalld (or edit the configuration of the firewall)
[root@psqlxlb filebeat]# service filebeat start Starting filebeat (via systemctl): [ OK ] [root@psqlxlb filebeat]#
This will ONLY enable the agent but wont upload any metrics, in order to monitor the host we have to edit 2 variables in the filebeat.yml configuration file:
Please ssure that the xpack.monitoring.enabled is set to true.
[root@mysqlmaster filebeat]# cat filebeat.yml | grep monitor # filebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # Set to true to enable the monitoring reporter. xpack.monitoring.enabled: true xpack.monitoring.elasticsearch: [root@mysqlmaster filebeat]#
You should be able to see this in Kibana interface now:
Upload MySQL Data to Elasticsearch
After the filebeat is configured we have enable the mysql plugin as follows:
[root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules enable mysqld Module mysqld doesn't exists! [root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules enable mysql Enabled mysql [root@mysqlmaster yum.repos.d]# /usr/bin/filebeat modules list Enabled: mysql Disabled: apache2 auditd icinga kafka logstash nginx osquery postgresql redis system traefik
Once the plugin is enabled we can load the elasticsearch data:
[root@mysqlmaster filebeat]# /usr/bin/filebeat setup -e 2018-06-05T10:12:52.874-0400 INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] 2018-06-05T10:12:52.874-0400 INFO instance/beat.go:475 Beat UUID: ddba7bc3-7bc5-4450-bebf-740353ae0603 2018-06-05T10:12:52.874-0400 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4 2018-06-05T10:12:52.874-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200 2018-06-05T10:12:52.875-0400 INFO pipeline/module.go:76 Beat name: mysqlmaster 2018-06-05T10:12:52.875-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200 2018-06-05T10:12:52.877-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4 2018-06-05T10:12:53.019-0400 INFO template/load.go:73 Template already exists and will not be overwritten. Loaded index template 2018-06-05T10:12:53.019-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200 2018-06-05T10:12:53.020-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4 2018-06-05T10:12:53.020-0400 INFO kibana/client.go:69 Kibana url: http://192.168.0.13:5601 2018-06-05T10:13:45.286-0400 INFO instance/beat.go:583 Kibana dashboards successfully loaded. Loaded dashboards 2018-06-05T10:13:45.286-0400 INFO elasticsearch/client.go:145 Elasticsearch url: http://192.168.0.13:9200 2018-06-05T10:13:45.288-0400 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4 Loaded machine learning job configurations [root@mysqlmaster filebeat]#
Set up the mysql configuration
Configure the configure for mysql in: /etc/fileabeat/modules.d/mysql.yml
[root@mysqlmaster modules.d]# cat mysql.yml
- module: mysql
error:
enabled: true
var.paths: ["/var/log/mysqld.log*"]
slowlog:
enabled: false
var.paths: ["/var/log/mysql/mysql-slow.log*"]
[root@mysqlmaster modules.d]#
Restart the filebeat and you should be able to see the metrics in Kibana
[root@mysqlmaster filebeat]# service filebeat restart Restarting filebeat (via systemctl): [ OK ] [root@mysqlmaster filebeat]#