数据流走向
想要采集的数据 ---> 通过filebeat软件采集 ---> 数据打到ES集群里--->通过 Kibana图形化展示- 1.下载Filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.28-amd64.deb- 2.安装Filebeat
[root@elk92 ~]# dpkg -i filebeat-7.17.28-amd64.deb
- 3.编写Filebeat配置文件
[root@elk92 ~]# cat /etc/filebeat/config/01-log-to-console.yaml
# 定义数据从哪里来
filebeat.inputs:
# 指定数据源的类型是log，表示从文件读取数据
- type: log
# 指定文件的路径
paths:
- /tmp/student.log
# 定义数据到终端
output.console:
pretty: true
[root@elk92 ~]#
- 4.启动Filebeat实例
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/01-log-to-console.yaml- 5.向源数据文件(“/tmp/student.log”)写入测试数据
echo 1111111111111>/tmp/student.log温馨提示:
– 1.filebeat默认是按行采集数据;
– 2.filebeat默认会在”/var/lib/filebeat”目录下记录已经采集的文件offset信息，以便于下一次采集接着该位置继续采集数据;所以一般在采集新数据的时候，先执行一下如下操作：
[root@elk92 ~]# rm -rf /var/lib/filebeat/* #当然第一次执行不需要清除数据filebeat采集日志实战案例：
1.编写配置文件
[root@elk92 ~]# vim /etc/filebeat/config/01-log-to-es-custom-index.yaml
filebeat.inputs:
- type: log
paths:
- /tmp/student.log
# 将数据写入到ES集群
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
# 指定索引的名称
index: oldboyedu-linux95-tmp-%{+yyyy.MM.dd}
# 禁用索引生命周期管理(index lifecycle management，ILM)
# 如果启用了此配置，则忽略自定义索引的所有信息
setup.ilm.enabled: false
# 定义索引模板(就是创建索引的规则)的名称
setup.template.name: "custom-index-example"
# 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
setup.template.pattern: "custom-index-example-*"
# 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
# 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
setup.template.overwrite: true
# 定义索引模板的规则信息
setup.template.settings:
# 指定索引能够创建的分片的数量
index.number_of_shards: 5
# 指定每个分片有多少个副本
index.number_of_replicas: 0
[root@elk92 ~]#
[root@elk92 ~]# rm -rf /var/lib/filebeat/
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/01-log-to-es-custom-index.yaml- 2.项目案例: filebeat采集nginx实战案例
1.安装nginx服务
[root@elk92 ~]# apt -y install nginx
2.启动nginx服务
[root@elk92 ~]# systemctl enable --now nginx
3.访问测试
[root@elk92 ~]# curl 10.0.0.92
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@elk92 ~]#
4.查看日志信息
[root@elk92 ~]# cat /var/log/nginx/access.log
10.0.0.92 - - [11/Mar/2025:18:27:23 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
[root@elk92 ~]#
5.使用Filebeat采集nginx日志
[root@elk92 ~]# cat /etc/filebeat/config/02-nginx-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/access.log*
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-nginx-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk92 ~]#
6.启动Filebeat实例
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/02-nginx-to-es.yaml
7.kibana查看数据
8.发送测试数据
[root@elk93 ~]# for i in `seq 10`;do curl http://10.0.0.92;done
filebeat多实例
1.启动实例1
filebeat -e -c /etc/filebeat/config/01-log-to-console.yaml --path.data /tmp/xixi
2.启动实例2
filebeat -e -c /etc/filebeat/config/02-log-to-es.yaml --path.data /tmp/haha
说白了就是指定要采集的文件路径即可，想同时启动几个就启动几个Filebeat模块管理
- 1 查看支持的模块列表
[root@elk92 ~]# filebeat modules list
Enabled: # 表示启用的模块
Disabled: # 表示禁用的模块列表
activemq
apache
auditd
aws
awsfargate
azure
barracuda
...- 2 启动多个模块
[root@elk92 ~]# filebeat modules enable mysql nginx redis mongodb
Enabled mysql
Enabled nginx
Enabled mongodb
Enabled redis
[root@elk92 ~]#
[root@elk92 ~]# ll -1 /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 297 Feb 14 00:58 /etc/filebeat/modules.d/mongodb.yml
-rw-r--r-- 1 root root 472 Feb 14 00:58 /etc/filebeat/modules.d/mysql.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
-rw-r--r-- 1 root root 567 Feb 14 00:58 /etc/filebeat/modules.d/redis.yml
[root@elk92 ~]#
[root@elk92 ~]# filebeat modules list
Enabled: # 发现启用的模块成功啦~
mongodb
mysql
nginx
redis
Disabled:
activemq
apache
auditd
aws
awsfargate
...- 3.禁用多个模块
[root@elk92 ~]# filebeat modules disable mysql mongodb redis
Disabled mysql
Disabled mongodb
Disabled redis
[root@elk92 ~]#
[root@elk92 ~]# ll -1 /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
[root@elk92 ~]#
[root@elk92 ~]# filebeat modules list
Enabled:
nginx
Disabled:
activemq
apache
auditd
aws
...
- 4.编写Filebeat的配置文件
[root@elk92 ~]# vim /etc/filebeat/config/05-modules-nginx-to-es.yaml
# 配置模块功能
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml # 加载Filebeat配置文件目录的"modules.d"子目录下的所有"*.yml"文件
reload.enabled: true # 是否支持热加载
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-nginx-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk92 ~]# - 5.修改nginx模块文件
[root@elk92 ~]# vim /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log*"]
error:
enabled: false
ingress_controller:
enabled: false
[root@elk92 ~]# - 6.启动Filebeat实例
[root@elk92 ~]# rm -rf /var/lib/filebeat/
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-modules-nginx-to-es.yaml - 7.kibana出图展示
Filebeat采集tomcat日志
- 1.二进制部署tomcat环境
1.1 下载tomcat软件包
wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.5/bin/apache-tomcat-11.0.5.tar.gz
1.2 解压软件包
[root@elk93 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
1.3 配置环境变量
[root@elk93 ~]# cat /etc/profile.d/tomcat.sh
#!/bin/bash
export JAVA_HOME=/usr/share/elasticsearch/jdk
export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.5
export PATH=$PATH:$JAVA_HOME/bin:$TOMCAT_HOME/bin
[root@elk93 ~]#
[root@elk93 ~]# source /etc/profile.d/tomcat.sh
[root@elk93 ~]#
[root@elk93 ~]# java --version
openjdk 22.0.2 2024-07-16
OpenJDK Runtime Environment (build 22.0.2+9-70)
OpenJDK 64-Bit Server VM (build 22.0.2+9-70, mixed mode, sharing)
[root@elk93 ~]#
1.4 修改tomcat配置文件
[root@elk93 ~]# vim /usr/local/apache-tomcat-11.0.5/conf/server.xml
...
<Host name="tomcat.oldboyedu.com" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat.oldboyedu.com_access_log" suffix=".json"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","request":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","http_user_agent":"%{User-Agent}i"}"/>
</Host>
[root@elk93 ~]#
1.5 启动tomcat
[root@elk93 ~]# catalina.sh start
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.5
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.5
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.5/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.5/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.5/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[root@elk93 ~]#
[root@elk93 ~]#
[root@elk93 ~]# ss -ntl | grep 8080
LISTEN 0 100 *:8080 *:*
[root@elk93 ~]#
1.6 访问测试
http://tomcat.oldboyedu.com:8080/- 2.配置Filebeat采集日志
2.1 主配置加载模块文件
[root@elk93 ~]# cat /tmp/modules-tomcat-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-tomcat-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]#
[root@elk93 ~]#
2.2 启动tomcat模块
[root@elk93 ~]# filebeat modules enable tomcat
Enabled tomcat
[root@elk93 ~]#
[root@elk93 ~]# filebeat modules enable list
Module list doesn't exist!
[root@elk93 ~]#
[root@elk93 ~]# filebeat modules list
Enabled:
tomcat
Disabled:
...
2.3 修改tomcat的模块文件
[root@elk93 ~]# vim /etc/filebeat/modules.d/tomcat.yml
- module: tomcat
log:
enabled: true
var.input: file
var.paths:
- /usr/local/apache-tomcat-11.0.5/logs/*.json
var.tz_offset: +08:00
[root@elk93 ~]#
2.4 启动tomcat实例
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-to-es.yaml filebeat的processors处理器实战案例
- 1.编写Filebeat的配置文件
[root@elk93 ~]# cat /tmp/modules-tomcat-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
# 定义处理器
processors:
- decode_json_fields: # 对json字段进行解码
fields: ["event.original"] # 指定要解码的字段
target: "" # 将解码后的json数据放在根字段下
overwrite_keys: true # 覆盖已经有的字段，默认值为false
add_error_key: true # 保留解码错误信息
# 删除字段
- drop_fields:
# 当条件符合时才会执行当前的处理逻辑
when:
equals:
status: "404"
# 要删除的特定字段
fields: ["log.file.path"]
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-tomcat-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]# - 2.启动Filebeat实例
[root@elk93 ~]# rm -rf /var/lib/filebeat/
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-to-es.yaml - 3.测试验证
参考字段"log.file.path"。
参考链接:
https://www.elastic.co/guide/en/beats/filebeat/7.17/filtering-and-enhancing-data.htmlEFK分析web集群架构
- 1.web集群环境准备
1.1 91节点准备tomcat环境
[root@elk91 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
[root@elk91 ~]# echo 10.0.0.91 > /usr/local/apache-tomcat-11.0.5/webapps/ROOT/index.html
[root@elk91 ~]# catalina.sh start
1.2 93节点准备测试环境
[root@elk93 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local/
[root@elk93 ~]# echo 10.0.0.93 > /usr/local/apache-tomcat-11.0.5/webapps/ROOT/index.html
[root@elk93 ~]# catalina.sh start
1.3 92节点nginx代理tomcat服务
[root@elk92 ~]# vim /etc/nginx/conf.d/tomcat.conf
upstream webs {
server 10.0.0.91:8080;
server 10.0.0.93:8080;
}
server {
server_name tomcat.web.com;
location / {
proxy_pass http://webs;
}
}
1.4 热加载nginx服务
[root@elk92 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@elk92 ~]#
[root@elk92 ~]# nginx -s reload
[root@elk92 ~]#
1.5 访问页面测试
[root@elk92 ~]# for i in `seq 10`;do curl 10.0.0.92 -H 'host: tomcat.web.com'; sleep 0.1;done
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
10.0.0.91
10.0.0.93
[root@elk92 ~]#
别忘了在92节点做解析 10.0.0.92 tomcat.web.com
如果出现不了，请将"server.xml"文件还原，并重启tomcat服务即可。
[root@elk91 ~]# tail -100f /usr/local/apache-tomcat-11.0.5/logs/tomcat.oldboyedu.com_access_log.2025-03-12.json
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:05 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:07 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
...
[root@elk93 ~]# tail -100f /usr/local/apache-tomcat-11.0.5/logs/tomcat.oldboyedu.com_access_log.2025-03-12.json
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:06 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:08 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
{"clientip":"10.0.0.92","ClientUser":"-","authenticated":"-","AccessTime":"[12/Mar/2025:17:35:28 +0800]","request":"GET / HTTP/1.0","status":"200","SendBytes":"10","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}
...- 2.采集web集群日志
2.1 采集nginx日志
[root@elk92 ~]# cat /etc/filebeat/config/05-modules-nginx-to-es.yaml
# 配置模块功能
filebeat.config.modules:
# 加载Filebeat配置文件目录的"modules.d"子目录下的所有"*.yml"文件
path: ${path.config}/modules.d/*.yml
# 是否支持热加载
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-nginx-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk92 ~]#
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-modules-nginx-to-es.yaml
2.2 采集93节点的tomcat日志
2.2.1 启动Filebeat实例
[root@elk93 ~]# cat /tmp/modules-tomcat-project-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
processors:
- decode_json_fields:
fields: ["event.original"]
target: ""
overwrite_keys: true
add_error_key: true
# 转换数据类型
- convert:
# 定义字段的转换方式，其中from表示源字段，type表示要转换的类型。
# 如果想要将转换的数据存储到一个新字段中，则可以使用"to"关键字，不写则就地更新。
fields:
- {from: "SendBytes", type: "long"}
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-tomcat-project-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/modules-tomcat-project-to-es.yaml
2.2.2 启动tomcat模块
[root@elk93 ~]# vim /etc/filebeat/modules.d/tomcat.yml
- module: tomcat
log:
enabled: true
var.input: file
var.paths:
- /usr/local/apache-tomcat-11.0.5/logs/*.json
var.tz_offset: +08:00
[root@elk93 ~]#
2.3 采集91节点的tomcat日志
2.3.1 启动Filebeat实例
[root@elk91 ~]# cat /tmp/modules-tomcat-project-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
processors:
- decode_json_fields:
fields: ["event.original"]
target: ""
overwrite_keys: true
add_error_key: true
# 转换数据类型
- convert:
# 定义字段的转换方式，其中from表示源字段，type表示要转换的类型。
# 如果想要将转换的数据存储到一个新字段中，则可以使用"to"关键字，不写则就地更新。
fields:
- {from: "SendBytes", type: "long"}
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-tomcat-project-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk91 ~]#
[root@elk91 ~]# filebeat -e -c /tmp/modules-tomcat-project-to-es.yaml
2.3.2 启动tomcat模块
[root@elk91 ~]# filebeat modules enable tomcat
Enabled tomcat
[root@elk91 ~]#
[root@elk91 ~]# vim /etc/filebeat/modules.d/tomcat.yml
- module: tomcat
log:
enabled: true
var.input: file
var.paths:
- /usr/local/apache-tomcat-11.0.5/logs/*.json
var.tz_offset: +08:00
[root@elk91 ~]# Filebeat模块采集elasticsearch集群日志
- 1.启动Filebeat实例
[root@elk93 ~]# cat /tmp/modules-eslog-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-eslog-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/modules-eslog-to-es.yaml以上就是关于filebeat如何部署以及如何操作使用！
- 2.启动模块文件
[root@elk93 ~]# mv /etc/filebeat/modules.d/elasticsearch.yml{.disabled,}
[root@elk93 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 965 Mar 13 09:30 /etc/filebeat/modules.d/elasticsearch.yml
-rw-r--r-- 1 root root 737 Mar 12 17:45 /etc/filebeat/modules.d/tomcat.yml
[root@elk93 ~]#
- 3.kibana查询数据
GET custom-index-example-modules-eslog-2025.03.13/_search
{
"query": {
"match": {
"message": "4064141312"
}
}
, "_source": ["message"]
}Filebeat模块采集mysql集群日志
- 1.部署MySQL
1.1 下载MySQL软件包
wget https://dev.mysql.com/get/Downloads/MySQL-8.4/mysql-8.4.4-linux-glibc2.28-x86_64.tar.xz
1.2 解压软件包
[root@elk93 ~]# tar xf mysql-8.4.4-linux-glibc2.28-x86_64.tar.xz -C /usr/local/
1.3 准备启动脚本并授权
[root@elk93 ~]# cp /usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/support-files/mysql.server /etc/init.d/
[root@elk93 ~]# egrep "^basedir=|^datadir=" /etc/init.d/mysql.server
basedir=/usr/local/mysql844
datadir=/var/lib/mysql
[root@elk93 ~]#
[root@elk93 ~]# useradd -m mysql
[root@elk93 ~]#
[root@elk93 ~]# install -d /var/lib/mysql -o mysql -g mysql
[root@elk93 ~]#
[root@elk93 ~]# ll /var/lib/mysql/ -d
drwxr-xr-x 2 mysql mysql 4096 Mar 13 09:52 /var/lib/mysql//
[root@elk93 ~]#
1.4 准备配置文件
[root@elk93 ~]# cat /etc/my.cnf
[mysqld]
basedir=/usr/local/mysql844
datadir=/var/lib/mysql
socket=/tmp/mysql80.sock
port=3306
[client]
socket=/tmp/mysql80.sock
[root@elk93 ~]#
1.5 启动MySQL服务
[root@elk93 ~]# cat /etc/profile.d/mysql.sh
#!/bin/bash
export MYSQL_HOME=/usr/local/mysql844
export PATH=$PATH:$MYSQL_HOME/bin
[root@elk93 ~]#
[root@elk93 ~]# source /etc/profile.d/mysql.sh
[root@elk93 ~]#
[root@elk93 ~]# ln -svf /usr/local/mysql-8.4.4-linux-glibc2.28-x86_64/ /usr/local/mysql844
[root@elk93 ~]#
[root@elk93 ~]# mysqld --initialize-insecure --user=mysql --datadir=/var/lib/mysql --basedir=/usr/local/mysql844
[root@elk93 ~]#
[root@elk93 ~]# /etc/init.d/mysql.server start
Starting mysql.server (via systemctl): mysql.server.service.
[root@elk93 ~]#
[root@elk93 ~]# ss -ntl | grep 3306
LISTEN 0 151 *:3306 *:*
LISTEN 0 70 *:33060 *:*
[root@elk93 ~]# - 2.Filebeat采集mysql日志
2.1 启动Filebeat实例
[root@elk93 ~]# cat /tmp/modules-mysql-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/mysql.yml
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-mysql844-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/modules-mysql-to-es.yaml
2.2 启动模块文件
[root@elk93 ~]# filebeat modules enable mysql
Enabled mysql
[root@elk93 ~]#
[root@elk93 ~]# vim /etc/filebeat/modules.d/mysql.yml
- module: mysql
error:
enabled: true
var.paths: ["/var/lib/mysql/elk93.err"]
slowlog:
enabled: true
[root@elk93 ~]#
2.3 kibana查看数据验证(lvm扩展)
[root@elk91 ~]# df -h | grep mapper
/dev/mapper/ubuntu--vg-ubuntu--lv 48G 12G 35G 25% /
[root@elk91 ~]#
[root@elk91 ~]# lvextend /dev/mapper/ubuntu--vg-ubuntu--lv -l +100%FREE
Size of logical volume ubuntu-vg/ubuntu-lv changed from <49.00 GiB (12543 extents) to <98.00 GiB (25087 extents).
Logical volume ubuntu-vg/ubuntu-lv successfully resized.
[root@elk91 ~]#
[root@elk91 ~]# resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 13
The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 25689088 (4k) blocks long.
[root@elk91 ~]#
[root@elk91 ~]# df -h | grep mapper
/dev/mapper/ubuntu--vg-ubuntu--lv 97G 12G 81G 13% /
[root@elk91 ~]# Filebeat模块采集Redis集群日志
- 1.安装redis
[root@elk93 ~]# apt -y install redis- 2.启动Filebeat实例
[root@elk93 ~]# cat /tmp/modules-redis-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/redis.yml
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-redis-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "custom-index-example"
setup.template.pattern: "custom-index-example-*"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk93 ~]#
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/modules-redis-to-es.yaml- 3.启动redis
[root@elk93 ~]# filebeat modules enable redis
Enabled redis
[root@elk93 ~]# filebeat采集redis启动日志多行合并案例
[root@elk93 filebeat]# cat 11-filestream-multiline-redis-to-es.yaml
filebeat.inputs:
- type: filestream
paths:
- /var/log/redis/redis-server.log*
# 配置解析器
parsers:
# 定义多行匹配
- multiline:
# 指定匹配的类型
type: pattern
# 定义匹配模式
pattern: '^\d'
# 参考官网: #https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
negate: true
match: after
#output.console:
# pretty: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-redis-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: "linux95"
setup.template.pattern: "linux95-*"
[root@elk93 filebeat]# filebeat的多行匹配采集tomcat错误日志
- 1.准备tomcat错误日志
[root@elk93 ~]# shutdown.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.5
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.5
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.5/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.5/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.5/bin/tomcat-juli.jar
Using CATALINA_OPTS:
[root@elk93 ~]#
[root@elk93 ~]#
[root@elk93 ~]# vim /usr/local/apache-tomcat-11.0.5/conf/server.xml # 随意改错配置就可以
[root@elk93 ~]#
[root@elk93 ~]# catalina.sh start
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.5
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.5
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.5/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.5/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.5/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[root@elk93 ~]#
- 2.将日志采集后写入ES集群
[root@elk93 ~]# cat /tmp/filestream-tomcat-error-to-es.yaml
filebeat.inputs:
- type: filestream
paths:
- /usr/local/apache-tomcat-11.0.5/logs/catalina*
parsers:
- multiline:
type: pattern
pattern: '^\d'
negate: true
match: after
#output.console:
# pretty: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: custom-index-example-modules-tomcat-err-%{+yyyy.MM.dd}
setup.ilm.enabled: false
setup.template.name: " custom-index-example"
setup.template.pattern: " custom-index-example-*"
[root@elk93 ~]#
[root@elk93 ~]#
[root@elk93 ~]# rm -rf /var/lib/filebeat/
[root@elk93 ~]#
[root@elk93 ~]# filebeat -e -c /tmp/filestream-tomcat-error-to-es.yaml
Categories: