如何基于Oracle Linux 9.x部署Logstash？
- By : Will
- Category : Elastic Stack
1 基础知识
1.1 软件简介
– Logstash是免费且开放的服务器端实时数据处理管道
– Logstash能够从多个源采集数据，转换数据，然后将数据发送到存储库中
– Logstash是Elastic Stack的中央数据流引擎，用于收集、丰富和统一所有数据（兼容格式或模式）
– Logstash与Elasticsearch、Kibana和Beats共同使用时便具有强大的实时数据处理能力
1.2 工作原理
– Logstash事件处理管道分三个阶段，输入、过滤器、输出
– Logstash的输入负责生成事件（支持编码解码器）
– Logstash的过滤器负责修改事件
– Logstash的输出将事件发送到其他地方（支持编码解码器）
1.3 常用数据类型与插件
1.3.1 输入数据类型与插件
– file，从文件系统上读取文件，类似UNIX“tail -0F”命令
– syslog，从514端口监听syslog消息并根据RFC3164格式进行解析
– redis，使用redis通道和redis列表从redis服务器读取，Redis通常作为集中式Logstash的“代理”，负责远程Logstash发货人的事件排队
– beats，处理Beats发送的事件
详细的插件，请参阅，
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
1.3.2 过滤器数据类型与插件
– grok，解析和构造任意文本，是Logstash中将非结构化日志数据解析为结构化和可查询数据的最佳方法（内置120中模式）
– mutate，对事件字段执行一般转换，包括重命名、删除、替换和修改事件中的字段
– drop，完全删除事件，例如调试事件
– clone，复制事件，可能添加或删除字段
– geoip，添加有关IP地址地理位置的信息
详细的插件，请参阅，
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
1.3.3 输出数据类型与插件
– elasticseaarch，将事件数据发送到Elasticsearch
– file，将事件写入磁盘
– Graphite，将事件发送到Graphite（一种用于存储和绘制指标的流行开源工具）
– statsd，将事件数据发送到statsd（支持倾听UDP发送的统计信息并将聚合发送到一个或多个可插入后端服务的服务）
详细的插件，请参阅，
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
1.3.4 编码器的数据类型与插件
– json，以JSON格式对数据进行编码和解压
– multiline，将多行文本事件（例如Java异常和堆栈跟踪消息）合并到单个事件
详细的插件，请参阅，
https://www.elastic.co/guide/en/logstash/current/codec-plugins.html
1.4 logstash配置文件
– logstash配置文件分两种，管道配置文件和设置文件
– logstash管道配置文件定义Logstash处理的管道
– logstash设置文件定义Logstash的启动和执行选项
– logstash管道配置文件需要符合“/etc/logstash/conf.d/*.conf”定义
2 最佳实践
2.1 环境信息
2.1.1 主机信息
Host Name = elasticsearch[01-03].cmdschool.org
IP Address = 10.168.0.[100 – 102] OS = Oracle Linux 9.x x86_64
Elasticsearch Version = 8.11.3
另外，如果你需要部署Elasticsearch集群环境，请参阅以下文档，
Host Name = logstash.cmdschool.org
OS = Oracle Linux 9.x x86_64
IP Address = 10.168.0.103
2.2.2 配置名称解析
In All,
vim /etc/hosts
加入如下配置，
10.168.0.100 elasticsearch01 elasticsearch01.cmdschool.org 10.168.0.101 elasticsearch02 elasticsearch02.cmdschool.org 10.168.0.102 elasticsearch03 elasticsearch03.cmdschool.org 10.168.0.103 logstash logstash.cmdschool.org
注：以上配置仅用于测试，正式环境请使用DNS代替
2.2 安装前的准备
In Logstash,
2.2.1 基本环境配置
2.2.2 配置安装源
vim /etc/yum.repos.d/elasticsearc.repo
加入如下配置，
[elasticsearch] name=Elasticsearch repository for 8.x packages baseurl=https://artifacts.elastic.co/packages/8.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
2.3 安装logstash
2.3.1 安装指定的版本
dnf install -y logstash-8.11.3
2.3.2 启动服务并设置自启动
systemctl start logstash.service systemctl enable logstash.service
另外，建议你使用如下命令检查服务状态，
systemctl status logstash.service
可见如下提示，
● logstash.service - logstash
Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: disabled)
Active: active (running) since Thu 2024-01-11 14:54:56 CST; 9s ago
Main PID: 5201 (java)
Tasks: 16 (limit: 9132)
Memory: 577.6M
CPU: 27.463s
CGroup: /system.slice/logstash.service
└─5201 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+>
Jan 11 14:54:56 logstash.cmdschool.org systemd[1]: Started logstash.
Jan 11 14:54:56 logstash.cmdschool.org logstash[5201]: Using bundled JDK: /usr/share/logstash/jdk
Jan 11 14:55:03 logstash.cmdschool.org logstash[5201]: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/conc>
Jan 11 14:55:03 logstash.cmdschool.org logstash[5201]: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/conc>
如果需要查询运行的进程号，可使用如下命令，
pgrep -u logstash java -a
可见如下显示，
6043 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /usr/share/logstash/vendor/jruby/lib/jruby.jar:/usr/share/logstash/logstash-core/lib/jars/checker-qual-3.33.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.15.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.18.0.jar:/usr/share/logstash/logstash-core/lib/jars/failureaccess-1.0.1.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.15.0.jar:/usr/share/logstash/logstash-core/lib/jars/guava-32.1.2-jre.jar:/usr/share/logstash/logstash-core/lib/jars/httpclient-4.5.13.jar:/usr/share/logstash/logstash-core/lib/jars/httpcore-4.4.14.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-3.0.2.jar:/usr/share/logstash/logstash-core/lib/jars/jvm-options-parser-8.11.3.jar:/usr/share/logstash/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.10.2.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-2.2.jar org.logstash.Logstash --path.settings /etc/logstash
由以上可知logstash是由Java开发的，如果需要查询运行的进程号，可使用如下命令，
/usr/share/logstash/jdk/bin/jps
可见如下显示，
16117 Jps 16062 Logstash
2.3.3 修改服务配置文件
cp /etc/logstash/logstash.yml /etc/logstash/logstash.yml.default vim /etc/logstash/logstash.yml
配置修改如下，
node.name: logstash.cmdschool.org path.data: /data/logstash pipeline.workers: 4 config.reload.automatic: true api.http.host: 127.0.0.1 api.http.port: 9600-9700 path.logs: /var/log/logstash
根据配置文件创建服务所需的目录，
mkdir -p /data/logstash chown logstash:logstash /data/logstash chmod 750 /data/logstash chmod g+s /data/logstash
另外，建议修改JVM内存值，
vim /etc/logstash/jvm.options
修改如下配置，
-Xms6g -Xmx6g
重启服务使配置生效，
systemctl restart logstash.service
2.3.4 创建管道配置文件
cat /etc/logstash/logstash-sample.conf > /etc/logstash/conf.d/elasticsearch0x.cmdschool.org.conf vim /etc/logstash/conf.d/elasticsearch0x.cmdschool.org.conf
如果管道配置文件需要定义正确否则服务无法正常运行，配置修改如下，
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch01:9200", "http://elasticsearch02:9200", "http://elasticsearch03:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
配置创建后，你可以使用如下命令测试配置，
/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash -f /etc/logstash/conf.d/dn2elasticsearch0x.sa e.com.hk.conf
重启服务使配置生效，
systemctl restart logstash.service
此时，服务可以正常运行，可使用如下命令查看管道的倾听端口，
for i in `pgrep -u logstash java`; do netstat -anp | grep $i; done
可见如下显示，
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 17351/java tcp6 0 0 :::5044 :::* LISTEN 17351/java tcp6 0 0 10.168.0.103:36008 10.168.0.101:9200 ESTABLISHED 16910/java tcp6 0 0 10.168.0.103:41270 10.168.0.100:9200 ESTABLISHED 16910/java tcp6 0 0 10.168.0.103:35996 10.168.0.102:9200 ESTABLISHED 16910/java unix 2 [ ] STREAM CONNECTED 61815 16910/java unix 3 [ ] STREAM CONNECTED 61790 16910/java
2.3.5 开放管道服务端口
firewall-cmd --permanent --add-port 5044/tcp firewall-cmd --reload firewall-cmd --list-all
参阅文档
=====================
软件的简介
————-
https://www.elastic.co/guide/en/logstash/current/introduction.html
https://www.elastic.co/cn/logstash
https://www.elastic.co/cn/webinars/getting-started-logstash
软件的下载
————-
https://www.elastic.co/cn/downloads/logstash
软件的安装
————-
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
logstash配置文件
—————
https://www.elastic.co/guide/en/logstash/current/config-setting-files.html
创建管道配置文件
—————–
https://www.elastic.co/guide/en/logstash/current/configuration.html
没有评论