如何基于Oracle Linux 9.x部署Logstash?

Elastic Stack

1 基础知识

1.1 软件简介

– Logstash是免费且开放的服务器端实时数据处理管道
– Logstash能够从多个源采集数据,转换数据,然后将数据发送到存储库中
– Logstash是Elastic Stack的中央数据流引擎,用于收集、丰富和统一所有数据(兼容格式或模式)
– Logstash与Elasticsearch、Kibana和Beats共同使用时便具有强大的实时数据处理能力

1.2 工作原理

– Logstash事件处理管道分三个阶段,输入、过滤器、输出
– Logstash的输入负责生成事件(支持编码解码器)
– Logstash的过滤器负责修改事件
– Logstash的输出将事件发送到其他地方(支持编码解码器)

1.3 常用数据类型与插件

1.3.1 输入数据类型与插件

– file,从文件系统上读取文件,类似UNIX“tail -0F”命令
– syslog,从514端口监听syslog消息并根据RFC3164格式进行解析
– redis,使用redis通道和redis列表从redis服务器读取,Redis通常作为集中式Logstash的“代理”,负责远程Logstash发货人的事件排队
– beats,处理Beats发送的事件
详细的插件,请参阅,
https://www.elastic.co/guide/en/logstash/current/input-plugins.html

1.3.2 过滤器数据类型与插件

– grok,解析和构造任意文本,是Logstash中将非结构化日志数据解析为结构化和可查询数据的最佳方法(内置120中模式)
– mutate,对事件字段执行一般转换,包括重命名、删除、替换和修改事件中的字段
– drop,完全删除事件,例如调试事件
– clone,复制事件,可能添加或删除字段
– geoip,添加有关IP地址地理位置的信息
详细的插件,请参阅,
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

1.3.3 输出数据类型与插件

– elasticseaarch,将事件数据发送到Elasticsearch
– file,将事件写入磁盘
– Graphite,将事件发送到Graphite(一种用于存储和绘制指标的流行开源工具)
– statsd,将事件数据发送到statsd(支持倾听UDP发送的统计信息并将聚合发送到一个或多个可插入后端服务的服务)
详细的插件,请参阅,
https://www.elastic.co/guide/en/logstash/current/output-plugins.html

1.3.4 编码器的数据类型与插件

– json,以JSON格式对数据进行编码和解压
– multiline,将多行文本事件(例如Java异常和堆栈跟踪消息)合并到单个事件
详细的插件,请参阅,
https://www.elastic.co/guide/en/logstash/current/codec-plugins.html

1.4 logstash配置文件

– logstash配置文件分两种,管道配置文件和设置文件
– logstash管道配置文件定义Logstash处理的管道
– logstash设置文件定义Logstash的启动和执行选项
– logstash管道配置文件需要符合“/etc/logstash/conf.d/*.conf”定义

2 最佳实践

2.1 环境信息

2.1.1 主机信息

Host Name = elasticsearch[01-03].cmdschool.org
IP Address = 10.168.0.[100 – 102] OS = Oracle Linux 9.x x86_64
Elasticsearch Version = 8.11.3
另外,如果你需要部署Elasticsearch集群环境,请参阅以下文档,

如何部署Elasticsearch 8.x集群?

Host Name = logstash.cmdschool.org
OS = Oracle Linux 9.x x86_64
IP Address = 10.168.0.103

2.2.2 配置名称解析

In All,

vim /etc/hosts

加入如下配置,

10.168.0.100 elasticsearch01 elasticsearch01.cmdschool.org
10.168.0.101 elasticsearch02 elasticsearch02.cmdschool.org
10.168.0.102 elasticsearch03 elasticsearch03.cmdschool.org
10.168.0.103 logstash logstash.cmdschool.org

注:以上配置仅用于测试,正式环境请使用DNS代替

2.2 安装前的准备

In Logstash,

2.2.1 基本环境配置

如何完成CentOS 7.x的基本服务?

2.2.2 配置安装源

vim /etc/yum.repos.d/elasticsearc.repo

加入如下配置,

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

2.3 安装logstash

2.3.1 安装指定的版本

dnf install -y logstash-8.11.3

2.3.2 启动服务并设置自启动

systemctl start logstash.service
systemctl enable logstash.service

另外,建议你使用如下命令检查服务状态,

systemctl status logstash.service

可见如下提示,

● logstash.service - logstash
     Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-01-11 14:54:56 CST; 9s ago
   Main PID: 5201 (java)
      Tasks: 16 (limit: 9132)
     Memory: 577.6M
        CPU: 27.463s
     CGroup: /system.slice/logstash.service
             └─5201 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+>

Jan 11 14:54:56 logstash.cmdschool.org systemd[1]: Started logstash.
Jan 11 14:54:56 logstash.cmdschool.org logstash[5201]: Using bundled JDK: /usr/share/logstash/jdk
Jan 11 14:55:03 logstash.cmdschool.org logstash[5201]: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/conc>
Jan 11 14:55:03 logstash.cmdschool.org logstash[5201]: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/conc>

如果需要查询运行的进程号,可使用如下命令,

pgrep -u logstash java -a

可见如下显示,

6043 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /usr/share/logstash/vendor/jruby/lib/jruby.jar:/usr/share/logstash/logstash-core/lib/jars/checker-qual-3.33.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.15.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.18.0.jar:/usr/share/logstash/logstash-core/lib/jars/failureaccess-1.0.1.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.15.0.jar:/usr/share/logstash/logstash-core/lib/jars/guava-32.1.2-jre.jar:/usr/share/logstash/logstash-core/lib/jars/httpclient-4.5.13.jar:/usr/share/logstash/logstash-core/lib/jars/httpcore-4.4.14.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.3.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-3.0.2.jar:/usr/share/logstash/logstash-core/lib/jars/jvm-options-parser-8.11.3.jar:/usr/share/logstash/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.10.2.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-2.2.jar org.logstash.Logstash --path.settings /etc/logstash

由以上可知logstash是由Java开发的,如果需要查询运行的进程号,可使用如下命令,

/usr/share/logstash/jdk/bin/jps

可见如下显示,

16117 Jps
16062 Logstash

2.3.3 修改服务配置文件

cp /etc/logstash/logstash.yml /etc/logstash/logstash.yml.default
vim /etc/logstash/logstash.yml

配置修改如下,

node.name: logstash.cmdschool.org
path.data: /data/logstash
pipeline.workers: 4
config.reload.automatic: true
api.http.host: 127.0.0.1
api.http.port: 9600-9700
path.logs: /var/log/logstash

根据配置文件创建服务所需的目录,

mkdir -p /data/logstash
chown logstash:logstash /data/logstash
chmod 750 /data/logstash
chmod g+s /data/logstash

另外,建议修改JVM内存值,

vim /etc/logstash/jvm.options

修改如下配置,

-Xms6g
-Xmx6g

重启服务使配置生效,

systemctl restart logstash.service

2.3.4 创建管道配置文件

cat /etc/logstash/logstash-sample.conf > /etc/logstash/conf.d/elasticsearch0x.cmdschool.org.conf
vim /etc/logstash/conf.d/elasticsearch0x.cmdschool.org.conf

如果管道配置文件需要定义正确否则服务无法正常运行,配置修改如下,

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch01:9200", "http://elasticsearch02:9200", "http://elasticsearch03:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

配置创建后,你可以使用如下命令测试配置,

/usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash -f /etc/logstash/conf.d/elasticsearch0x.cmdschool.org.conf

重启服务使配置生效,

systemctl restart logstash.service

此时,服务可以正常运行,可使用如下命令查看管道的倾听端口,

for i in `pgrep -u logstash java`; do netstat -anp | grep $i; done

可见如下显示,

tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      17351/java     
tcp6       0      0 :::5044                 :::*                    LISTEN      17351/java       
tcp6       0      0 10.168.0.103:36008      10.168.0.101:9200       ESTABLISHED 16910/java          
tcp6       0      0 10.168.0.103:41270      10.168.0.100:9200       ESTABLISHED 16910/java          
tcp6       0      0 10.168.0.103:35996      10.168.0.102:9200       ESTABLISHED 16910/java          
unix  2      [ ]         STREAM     CONNECTED     61815    16910/java           
unix  3      [ ]         STREAM     CONNECTED     61790    16910/java  

2.3.5 开放管道服务端口

firewall-cmd --permanent --add-port 5044/tcp
firewall-cmd --reload
firewall-cmd --list-all

参阅文档
=====================

软件的简介
————-
https://www.elastic.co/guide/en/logstash/current/introduction.html
https://www.elastic.co/cn/logstash
https://www.elastic.co/cn/webinars/getting-started-logstash

软件的下载
————-
https://www.elastic.co/cn/downloads/logstash

软件的安装
————-
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

logstash配置文件
—————
https://www.elastic.co/guide/en/logstash/current/config-setting-files.html

创建管道配置文件
—————–
https://www.elastic.co/guide/en/logstash/current/configuration.html

没有评论

发表回复

Elastic Stack
如何部署带安全认证的WinLogBeat?

1 前言 一个问题,一篇文章,一出故事。 由于笔者需要收集Windows的日志,于是本章将整理Win …

Elastic Stack
如何部署带安全认证的Filebeat与logstash集成?

1 前言 一个问题,一篇文章,一出故事。 本章将整理Filebeat与Logstash集成,然后通过 …

Elastic Stack
如何部署带安全认证的Logstash?

1 基础知识 1.1 软件简介 – Logstash是免费且开放的服务器端实时数据处理管 …