01
—
前言
相信作为一个资深的搬砖人，在处理问题的时候免不了查看应用系统日志，且可以根据这个日志日志精准、快速的解决实际的问题。一般情况下我们的系统日志都放置在包的运行目录下面，非常不便于查看和分类。那么。今天我们就引入ELK的日志处理架构来解决它。
02
—
技术积累
ELK组成及功能
ELK是logstash、elasticsearch、kibana的简称，和其名字一样，elk架构就是将这三个中间件进行整合搭建一个日志系统。
首先我们应用系统集成logstash客户端并采集日志上传到logstash服务端进行过滤、转换，转换后的日志写入elasticsearch，es的强大功能提供数据存储，分词和倒排索引提升查询效率；最后的kibana直接是渲染日志数据的分析和可视化平台。
框架搭建基础
为方便我们架构的搭建，我们用docker-compose进行容器化编排，只要保存elk三个组件同网络下它们就能够根据服务名进行通讯。
当然，对于向外暴露的接口我们仅仅需要暴露logstash的进行数据上传，es的进行数据外部查询即可。每个应用服务都必须有自己的logstash配置，在配置中提供输入、输出路径和过滤参数，对于的端口我们也需要向外暴露以便于数据的上传。
03
—
ELK环境搭建
elk目录下文件树：
./ ├── docker-compose.yml ├── elasticsearch │ ├── config │ │ └── elasticsearch.yml │ ├── data │ └── logs ├── kabana │ └── config │ └── kabana.yml └── logstash ├── config │ ├── logstash.yml │ └── small-tools │ └── demo.config └── data
elasticsearch配置相关
mkdie elk #增加es目录 cd elk mkdir -p ./elasticsearch/logs ./elasticsearch/data ./elasticsearch/config chmod 777 ./elasticsearch/data #./elasticsearch/config 下增加es配置文件 cd elasticsearch/config vim elasticsearch.yml
cluster.name: "docker-cluster" network.host: 0.0.0.0 http.port: 9200 # 开启es跨域 http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-headers: Authorization # 开启安全控制 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true
kibana配置相关
cd elk
mkdir -p ./kibana/config
#./kibana/config 下增加kibana配置文件
cd kibana/config
vim kibana.yml
server.name: kibana server.host: "0.0.0.0" server.publicBaseUrl: "http://kibana:5601" elasticsearch.hosts: [ "http://elasticsearch:9200" ] xpack.monitoring.ui.container.elasticsearch.enabled: true elasticsearch.username: "elastic" elasticsearch.password: "123456" i18n.locale: zh-CN
logstash配置相关
cd elk
mkdir -p ./logstash/data ./logstash/config ./logstash/config/small-tools
chmod 777 ./logstash/data
#./logstash/config 下增加logstash配置文件
cd logstash/config
vim logstash.yml
http.host: "0.0.0.0" xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ] xpack.monitoring.elasticsearch.username: "elastic" xpack.monitoring.elasticsearch.password: "123456"
#./logstash/config/small-tools 下增加demo项目监控配置文件
cd small-tools
vim demo.config
input { #输入 tcp { mode => "server" host => "0.0.0.0" # 允许任意主机发送日志 type => "demo" # 设定type以区分每个输入源 port => 9999 codec => json_lines # 数据格式 } } filter { mutate { # 导入之过滤字段 remove_field => ["LOG_MAX_HISTORY_DAY", "LOG_HOME", "APP_NAME"] remove_field => ["@version", "_score", "port", "level_value", "tags", "_type", "host"] } } output { #输出-控制台 stdout{ codec => rubydebug } } output { #输出-es if [type] == "demo" { elasticsearch { action => "index" # 输出时创建映射 hosts => "http://elasticsearch:9200" # ES地址和端口 user => "elastic" # ES用户名 password => "123456" # ES密码 index => "demo-%{+YYYY.MM.dd}" # 指定索引名-按天 codec => "json" } } }
elk目录下增加docker-compose文件
docker-compose.yml
version: '3.3' networks: elk: driver: bridge services: elasticsearch: image: registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1 container_name: elk_elasticsearch restart: unless-stopped volumes: - "./elasticsearch/data:/usr/share/elasticsearch/data" - "./elasticsearch/logs:/usr/share/elasticsearch/logs" - "./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml" environment: TZ: Asia/Shanghai LANG: en_US.UTF-8 TAKE_FILE_OWNERSHIP: "true" # 权限 discovery.type: single-node ES_JAVA_OPTS: "-Xmx512m -Xms512m" ELASTIC_PASSWORD: "123456" # elastic账号密码 ports: - "9200:9200" - "9300:9300" networks: - elk kibana: image: registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1 container_name: elk_kibana restart: unless-stopped volumes: - "./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml" ports: - "5601:5601" depends_on: - elasticsearch links: - elasticsearch networks: - elk logstash: image: registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1 container_name: elk_logstash restart: unless-stopped environment: LS_JAVA_OPTS: "-Xmx512m -Xms512m" volumes: - "./logstash/data:/usr/share/logstash/data" - "./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml" - "./logstash/config/small-tools:/usr/share/logstash/config/small-tools" command: logstash -f /usr/share/logstash/config/small-tools ports: - "9600:9600" - "9999:9999" depends_on: - elasticsearch networks: - elk
查看elk目录文件树
yum -y install tree #查看当前目录下4层 tree -L 4 #显示所有文件、文件夹 tree -a #显示大小 tree -s [root@devops-01 elk]# pwd /home/test/demo/elk [root@devops-01 elk]# tree ./ ./ ├── docker-compose.yml ├── elasticsearch │ ├── config │ │ └── elasticsearch.yml │ ├── data │ └── logs ├── kabana │ └── config │ └── kabana.yml └── logstash ├── config │ ├── logstash.yml │ └── small-tools │ └── demo.config └── data 10 directories, 5 files
编排elk
docker-compose up -d
编排成功查看容器是否成功启动
[root@devops-01 elk]# docker ps | grep elk edcf6c1cecb3 registry.cn-hangzhou.aliyuncs.com/zhengqing/kibana:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 10 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp elk_kibana 7c24b65d2a27 registry.cn-hangzhou.aliyuncs.com/zhengqing/logstash:7.14.1 “/usr/local/bin/dock…” 6 minutes ago Up 13 seconds 5044/tcp, 9600/tcp elk_logstash b4be2f1c0a28 registry.cn-hangzhou.aliyuncs.com/zhengqing/elasticsearch:7.14.1 “/bin/tini – /usr/l…” 6 minutes ago Up 6 minutes 0.0.0.0:9800->9200/tcp, :::9800->9200/tcp, 0.0.0.0:9900->9300/tcp, :::9900->9300/tcp elk_elasticsearch
编排成功访问kibana页面
http://10.10.22.174:5601/app/home#/
04
—
springboot集成logstash
pom.xml
<!--logstash start--> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>6.6</version> </dependency> <!--logstash end-->
logback-spring.xml
<springProfile name="uat"> <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>10.10.22.174:9999</destination> <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/> </appender> <root level="INFO"> <appender-ref ref="logstash"/> </root> </springProfile>
启动项目logstash采集日志
kibana配置查看日志
http://10.10.22.174:5601/app/home#/ 输入ES用户名和密码进入kibana控制台
点击管理按钮进入管理界面
点击索引模式进入–>创建索引模式
输入配置日志表达式–>点击下一步
选择timestamp -->创建索引模式
创建完成如下所示代表成功
查看日志
菜单点击–>discover
05
—
写在最后
ELK环境部署并采集springboot项目日志还是比较简单，我们只需要用docker容器化技术搭建起elk框架，然后在自己的项目中进行数据采集上传即可。当然对于elk组成元素的logstash、elasticsearch、kibana还是需要一些基础的了解，方便在实战的时候进行操作。
