springboot集成plumelog日志系统

  |   1 评论   |   530 浏览

近几日闲来无事,工作摸鱼之时在码云上发现一个更加轻量级的分布式日志系统PlumeLog,就研究了一下,写了一个demo,做个记录

一、PlumeLog简介

  • 无入侵的分布式日志系统,基于log4j、log4j2、logback搜集日志,设置链路ID,方便查询关联日志
  • 基于elasticsearch作为查询引擎
  • 高吞吐,查询效率高
  • 全程不占应用程序本地磁盘空间,免维护;对于项目透明,不影响项目本身运行
  • 无需修改老项目,引入直接使用,支持dubbo,支持springcloud

二、准备工作

服务端安装

服务启动

  • 启动redis,确保本地可以连接redis(服务器安全组开放端口,redis配置可访问ip)
  • 启动elasticsearch,默认启动端口是9200,直接访问image.png
    显示这个就代表启动成功了

三、修改配置

Plumelog的压缩包解压后有这些文件image.png

修改application.properties,这边贴上我的配置,主要要改的就是redis和es的配置

 1spring.application.name=plumelog_server
 2server.port=8891
 3spring.thymeleaf.mode=LEGACYHTML5
 4spring.mvc.view.prefix=classpath:/templates/
 5spring.mvc.view.suffix=.html
 6spring.mvc.static-path-pattern=/plumelog/**
 7
 8#值为4种 redis,kafka,rest,restServer
 9#redis 表示用redis当队列
10#kafka 表示用kafka当队列
11#rest 表示从rest接口取日志
12#restServer 表示作为rest接口服务器启动
13#ui 表示单独作为ui启动
14plumelog.model=redis
15
16#如果使用kafka,启用下面配置
17#plumelog.kafka.kafkaHosts=172.16.247.143:9092,172.16.247.60:9092,172.16.247.64:9092
18#plumelog.kafka.kafkaGroupName=logConsumer
19
20#redis配置,3.0版本必须配置redis地址,因为需要监控报警
21plumelog.redis.redisHost=127.0.0.1:6379
22#如果使用redis有密码,启用下面配置
23plumelog.redis.redisPassWord=123456
24plumelog.redis.redisDb=0
25
26#如果使用rest,启用下面配置
27#plumelog.rest.restUrl=http://127.0.0.1:8891/getlog
28#plumelog.rest.restUserName=plumelog
29#plumelog.rest.restPassWord=123456
30
31#elasticsearch相关配置
32plumelog.es.esHosts=127.0.0.1:9200
33#ES7.*已经去除了索引type字段,所以如果是es7不用配置这个,7.*以下不配置这个会报错
34#plumelog.es.indexType=plumelog
35#索引分片数量设定,建议值:单日日志大小/ES节点机器jvm内存大小 合理的分片大小保证ES写入效率和查询效率
36plumelog.es.shards=5
37plumelog.es.replicas=0
38plumelog.es.refresh.interval=30s
39#日志索引建立方式day表示按天、hour表示按照小时
40plumelog.es.indexType.model=day
41#ES设置密码,启用下面配置
42#plumelog.es.userName=elastic
43#plumelog.es.passWord=FLMOaqUGamMNkZ2mkJiY
44
45#单次拉取日志条数
46plumelog.maxSendSize=5000
47#拉取时间间隔,kafka不生效
48plumelog.interval=1000
49
50#plumelog-ui的地址 如果不配置,报警信息里不可以点连接
51plumelog.ui.url=http://127.0.0.1:8891
52
53#管理密码,手动删除日志的时候需要输入的密码
54admin.password=123456
55#日志保留天数,配置0或者不配置默认永久保留
56admin.log.keepDays=30
57
58#登录配置
59#login.username=admin
60#login.password=admin

先不启动plumelog-server,等最后启动

提升性能推荐参考配置方法

单日日志体量在50G以内,并使用的SSD硬盘

plumelog.es.shards=5

plumelog.es.replicas=0

plumelog.es.refresh.interval=30s

plumelog.es.indexType.model=day

单日日志体量在50G以上,并使用的机械硬盘

plumelog.es.shards=5

plumelog.es.replicas=0

plumelog.es.refresh.interval=30s

plumelog.es.indexType.model=hour

单日日志体量在100G以上,并使用的机械硬盘

plumelog.es.shards=10

plumelog.es.replicas=0

plumelog.es.refresh.interval=30s

plumelog.es.indexType.model=hour

单日日志体量在1000G以上,并使用的SSD硬盘,这个配置可以跑到10T一天以上都没问题

plumelog.es.shards=10

plumelog.es.replicas=1

plumelog.es.refresh.interval=30s

plumelog.es.indexType.model=hour

plumelog.es.shards的增加和hour模式下需要调整ES集群的最大分片数

1PUT /_cluster/settings
2{
3  "persistent": {
4    "cluster": {
5      "max_shards_per_node":100000
6    }
7  }
8}

四、创建springboot项目

项目配置

pom.xml

 1<?xml version="1.0" encoding="UTF-8"?>
 2<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 3         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
 4    <modelVersion>4.0.0</modelVersion>
 5    <parent>
 6        <groupId>org.springframework.boot</groupId>
 7        <artifactId>spring-boot-starter-parent</artifactId>
 8        <version>2.4.3</version>
 9        <relativePath/> <!-- lookup parent from repository -->
10    </parent>
11    <groupId>com.example</groupId>
12    <artifactId>demo-mybatis</artifactId>
13    <version>0.0.1-SNAPSHOT</version>
14    <name>demo-mybatis</name>
15    <description>Demo project for Spring Boot</description>
16    <properties>
17        <java.version>1.8</java.version>
18    </properties>
19    <dependencies>
20        <dependency>
21            <groupId>org.springframework.boot</groupId>
22            <artifactId>spring-boot-starter-web</artifactId>
23        </dependency>
24        <dependency>
25            <groupId>org.mybatis.spring.boot</groupId>
26            <artifactId>mybatis-spring-boot-starter</artifactId>
27            <version>2.1.4</version>
28        </dependency>
29
30        <dependency>
31            <groupId>mysql</groupId>
32            <artifactId>mysql-connector-java</artifactId>
33            <scope>runtime</scope>
34        </dependency>
35        <dependency>
36            <groupId>org.projectlombok</groupId>
37            <artifactId>lombok</artifactId>
38            <optional>true</optional>
39        </dependency>
40        <dependency>
41            <groupId>org.springframework.boot</groupId>
42            <artifactId>spring-boot-starter-test</artifactId>
43            <scope>test</scope>
44        </dependency>
45        <!--分布式日志收集plumelog-->
46        <dependency>
47            <groupId>com.plumelog</groupId>
48            <artifactId>plumelog-logback</artifactId>
49            <version>3.3</version>
50        </dependency>
51        <dependency>
52            <groupId>com.plumelog</groupId>
53            <artifactId>plumelog-trace</artifactId>
54            <version>3.3</version>
55        </dependency>
56        <dependency>
57            <groupId>org.springframework.boot</groupId>
58            <artifactId>spring-boot-starter-aop</artifactId>
59            <version>2.1.11.RELEASE</version>
60            <scope>provided</scope>
61        </dependency>
62        <dependency>
63            <groupId>cn.hutool</groupId>
64            <artifactId>hutool-core</artifactId>
65            <version>5.5.8</version>
66        </dependency>
67    </dependencies>
68
69    <build>
70        <plugins>
71            <plugin>
72                <groupId>org.springframework.boot</groupId>
73                <artifactId>spring-boot-maven-plugin</artifactId>
74                <configuration>
75                    <excludes>
76                        <exclude>
77                            <groupId>org.projectlombok</groupId>
78                            <artifactId>lombok</artifactId>
79                        </exclude>
80                    </excludes>
81                </configuration>
82            </plugin>
83        </plugins>
84    </build>
85
86</project>

resources下两个配置文件

application.properties

 1server.port=8888
 2spring.datasource.url=jdbc:mysql://127.0.0.1:3306/test?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false
 3spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
 4spring.datasource.username=root
 5spring.datasource.password=123456
 6spring.datasource.hikari.maximum-pool-size=10
 7# mybatis配置
 8mybatis.type-aliases-package=com.example.demomybatis.model
 9mybatis.configuration.map-underscore-to-camel-case=true
10mybatis.configuration.default-fetch-size=100
11mybatis.configuration.default-statement-timeout=30
12# jackson格式化日期时间
13spring.jackson.date-format=YYYY-MM-dd HH:mm:ss
14spring.jackson.time-zone=GMT+8
15spring.jackson.serialization.write-dates-as-timestamps=false
16# sql日志打印
17logging.level.com.example.demomybatis.dao=debug
18## 输出的日志文件
19logging.file.path=/log

logback-spring.xml

 1<?xml version="1.0" encoding="UTF-8" ?>
 2<configuration>
 3    <appender name="consoleApp" class="ch.qos.logback.core.ConsoleAppender">
 4        <layout class="ch.qos.logback.classic.PatternLayout">
 5            <pattern>
 6                %date{yyyy-MM-dd HH:mm:ss.SSS} %-5level[%thread]%logger{56}.%method:%L -%msg%n
 7            </pattern>
 8        </layout>
 9    </appender>
10
11    <appender name="fileInfoApp" class="ch.qos.logback.core.rolling.RollingFileAppender">
12        <filter class="ch.qos.logback.classic.filter.LevelFilter">
13            <level>ERROR</level>
14            <onMatch>DENY</onMatch>
15            <onMismatch>ACCEPT</onMismatch>
16        </filter>
17        <encoder>
18            <pattern>
19                %date{yyyy-MM-dd HH:mm:ss.SSS} %-5level[%thread]%logger{56}.%method:%L -%msg%n
20            </pattern>
21        </encoder>
22        <!-- 滚动策略 -->
23        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
24            <!-- 路径 -->
25            <fileNamePattern>log/info.%d.log</fileNamePattern>
26        </rollingPolicy>
27    </appender>
28
29    <appender name="fileErrorApp" class="ch.qos.logback.core.rolling.RollingFileAppender">
30        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
31            <level>ERROR</level>
32        </filter>
33        <encoder>
34            <pattern>
35                %date{yyyy-MM-dd HH:mm:ss.SSS} %-5level[%thread]%logger{56}.%method:%L -%msg%n
36            </pattern>
37        </encoder>
38
39        <!-- 设置滚动策略 -->
40        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
41            <!-- 路径 -->
42            <fileNamePattern>log/error.%d.log</fileNamePattern>
43
44            <!-- 控制保留的归档文件的最大数量,超出数量就删除旧文件,假设设置每个月滚动,
45            且<maxHistory> 是1,则只保存最近1个月的文件,删除之前的旧文件 -->
46            <MaxHistory>1</MaxHistory>
47        </rollingPolicy>
48    </appender>
49
50    <appender name="plumelog" class="com.plumelog.logback.appender.RedisAppender">
51        <appName>mybatisDemo</appName>
52        <redisHost>127.0.0.1</redisHost>
53        <redisAuth>123456</redisAuth>
54        <redisPort>6379</redisPort>
55        <runModel>2</runModel>
56    </appender>
57
58    <!-- root 一定要放在最后,因有加载顺序的问题 -->
59    <root level="INFO">
60        <appender-ref ref="consoleApp"/>
61        <appender-ref ref="fileInfoApp"/>
62        <appender-ref ref="fileErrorApp"/>
63        <appender-ref ref="plumelog"/>
64    </root>
65</configuration>

logback配置文件中注意要加上plumelog的appender,然后在root中引用appender,这个是推送日志到redis队列中的配置

traceid设置及链路追踪全局打点配置

traceid拦截器配置
 1package com.example.demomybatis.common.interceptor;
 2
 3import com.plumelog.core.TraceId;
 4import org.springframework.stereotype.Component;
 5import org.springframework.web.servlet.HandlerInterceptor;
 6import org.springframework.web.servlet.ModelAndView;
 7
 8import javax.servlet.http.HttpServletRequest;
 9import javax.servlet.http.HttpServletResponse;
10import java.util.UUID;
11
12/**
13 * Created by wenbo on 2021/2/25.
14 */
15@Component
16public class Interceptor implements HandlerInterceptor {
17    @Override
18    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
19        //设置TraceID值,不埋此点链路ID就没有
20        TraceId.logTraceID.set(UUID.randomUUID().toString().replaceAll("-", ""));
21        return true;
22    }
23
24    @Override
25    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) {
26    }
27
28    @Override
29    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
30    }
31}
 1package com.example.demomybatis.common.interceptor;
 2
 3import org.springframework.context.annotation.Configuration;
 4import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
 5import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
 6
 7/**
 8 * Created by wenbo on 2021/2/25.
 9 */
10@Configuration
11public class InterceptorConfig implements WebMvcConfigurer {
12    @Override
13    public void addInterceptors(InterceptorRegistry registry) {
14        // 自定义拦截器,添加拦截路径和排除拦截路径
15        registry.addInterceptor(new Interceptor()).addPathPatterns("/**");
16    }
17}
链路追踪全局打点配置
 1package com.example.demomybatis.common.config;
 2
 3import com.plumelog.trace.aspect.AbstractAspect;
 4import org.aspectj.lang.JoinPoint;
 5import org.aspectj.lang.annotation.Around;
 6import org.aspectj.lang.annotation.Aspect;
 7import org.springframework.stereotype.Component;
 8
 9/**
10 * Created by wenbo on 2021/2/25.
11 */
12@Aspect
13@Component
14public class AspectConfig extends AbstractAspect {
15    @Around("within(com.example..*))")
16    public Object around(JoinPoint joinPoint) throws Throwable {
17        return aroundExecute(joinPoint);
18    }
19}

这里要把切入点路径换成自己的包路径
最后还需要把在启动类添加扫描注解,com.example.demomybatis换成自己的包路径

@ComponentScan({"com.plumelog","com.example.demomybatis"})

最后放上demo链接:https://gitee.com/wen_bo/demo-plumelog

五、启动项目

先将springboot启动起来,然后java -jar plumelog-server-3.2.jar启动plumelog
访问http://127.0.0.1
默认用户名和密码都是admin
最后展示一下效果图
image.png

image.png

image.png

image.png

image.png

image.png


作者:wenbo

评论

发表评论


取消