记一次jedis连接池顽固问题排查与修改

2023-04-21 16:14 由 dawndust 发表于 #后端开发

这辈子不想再看到jedisBrokenPipe！！

测试环境运行16天后报错信息：

05:42:32.629 [http-nio-8093-exec-2] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - [log,175] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Connection reset] with root cause java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.net.SocketInputStream.read(SocketInputStream.java:127) at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196) at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) at redis.clients.jedis.Protocol.process(Protocol.java:151) at redis.clients.jedis.Protocol.read(Protocol.java:215) at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:259) at redis.clients.jedis.Connection.getBulkReply(Connection.java:248) at redis.clients.jedis.Jedis.hget(Jedis.java:674) at com.chint.cdc.TenantAPI.getTenanatInfoWithNumberFromRedis(TenantAPI.java:224) at com.chint.cdc.TenantAPI.getInfoByNumber(TenantAPI.java:126) at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.getJedis(GeneralAuthorizationAspect.java:224) at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.around(GeneralAuthorizationAspect.java:131)

本地调试时，130并发的报错信息：

15:08:33.080 [http-nio-8093-exec-135] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - [log,175] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: socket write error] with root cause
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216)
at redis.clients.jedis.Connection.flush(Connection.java:331)
at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:257)
at redis.clients.jedis.Connection.getBulkReply(Connection.java:248)
at redis.clients.jedis.Jedis.hget(Jedis.java:674)
at com.chint.cdc.TenantAPI.getTenanatInfoWithNumberFromRedis(TenantAPI.java:224)
at com.chint.cdc.TenantAPI.getInfoByNumber(TenantAPI.java:126)
at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.getJedis(GeneralAuthorizationAspect.java:231)
at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.around(GeneralAuthorizationAspect.java:134)
at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

大佬一直说要记得释放连接，于是我改成以下写法：

但还是有问题，本地30并发没问题。130并发就跪了。怎么想怎么奇怪，明明都关闭连接了，为啥还会无法获取jedis连接，最后换了一种思路。不在当前切面类维护jedisPoolMap，而是用jedisConfig里的通用map，代码见下图：

最后扛住1000并发。10000会报错连接超时，但是后续请求会完全正常得到响应。

因为本地性能受限，所以10s后报超时我觉得是正常的

15:56:42.773 [http-nio-8093-exec-450] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - [log,175] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool] with root cause
java.util.NoSuchElementException: Timeout waiting for idle object, borrowMaxWaitDuration=PT10S
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:312)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
at redis.clients.util.Pool.getResource(Pool.java:49)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.getJedis(GeneralAuthorizationAspect.java:222)
at com.chint.cdc.app.maintenance.authorization.GeneralAuth.GeneralAuthorizationAspect.around(GeneralAuthorizationAspect.java:134)
at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)

热门相关：骑士归来刺客之王刺客之王薄先生，情不由己豪门闪婚：帝少的神秘冷妻