[Bug] The consumer keep executing the reconnection, after all providers are offline #15880
Description
Pre-check
- I am sure that all the content I provide is in English.
Search before asking
- I had searched in the issues and found no similar issues.
Apache Dubbo Component
Java SDK (apache/dubbo)
Dubbo Version
Dubbo Java 3.3.0, OpenJDK 1.8, Springboot 2.7.18
Steps to reproduce this issue
One consumer instance and one provider instance and nacos server, just shutdown provider instance.
And the consumer keep executing the reconnection, after all providers are shutdown.
What you expected to happen
Logs：
2025-12-18 16:22:03.563 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:03.593 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Received instance notification, serviceName: MSEProvider, instances: 0, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] 0 unique working revisions: , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Notify service com.cabbage.demo.mse.common.api.UserService:default:null with urls 1, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.191 WARN 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received url with EMPTY protocol from registry ServiceDiscoveryRegistryDirectory(registry: 127.0.0.1:8848, subscribed key: [MSEProvider, MSEServiceProvider, Provider])-Directory(invokers: 1[172.16.14.122:18616], validInvokers: 1[172.16.14.122:18616], invokersToReconnect: 0[]), will clear all available addresses., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 4-1. This may be caused by , go to https://dubbo.apache.org/faq/4/1 to find instructions.
2025-12-18 16:22:04.193 INFO 39120 --- [ncesChangeEvent] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0x6733d82a, L:/172.16.14.122:53239 - R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .r.c.m.DefaultMigrationAddressComparator : [DUBBO] No instance address available, stop compare., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.cabbage.demo.mse.common.api.UserService:default. Urls Size : 1. Invokers Size : 0. Available Size: 0. Available Invokers : empty, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.203 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] **The client has been closed and will not reconnect**. , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.204 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] The connection [id: 0x6733d82a, L:/172.16.14.122:53239 ! R:/172.16.14.122:18616] of 172.16.14.122:53239 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:13.527 INFO 39120 --- [lientWorker-4-1] o.a.d.r.t.netty4.NettyClientHandler : [DUBBO] The connection [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616] of 172.16.14.122:53220 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.839 INFO 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.840 INFO 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:30.892 ERROR 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyClient : [DUBBO] Failed to connect to provider server by other reason., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-1. This may be caused by network disconnected, go to https://dubbo.apache.org/faq/6/1 to find instructions.
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /172.16.14.122:18616
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_472]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_472]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Thread.java:750) [na:1.8.0_472]
2025-12-18 16:22:30.894 ERROR 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Fail to connect toHeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-16. This may be caused by , go to https://dubbo.apache.org/faq/6/16 to find instructions.
The log tells that the client has been closed and will not reconnect, but the consumer keep executing the reconnection.
In the K8s environment, since the provider pod will have a new IP after restart, the consumer keeps reporting errors of attempting to connect to the old IP, which continuously triggers alerts.
Anything else
No response
Are you willing to submit a pull request to fix on your own?
- Yes I am willing to submit a pull request on my own!
Code of Conduct
- I agree to follow this project's Code of Conduct