Connection pool issues after upgrading to v9.16.0

[yolu16]

Issue tracker is used for reporting bugs and discussing new features. Please use
stackoverflow for supporting issues.

We upgraded from v9.14.1 to v9.16.0, and we started observing recurrent conn[15] rejected by hook, returning to pool and was not able to get a healthy connection after 3 attempts logs. Once these logs start, we observe the db.client.connections.use.time.milliseconds.bucket increases significantly and stays high until we restart our service.

Expected Behavior

We expect the db.client.connections.use.time.milliseconds.bucket to stay stable and the redis connection pool issues to self-resolve without needing to restart the service.

Current Behavior

A few hours after deploying the upgraded service from version v9.14.1 to v9.16.0, we observed a significant increase in the db.client.connections.use.time.milliseconds.bucket metric. Specifically, the 99th percentile latency jumped from approximately 5ms to 10,000ms.

The service was deployed to our development environment last Friday, and we restarted it on Monday upon noticing the issue. The problem became apparent when our event processing began falling behind. During event handling, the service interacts with Redis.

Not sure if it's the same issue as #3571, since we observed an impact on the db.client.connections.use.time.milliseconds.bucket and our service functionality.

Logs:

redis: 2025/11/01 07:07:58 pool.go:481: redis: connection pool: conn[18] rejected by hook, returning to pool
redis: 2025/11/01 07:07:58 pool.go:481: redis: connection pool: conn[18] rejected by hook, returning to pool
redis: 2025/11/01 07:07:59 pool.go:481: redis: connection pool: conn[26] rejected by hook, returning to pool
redis: 2025/11/01 07:08:00 pool.go:481: redis: connection pool: conn[32] rejected by hook, returning to pool
redis: 2025/11/01 07:08:00 pool.go:481: redis: connection pool: conn[32] rejected by hook, returning to pool
redis: 2025/11/01 07:08:01 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 07:08:14 pool.go:481: redis: connection pool: conn[36] rejected by hook, returning to pool
redis: 2025/11/01 07:15:01 pool.go:481: redis: connection pool: conn[29] rejected by hook, returning to pool
redis: 2025/11/01 07:15:01 pool.go:481: redis: connection pool: conn[29] rejected by hook, returning to pool
redis: 2025/11/01 07:15:01 pool.go:608: redis: connection pool: failed to get a usable connection after 2 attempts
redis: 2025/11/01 11:17:45 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 14:19:57 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 14:19:57 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 15:48:07 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 18:48:49 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/01 22:15:57 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 00:11:20 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 00:45:41 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 07:48:02 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 15:06:24 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 16:46:05 pool.go:481: redis: connection pool: conn[138] rejected by hook, returning to pool
redis: 2025/11/02 16:47:10 pool.go:481: redis: connection pool: conn[134] rejected by hook, returning to pool
redis: 2025/11/02 17:18:07 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 22:17:01 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/02 23:47:50 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 00:48:29 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 02:48:00 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 05:48:29 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 07:48:59 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 09:45:53 pool.go:481: redis: connection pool: conn[200] rejected by hook, returning to pool
redis: 2025/11/03 09:45:53 pool.go:481: redis: connection pool: conn[201] rejected by hook, returning to pool
redis: 2025/11/03 09:45:53 pool.go:481: redis: connection pool: conn[201] rejected by hook, returning to pool
redis: 2025/11/03 09:45:53 pool.go:481: redis: connection pool: conn[198] rejected by hook, returning to pool
redis: 2025/11/03 10:00:39 pool.go:481: redis: connection pool: conn[197] rejected by hook, returning to pool
redis: 2025/11/03 10:48:33 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 13:50:05 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts
redis: 2025/11/03 11:47:28 pool.go:449: redis: connection pool: was not able to get a healthy connection after 3 attempts

After the restart, once again after a few hours, we observed the same issue.

Steps to Reproduce

The issue started occurring after upgrading from v9.14.1 to v9.16.0, we're not sure if there is an issue with our configuration. We did not observe this issue when we were using v9.14.0 or v9.14.1.

Here is a sample of our redis config (same as we had in #3542)

type RedisConfig struct {
	Host          string
	Port          int
}

func NewRedisClient(redisConfig RedisConfig) (redis.UniversalClient, error) {
	address := fmt.Sprintf("%s:%d", redisConfig.Host, redisConfig.Port)

	provider, err := entraid.NewDefaultAzureCredentialsProvider(entraid.DefaultAzureCredentialsProviderOptions{
		DefaultAzureIdentityProviderOptions: identity.DefaultAzureIdentityProviderOptions{
			Scopes: []string{"https://redis.azure.com/.default"},
		},
	})
	if err != nil {
		return nil, err
	}

	opt := &redis.ClusterOptions{
		Addrs:                        []string{address},
		StreamingCredentialsProvider: provider,
		TLSConfig: &tls.Config{
			ServerName: redisConfig.Host,
			MinVersion: tls.VersionTLS12,
		},
	}

	client := redis.NewClusterClient(opt)

	if err := redisotel.InstrumentTracing(client); err != nil {
		return nil, err
	}

	if err := redisotel.InstrumentMetrics(client); err != nil {
		return nil, err
	}

	return client, nil
}

Context (Environment)

go 1.25.3
go-redis/v9 v9.16.0
go-redis/extra/redisotel/v9 v9.16.0
go-redis-entraid v1.0.6

Our service is deployed on AKS with three active instances running concurrently. We are using Azure Managed Redis and Managed identities for auth.

[ndyakov]

Hello @yolu16 and thank you for reporting this. redis: connection pool: was not able to get a healthy connection after 3 attempts log message, which sounds scary, is somehow expected if you had old connections in the pool that had to be disconnected because of the default lifetime of the connection and we will change the content of the message. As for the rest of the things mentioned, I will check the code and get back to you later this week.

p.s. I am expecting the use time is exactly this - the connections were alive for longer and this is why you got the redis: connection pool: was not able to get a healthy connection after 3 attempts

[yolu16]

Thank you @ndyakov that makes sense👍Please let me know if there's anything I should adjust in the configuration or add proper handling code to better manage these older connections and to get new connections. I will downgrade to 9.14.1 for now and monitor if this issue also occurs

[ndyakov]

@yolu16 If you have a staging environment, it would be great if you can test ndyakov/state-machine-conn (which will soon be merged in master and released) to verify if it improves stability.