kotovalexarian-likes-github/mperham--sidekiq
1
0
Fork 0
mirror of https://github.com/mperham/sidekiq.git synced 2022-11-09 13:52:34 -05:00
Code Releases Activity

Reconnect on "UNBLOCKED force unblock" errors (#4985)

Browse source 
These errors can occur during Sidekiq's long-running job fetching
command. This uses Redis' blocking BRPOP primitive. On failover in a
cluster setup, these commands are interrupted by the server.

This error causes the worker threads to be restarted, but as they are
bubbled up to the top, they cause a lot of spam in our error logging
systems. As related errors from other commands are being handled (see
This commit is contained in:
Franz Liedke 2021-09-08 00:04:01 +02:00 committed by GitHub
parent 0bdec6f144
commit 237c70fe37
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 12 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
lib/sidekiq.rb
View file

@ -100,7 +100,8 @@ module Sidekiq
# 2550 Failover can cause the server to become a replica, need
# to disconnect and reopen the socket to get back to the primary.
# 4495 Use the same logic if we have a "Not enough replicas" error from the primary
if retryable && ex.message =~ /READONLY|NOREPLICAS/
# 4985 Use the same logic when a blocking command is force-unblocked
if retryable && ex.message =~ /READONLY|NOREPLICAS|UNBLOCKED/
conn.disconnect!
retryable = false
retry

10
test/test_sidekiq.rb
View file

@ -96,6 +96,16 @@ describe Sidekiq do
assert_equal 2, counts.size
assert_equal counts[0] + 1, counts[1]
end
it 'reconnects if instance state changed' do
counts = []
Sidekiq.redis do |c|
counts << c.info['total_connections_received'].to_i
raise Redis::CommandError, "UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)" if counts.size == 1
end
assert_equal 2, counts.size
assert_equal counts[0] + 1, counts[1]
end
end
describe 'redis info' do