This is a follow up to #2985 (52828e4) adding similar support for the
client connection pool. For Rails servers, Sidekiq is not loaded from
the CLI so the prior change to support setting the concurrency via
`RAILS_MAX_THREADS` is not applied to the web server process. This means
for Rails servers which do not configure a custom size through an
initializer they will run with the default connection pool size of 5.
When the Rails server runs the initial Redis connection may be made
through `Sidekiq::Client` (e.g. from [`ActiveJob::QueueAdapters::SidekiqAdapter`](https://github.com/rails/rails/blob/v5.1.5/activejob/lib/active_job/queue_adapters/sidekiq_adapter.rb#L20)).
This causes the `redis_pool` to be initialized without any options,
setting the pool size to the default of 5.
.gem/ruby/2.5.0/gems/sidekiq-5.1.1/lib/sidekiq.rb:125:in `redis_pool'
.gem/ruby/2.5.0/gems/sidekiq-5.1.1/lib/sidekiq/client.rb:42:in `initialize'
.gem/ruby/2.5.0/gems/sidekiq-5.1.1/lib/sidekiq/client.rb:131:in `new'
.gem/ruby/2.5.0/gems/sidekiq-5.1.1/lib/sidekiq/client.rb:131:in `push'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/queue_adapters/sidekiq_adapter.rb:20:in `enqueue'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/enqueuing.rb:51:in `block in enqueue'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/logging.rb:15:in `block (3 levels) in <module:Logging>'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/logging.rb:44:in `block in tag_logger'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:69:in `block in tagged'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:26:in `tagged'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:69:in `tagged'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/logging.rb:44:in `tag_logger'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/callbacks.rb:117:in `instance_exec'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
.gem/ruby/2.5.0/gems/activesupport-5.1.5/lib/active_support/callbacks.rb:135:in `run_callbacks'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/enqueuing.rb:47:in `enqueue'
.gem/ruby/2.5.0/gems/activejob-5.1.5/lib/active_job/enqueuing.rb:18:in `perform_later'
For the majority of cases, a client pool size of 5 is sufficient.
However, servers which utilize a high number of threads, with large job
payloads, and which may experience some network latency issues can see
`Timeout::Error` crashes. This may be further exacerbated by the ~2-20x
performance decrease through `ActiveJob` (#3782). Rails addresses this
general type of connection issue for the main database by suggesting
that the DB pool size match the number of threads running. This change
applies that logic to the default client pool size by leveraging the
same environment setting; this way there's a connection available per
thread.
This may also have the side effect of a slight performance boost, as
there is less of a chance that threads will be blocked waiting on
connections. The trade-off is that there may be a memory profile
increase to handle the additional Redis connections in the pool; note
the pool only creates new connections as necessary to handle the
requests.
Resolves#3806
b23c0ecf8e