2013-01-11 10:56:41 +00:00
|
|
|
|
## Description
|
|
|
|
|
|
2022-03-29 19:49:57 +00:00
|
|
|
|
*This project has been taken over by [@deanpcmad](https://github.com/deanpcmad)*
|
2013-01-11 10:56:41 +00:00
|
|
|
|
|
2022-03-25 22:55:49 +00:00
|
|
|
|
Sidekiq strategy to support a granular queue control – limiting, pausing, blocking, querying.
|
|
|
|
|
|
|
|
|
|
[![CI](https://github.com/deanpcmad/sidekiq-limit_fetch/actions/workflows/ci.yml/badge.svg)](https://github.com/deanpcmad/sidekiq-limit_fetch/actions/workflows/ci.yml)
|
2017-03-20 16:02:19 +00:00
|
|
|
|
[![Gem Version](https://badge.fury.io/rb/sidekiq-limit_fetch.svg)](http://badge.fury.io/rb/sidekiq-limit_fetch)
|
2013-01-24 13:54:26 +00:00
|
|
|
|
|
2022-03-25 23:20:24 +00:00
|
|
|
|
### Installation
|
2013-01-11 10:56:41 +00:00
|
|
|
|
|
|
|
|
|
Add this line to your application's Gemfile:
|
|
|
|
|
|
2022-03-29 19:49:57 +00:00
|
|
|
|
```
|
|
|
|
|
gem 'sidekiq-limit_fetch'
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Then `bundle install`.
|
2013-01-11 10:56:41 +00:00
|
|
|
|
|
2018-01-17 21:39:40 +00:00
|
|
|
|
### Limitations
|
2013-05-28 10:22:12 +00:00
|
|
|
|
|
2015-10-10 10:22:27 +00:00
|
|
|
|
**Important note:** At this moment, `sidekiq-limit_fetch` is incompatible with
|
2015-10-10 10:23:01 +00:00
|
|
|
|
- sidekiq pro's `reliable_fetch`
|
2015-10-10 10:22:27 +00:00
|
|
|
|
- `sidekiq-rate-limiter`
|
|
|
|
|
- any other plugin that rewrites fetch strategy of sidekiq.
|
2013-11-03 11:08:33 +00:00
|
|
|
|
|
2022-03-25 23:20:24 +00:00
|
|
|
|
### Usage
|
2013-01-11 10:56:41 +00:00
|
|
|
|
|
2022-03-25 23:20:24 +00:00
|
|
|
|
If you are using this with Rails, you don't need to require it as it's done automatically.
|
2021-10-20 10:19:57 +00:00
|
|
|
|
|
2022-03-25 23:20:24 +00:00
|
|
|
|
To use this Gem in other Ruby projects, just add `require 'sidekiq-limit_fetch'`.
|
2016-12-01 21:40:08 +00:00
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
|
### Limits
|
|
|
|
|
|
2013-01-11 10:56:41 +00:00
|
|
|
|
Specify limits which you want to place on queues inside sidekiq.yml:
|
|
|
|
|
|
|
|
|
|
```yaml
|
2022-03-29 19:49:57 +00:00
|
|
|
|
:limits:
|
|
|
|
|
queue_name1: 5
|
|
|
|
|
queue_name2: 10
|
2013-01-11 10:56:41 +00:00
|
|
|
|
```
|
|
|
|
|
|
2013-01-17 15:18:56 +00:00
|
|
|
|
Or set it dynamically in your code:
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['queue_name1'].limit = 5
|
|
|
|
|
Sidekiq::Queue['queue_name2'].limit = 10
|
2013-01-17 15:18:56 +00:00
|
|
|
|
```
|
|
|
|
|
|
2022-03-29 19:49:57 +00:00
|
|
|
|
In these examples, tasks for the `queue_name1` will be run by at most 5
|
|
|
|
|
workers at the same time and the `queue_name2` will have no more than 10
|
2013-01-11 11:02:39 +00:00
|
|
|
|
workers simultaneously.
|
2013-01-11 13:07:37 +00:00
|
|
|
|
|
2013-01-17 15:18:56 +00:00
|
|
|
|
Ability to set limits dynamically allows you to resize worker
|
|
|
|
|
distribution among queues any time you want.
|
|
|
|
|
|
2013-06-22 18:12:42 +00:00
|
|
|
|
### Limits per process
|
|
|
|
|
|
|
|
|
|
If you use multiple sidekiq processes then you can specify limits per process:
|
|
|
|
|
|
2013-06-22 18:14:21 +00:00
|
|
|
|
```yaml
|
2022-03-29 19:49:57 +00:00
|
|
|
|
:process_limits:
|
|
|
|
|
queue_name: 2
|
2013-06-22 18:12:42 +00:00
|
|
|
|
```
|
|
|
|
|
|
2013-06-22 18:14:21 +00:00
|
|
|
|
Or set it in your code:
|
|
|
|
|
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['queue_name'].process_limit = 2
|
2013-06-22 18:12:42 +00:00
|
|
|
|
```
|
|
|
|
|
|
2013-03-07 08:06:18 +00:00
|
|
|
|
### Busy workers by queue
|
|
|
|
|
|
|
|
|
|
You can see how many workers currently handling a queue:
|
|
|
|
|
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['name'].busy # number of busy workers
|
2013-03-07 08:06:18 +00:00
|
|
|
|
```
|
2013-01-24 15:35:54 +00:00
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
|
### Pauses
|
|
|
|
|
|
2017-02-02 17:17:26 +00:00
|
|
|
|
You can also pause your queues temporarily. Upon continuing their limits
|
2013-01-17 15:49:45 +00:00
|
|
|
|
will be preserved.
|
|
|
|
|
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['name'].pause # prevents workers from running tasks from this queue
|
|
|
|
|
Sidekiq::Queue['name'].paused? # => true
|
|
|
|
|
Sidekiq::Queue['name'].unpause # allows workers to use the queue
|
|
|
|
|
Sidekiq::Queue['name'].pause_for_ms(1000) # will pause for a second
|
2013-01-17 15:49:45 +00:00
|
|
|
|
```
|
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
|
### Blocking queue mode
|
|
|
|
|
|
2013-01-25 07:18:07 +00:00
|
|
|
|
If you use strict queue ordering (it will be used if you don't specify queue weights)
|
|
|
|
|
then you can set blocking status for queues. It means if a blocking
|
|
|
|
|
queue task is executing then no new task from lesser priority queues will
|
|
|
|
|
be ran. Eg,
|
|
|
|
|
|
|
|
|
|
```yaml
|
2022-03-29 19:49:57 +00:00
|
|
|
|
:queues:
|
|
|
|
|
- a
|
|
|
|
|
- b
|
|
|
|
|
- c
|
|
|
|
|
:blocking:
|
|
|
|
|
- b
|
2013-01-25 07:18:07 +00:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In this case when a task for `b` queue is ran no new task from `c` queue
|
|
|
|
|
will be started.
|
|
|
|
|
|
|
|
|
|
You can also enable and disable blocking mode for queues on the fly:
|
|
|
|
|
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['name'].block
|
|
|
|
|
Sidekiq::Queue['name'].blocking? # => true
|
|
|
|
|
Sidekiq::Queue['name'].unblock
|
2013-01-25 07:18:07 +00:00
|
|
|
|
```
|
2013-01-24 15:35:54 +00:00
|
|
|
|
|
2013-03-10 11:28:03 +00:00
|
|
|
|
### Advanced blocking queues
|
|
|
|
|
|
|
|
|
|
You can also block on array of queues. It means when any of them is
|
|
|
|
|
running only queues higher and queues from their blocking group can
|
|
|
|
|
run. It will be easier to understand with an example:
|
|
|
|
|
|
|
|
|
|
```yaml
|
2022-03-29 19:49:57 +00:00
|
|
|
|
:queues:
|
|
|
|
|
- a
|
|
|
|
|
- b
|
|
|
|
|
- c
|
|
|
|
|
- d
|
|
|
|
|
:blocking:
|
|
|
|
|
- [b, c]
|
2013-03-10 11:28:03 +00:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In this case tasks from `d` will be blocked when a task from queue `b` or `c` is executed.
|
|
|
|
|
|
|
|
|
|
You can dynamically set exceptions for queue blocking:
|
|
|
|
|
|
|
|
|
|
```ruby
|
2022-03-29 19:49:57 +00:00
|
|
|
|
Sidekiq::Queue['queue1'].block_except 'queue2'
|
2013-03-10 11:28:03 +00:00
|
|
|
|
```
|
2013-01-24 15:35:54 +00:00
|
|
|
|
|
2014-11-21 22:16:30 +00:00
|
|
|
|
### Dynamic queues
|
|
|
|
|
|
|
|
|
|
You can support dynamic queues (that are not listed in sidekiq.yml but
|
|
|
|
|
that have tasks pushed to them (usually with `Sidekiq::Client.push`)).
|
|
|
|
|
|
|
|
|
|
To use this mode you need to specify a following line in sidekiq.yml:
|
|
|
|
|
|
|
|
|
|
```yaml
|
2022-03-29 19:49:57 +00:00
|
|
|
|
:dynamic: true
|
2014-11-21 22:16:30 +00:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Dynamic queues will be ran at the lowest priority.
|
|
|
|
|
|
2015-10-12 13:34:48 +00:00
|
|
|
|
### Maintenance
|
|
|
|
|
|
2022-03-29 19:49:57 +00:00
|
|
|
|
If you use `flushdb`, restart the sidekiq process to re-populate the dynamic configuration.
|