2013-01-11 10:56:41 +00:00
|
|
|
## Description
|
|
|
|
|
2013-01-17 15:21:10 +00:00
|
|
|
Sidekiq strategy to restrict number of workers
|
2013-01-11 10:56:41 +00:00
|
|
|
which are able to run specified queues simultaneously.
|
|
|
|
|
2013-02-05 11:37:22 +00:00
|
|
|
[![Build Status](https://secure.travis-ci.org/brainopia/sidekiq-limit_fetch.png)](http://travis-ci.org/brainopia/sidekiq-limit_fetch)
|
|
|
|
[![Gem Version](https://badge.fury.io/rb/sidekiq-limit_fetch.png)](http://badge.fury.io/rb/sidekiq-limit_fetch)
|
|
|
|
[![Dependency Status](https://gemnasium.com/brainopia/sidekiq-limit_fetch.png)](https://gemnasium.com/brainopia/sidekiq-limit_fetch)
|
|
|
|
[![Code Climate](https://codeclimate.com/github/brainopia/sidekiq-limit_fetch.png)](https://codeclimate.com/github/brainopia/sidekiq-limit_fetch)
|
2013-01-24 13:54:26 +00:00
|
|
|
|
2013-01-11 10:56:41 +00:00
|
|
|
## Installation
|
|
|
|
|
|
|
|
Add this line to your application's Gemfile:
|
|
|
|
|
|
|
|
gem 'sidekiq-limit_fetch'
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
### Limits
|
|
|
|
|
2013-01-11 10:56:41 +00:00
|
|
|
Specify limits which you want to place on queues inside sidekiq.yml:
|
|
|
|
|
|
|
|
```yaml
|
2013-01-24 15:35:54 +00:00
|
|
|
:limits:
|
|
|
|
queue_name1: 5
|
|
|
|
queue_name2: 10
|
2013-01-11 10:56:41 +00:00
|
|
|
```
|
|
|
|
|
2013-01-17 15:18:56 +00:00
|
|
|
Or set it dynamically in your code:
|
|
|
|
```ruby
|
2013-01-24 12:03:28 +00:00
|
|
|
Sidekiq::Queue['queue_name1'].limit = 5
|
2013-01-17 15:18:56 +00:00
|
|
|
Sidekiq::Queue['queue_name2'].limit = 10
|
|
|
|
```
|
|
|
|
|
|
|
|
In these examples, tasks for the ```queue_name1``` will be run by at most 5
|
|
|
|
workers at the same time and the ```queue_name2``` will have no more than 10
|
2013-01-11 11:02:39 +00:00
|
|
|
workers simultaneously.
|
2013-01-11 13:07:37 +00:00
|
|
|
|
2013-01-17 15:18:56 +00:00
|
|
|
Ability to set limits dynamically allows you to resize worker
|
|
|
|
distribution among queues any time you want.
|
|
|
|
|
2013-01-24 15:35:54 +00:00
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
### Pauses
|
|
|
|
|
2013-01-17 15:49:45 +00:00
|
|
|
You can also pause your queues temporarely. Upon continuing their limits
|
|
|
|
will be preserved.
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
Sidekiq::Queue['name'].pause # prevents workers from running tasks from this queue
|
2013-01-25 07:18:07 +00:00
|
|
|
Sidekiq::Queue['name'].paused? # => true
|
|
|
|
Sidekiq::Queue['name'].unpause # allows workers to use the queue
|
2013-01-17 15:49:45 +00:00
|
|
|
```
|
|
|
|
|
2013-01-24 15:35:54 +00:00
|
|
|
|
2013-01-25 07:18:07 +00:00
|
|
|
You can see how many workers currently handling a queue:
|
2013-01-24 15:35:54 +00:00
|
|
|
|
|
|
|
```ruby
|
|
|
|
Sidekiq::Queue['name'].busy # number of busy workers
|
|
|
|
```
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
### Multiple processes
|
2013-01-24 15:35:54 +00:00
|
|
|
|
2013-01-18 11:46:58 +00:00
|
|
|
Limits are applied per process. In case you have several worker
|
|
|
|
processes and want to have global locks between them, you'll need to
|
2013-01-24 12:03:28 +00:00
|
|
|
enable global mode by setting global option, eg:
|
|
|
|
|
|
|
|
```yaml
|
2013-01-24 15:35:54 +00:00
|
|
|
:global: true
|
2013-01-24 12:03:28 +00:00
|
|
|
```
|
2013-01-17 15:18:56 +00:00
|
|
|
|
2013-01-25 07:38:32 +00:00
|
|
|
or
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
Sidekiq.options[:global] = true
|
|
|
|
```
|
|
|
|
|
2013-01-24 15:35:54 +00:00
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
### Blocking queue mode
|
|
|
|
|
2013-01-25 07:18:07 +00:00
|
|
|
If you use strict queue ordering (it will be used if you don't specify queue weights)
|
|
|
|
then you can set blocking status for queues. It means if a blocking
|
|
|
|
queue task is executing then no new task from lesser priority queues will
|
|
|
|
be ran. Eg,
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
:queues:
|
|
|
|
- a
|
|
|
|
- b
|
|
|
|
- c
|
2013-01-25 07:38:32 +00:00
|
|
|
:blocking:
|
2013-01-25 07:18:07 +00:00
|
|
|
- b
|
|
|
|
```
|
|
|
|
|
|
|
|
In this case when a task for `b` queue is ran no new task from `c` queue
|
|
|
|
will be started.
|
|
|
|
|
|
|
|
You can also enable and disable blocking mode for queues on the fly:
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
Sidekiq::Queue['name'].block
|
|
|
|
Sidekiq::Queue['name'].blocking? # => true
|
|
|
|
Sidekiq::Queue['name'].unblock
|
|
|
|
```
|
2013-01-24 15:35:54 +00:00
|
|
|
|
|
|
|
|
2013-01-25 07:42:21 +00:00
|
|
|
### Thanks
|
|
|
|
|
2013-01-11 13:07:37 +00:00
|
|
|
Sponsored by [Evil Martians].
|
|
|
|
[Evil Martians]: http://evilmartians.com/
|