mirror of
https://github.com/mperham/sidekiq.git
synced 2022-11-09 13:52:34 -05:00
Merge branch 'master' into frozen
This commit is contained in:
commit
72fe3289ea
42 changed files with 777 additions and 161 deletions
0
Contributing.md → .github/contributing.md
vendored
0
Contributing.md → .github/contributing.md
vendored
|
@ -3,15 +3,16 @@ sudo: false
|
|||
cache: bundler
|
||||
services:
|
||||
- redis-server
|
||||
before_install:
|
||||
- gem install bundler
|
||||
- gem update bundler
|
||||
rvm:
|
||||
- 2.1.7
|
||||
- 2.0.0
|
||||
- 2.1.8
|
||||
- 2.2.4
|
||||
- 2.3.0
|
||||
- jruby
|
||||
- jruby-head
|
||||
- rbx-2
|
||||
matrix:
|
||||
allow_failures:
|
||||
- rvm: rbx-2
|
||||
- rvm: jruby-head
|
||||
|
|
|
@ -28,7 +28,7 @@ In order to use the Software under this Agreement, you must receive a “Source
|
|||
|
||||
3.1 You shall not (and shall not allow any third party to): (a) decompile, disassemble, or otherwise reverse engineer the Software or attempt to reconstruct or discover any source code, underlying ideas, algorithms, file formats or programming interfaces of the Software by any means whatsoever (except and only to the extent that applicable law prohibits or restricts reverse engineering restrictions); (b) distribute, sell, sublicense, rent, lease or use the Software for time sharing, hosting, service provider or like purposes, except as expressly permitted under this Agreement; (c) redistribute the Software or Modifications other than by including the Software or a portion thereof within your own product, which must have substantially different functionality than the Software or Modifications and must not allow any third party to use the Software or Modifications, or any portions thereof, for software development or application development purposes; (d) redistribute the Software as part of a product, "appliance" or "virtual server"; (e) redistribute the Software on any server which is not directly under your control; (f) remove any product identification, proprietary, copyright or other notices contained in the Software; (g) modify any part of the Software, create a derivative work of any part of the Software (except as permitted in Section 4), or incorporate the Software, except to the extent expressly authorized in writing by Contributed Systems; (h) publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software; (i) utilize any equipment, device, software, or other means designed to circumvent or remove any form of Source URL or copy protection used by Contributed Systems in connection with the Software, or use the Software together with any authorization code, Source URL, serial number, or other copy protection device not supplied by Contributed Systems; (j) use the Software to develop a product which is competitive with any Contributed Systems product offerings; or (k) use unauthorized Source URLS or keycode(s) or distribute or publish Source URLs or keycode(s), except as may be expressly permitted by Contributed Systems in writing. If your unique Source URL is ever published, Contributed Systems reserves the right to terminate your access without notice.
|
||||
|
||||
3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE FOR A PRODUCT THAT IS INTENDED FOR SOFTWARE OR APPLICATION DEVELOPMENT PURPOSES.
|
||||
3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE AS PART OF A PRODUCT OR SERVICE THAT PROVIDES SIMILAR FUNCTIONALITY TO THE SOFTWARE ITSELF.
|
||||
|
||||
The Open Source version of the Software (“LGPL Version”) is licensed
|
||||
under the terms of the GNU Lesser General Public License versions 3.0
|
||||
|
|
56
Changes.md
56
Changes.md
|
@ -1,8 +1,58 @@
|
|||
# Sidekiq Changes
|
||||
|
||||
4.1.2
|
||||
-----------
|
||||
|
||||
- Client middleware can now stop bulk job push. [#2887]
|
||||
|
||||
4.1.1
|
||||
-----------
|
||||
|
||||
- Much better behavior when Redis disappears and comes back. [#2866]
|
||||
- Update FR locale [dbachet]
|
||||
- Don't fill logfile in case of Redis downtime [#2860]
|
||||
- Allow definition of a global retries_exhausted handler. [#2807]
|
||||
```ruby
|
||||
Sidekiq.configure_server do |config|
|
||||
config.default_retries_exhausted = -> (job, ex) do
|
||||
Sidekiq.logger.info "#{job['class']} job is now dead"
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
4.1.0
|
||||
-----------
|
||||
|
||||
- Tag quiet processes in the Web UI [#2757, jcarlson]
|
||||
- Pass last exception to sidekiq\_retries\_exhausted block [#2787, Nowaker]
|
||||
```ruby
|
||||
class MyWorker
|
||||
include Sidekiq::Worker
|
||||
sidekiq_retries_exhausted do |job, exception|
|
||||
end
|
||||
end
|
||||
```
|
||||
- Add native support for ActiveJob's `set(options)` method allowing
|
||||
you to override worker options dynamically. This should make it
|
||||
even easier to switch between ActiveJob and Sidekiq's native APIs [#2780]
|
||||
```ruby
|
||||
class MyWorker
|
||||
include Sidekiq::Worker
|
||||
sidekiq_options queue: 'default', retry: true
|
||||
|
||||
def perform(*args)
|
||||
# do something
|
||||
end
|
||||
end
|
||||
|
||||
MyWorker.set(queue: 'high', retry: false).perform_async(1)
|
||||
```
|
||||
|
||||
4.0.2
|
||||
-----------
|
||||
|
||||
- Better Japanese translations
|
||||
- Remove `json` gem dependency from gemspec. [#2743]
|
||||
- There's a new testing API based off the `Sidekiq::Queues` namespace. All
|
||||
assertions made against the Worker class still work as expected.
|
||||
[#2676, brandonhilkert]
|
||||
|
@ -28,6 +78,12 @@ Sidekiq::Queues.clear_all
|
|||
[detailed on my blog](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
|
||||
- See the [4.0 upgrade notes](4.0-Upgrade.md) for more detail.
|
||||
|
||||
3.5.4
|
||||
-----------
|
||||
|
||||
- Ensure exception message is a string [#2707]
|
||||
- Revert racy Process.kill usage in sidekiqctl
|
||||
|
||||
3.5.3
|
||||
-----------
|
||||
|
||||
|
|
|
@ -3,6 +3,38 @@ Sidekiq Enterprise Changelog
|
|||
|
||||
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
|
||||
|
||||
HEAD
|
||||
-------------
|
||||
|
||||
- Add API to check if a unique lock is present. See [#2932] for details.
|
||||
|
||||
1.2.1
|
||||
-------------
|
||||
|
||||
- Multi-Process mode can now monitor the RSS memory of children and
|
||||
restart any that grow too large. To limit children to 1GB each:
|
||||
```
|
||||
MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
|
||||
```
|
||||
|
||||
1.2.0
|
||||
-------------
|
||||
|
||||
- **NEW FEATURE** Multi-process mode! Sidekiq Enterprise can now fork multiple worker
|
||||
processes, enabling significant memory savings. See the [wiki
|
||||
documentation](https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process) for details.
|
||||
|
||||
|
||||
0.7.10
|
||||
-------------
|
||||
|
||||
- More precise gemspec dependency versioning
|
||||
|
||||
1.1.0
|
||||
-------------
|
||||
|
||||
- **NEW FEATURE** Historical queue metrics, [documented in the wiki](https://github.com/mperham/sidekiq/wiki/Ent-Historical-Metrics) [#2719]
|
||||
|
||||
0.7.9, 1.0.2
|
||||
-------------
|
||||
|
||||
|
|
|
@ -3,6 +3,75 @@ Sidekiq Pro Changelog
|
|||
|
||||
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
|
||||
|
||||
3.2.1
|
||||
---------
|
||||
|
||||
- timed\_fetch now works with namespaces. [ryansch]
|
||||
|
||||
|
||||
3.2.0
|
||||
---------
|
||||
|
||||
- Fixed detection of missing batches, `NoSuchBatch` should be raised
|
||||
properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
|
||||
longer in Redis.
|
||||
- Remove support for Pro 1.x format batches. This version will no
|
||||
longer seamlessly process batches created with Sidekiq Pro 1.x.
|
||||
As always, upgrade one major version at a time to ensure a smooth
|
||||
transition.
|
||||
- Fix edge case where a parent batch could expire before a child batch
|
||||
was finished processing, leading to missing batches [#2889]
|
||||
|
||||
2.1.5
|
||||
---------
|
||||
|
||||
- Fix edge case where a parent batch could expire before a child batch
|
||||
was finished processing, leading to missing batches [#2889]
|
||||
|
||||
3.1.0
|
||||
---------
|
||||
|
||||
- New container-friendly fetch algorithm: `timed_fetch`. See the
|
||||
[wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
|
||||
for trade offs between the two reliability options. You should
|
||||
use this if you are on Heroku, Docker, Amazon ECS or EBS or
|
||||
another container-based system.
|
||||
|
||||
|
||||
3.0.6
|
||||
---------
|
||||
|
||||
- Fix race condition on reliable fetch shutdown
|
||||
|
||||
3.0.5
|
||||
---------
|
||||
|
||||
- Statsd metrics now account for ActiveJob class names
|
||||
- Allow reliable fetch internals to be overridden [jonhyman]
|
||||
|
||||
3.0.4
|
||||
---------
|
||||
|
||||
- Queue pausing no longer requires reliable fetch. [#2786]
|
||||
|
||||
3.0.3, 2.1.4
|
||||
------------
|
||||
|
||||
- Convert Lua-based `Sidekiq::Queue#delete_by_class` to Ruby-based, to
|
||||
avoid O(N^2) performance and possible Redis failure. [#2806]
|
||||
|
||||
3.0.2
|
||||
-----------
|
||||
|
||||
- Make job registration with batch part of the atomic push so batch
|
||||
metadata can't get out of sync with the job data. [#2714]
|
||||
|
||||
3.0.1
|
||||
-----------
|
||||
|
||||
- Remove a number of Redis version checks since we can assume 2.8+ now.
|
||||
- Fix expiring jobs client middleware not loaded on server
|
||||
|
||||
3.0.0
|
||||
-----------
|
||||
|
||||
|
|
33
README.md
33
README.md
|
@ -11,7 +11,7 @@ Simple, efficient background processing for Ruby.
|
|||
|
||||
Sidekiq uses threads to handle many jobs at the same time in the
|
||||
same process. It does not require Rails but will integrate tightly with
|
||||
Rails 3/4 to make background processing dead simple.
|
||||
Rails to make background processing dead simple.
|
||||
|
||||
Sidekiq is compatible with Resque. It uses the exact same
|
||||
message format as Resque so it can integrate into an existing Resque processing farm.
|
||||
|
@ -31,10 +31,9 @@ DelayedJob 4.1.1 | - | - | 465 sec | 215 jobs/sec
|
|||
Requirements
|
||||
-----------------
|
||||
|
||||
I test with the latest CRuby (2.2, 2.1 and 2.0) and JRuby versions (9k). Other versions/VMs
|
||||
are untested but might work fine. CRuby 1.9 is not supported.
|
||||
Sidekiq supports CRuby 2.0+ and JRuby 9k.
|
||||
|
||||
All Rails releases from 3.2 are officially supported.
|
||||
All Rails releases >= 3.2 are officially supported.
|
||||
|
||||
Redis 2.8 or greater is required. 3.0.3+ is recommended for large
|
||||
installations with thousands of worker threads.
|
||||
|
@ -63,15 +62,8 @@ features, a commercial-friendly license and allow you to support high
|
|||
quality open source development all at the same time. Please see the
|
||||
[Sidekiq](http://sidekiq.org/) homepage for more detail.
|
||||
|
||||
|
||||
More Information
|
||||
-----------------
|
||||
|
||||
Please see the [sidekiq wiki](https://github.com/mperham/sidekiq/wiki) for the official documentation.
|
||||
[mperham/sidekiq on Gitter](https://gitter.im/mperham/sidekiq) is dedicated to this project,
|
||||
but bug reports or feature requests suggestions should still go through [issues on Github](https://github.com/mperham/sidekiq/issues). Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
|
||||
|
||||
You may also find useful a [Reddit area](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
|
||||
Subscribe to the **[quarterly newsletter](https://tinyletter.com/sidekiq)** to stay informed about the latest
|
||||
features and changes to Sidekiq and its bigger siblings.
|
||||
|
||||
|
||||
Problems?
|
||||
|
@ -79,9 +71,20 @@ Problems?
|
|||
|
||||
**Please do not directly email any Sidekiq committers with questions or problems.** A community is best served when discussions are held in public.
|
||||
|
||||
If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages. Searching the issues for your problem is also a good idea. If that doesn't help, feel free to email the Sidekiq mailing list, chat in Gitter, or open a new issue.
|
||||
StackOverflow or Reddit is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
|
||||
If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
|
||||
Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
|
||||
|
||||
Useful resources:
|
||||
|
||||
* Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
|
||||
* Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
|
||||
* Here's a [Reddit forum](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion
|
||||
* The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
|
||||
|
||||
**No support via Twitter, 140 characters is not enough.**
|
||||
|
||||
Every Friday morning 9am Pacific is Sidekiq happy hour: I video chat and answer questions.
|
||||
See the [Sidekiq support page](http://sidekiq.org/support).
|
||||
|
||||
Thanks
|
||||
-----------------
|
||||
|
|
50
code_of_conduct.md
Normal file
50
code_of_conduct.md
Normal file
|
@ -0,0 +1,50 @@
|
|||
# Contributor Code of Conduct
|
||||
|
||||
As contributors and maintainers of this project, and in the interest of
|
||||
fostering an open and welcoming community, we pledge to respect all people who
|
||||
contribute through reporting issues, posting feature requests, updating
|
||||
documentation, submitting pull requests or patches, and other activities.
|
||||
|
||||
We are committed to making participation in this project a harassment-free
|
||||
experience for everyone, regardless of level of experience, gender, gender
|
||||
identity and expression, sexual orientation, disability, personal appearance,
|
||||
body size, race, ethnicity, age, religion, or nationality.
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery
|
||||
* Personal attacks
|
||||
* Trolling or insulting/derogatory comments
|
||||
* Public or private harassment
|
||||
* Publishing other's private information, such as physical or electronic
|
||||
addresses, without explicit permission
|
||||
* Other unethical or unprofessional conduct
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
By adopting this Code of Conduct, project maintainers commit themselves to
|
||||
fairly and consistently applying these principles to every aspect of managing
|
||||
this project. Project maintainers who do not follow or enforce the Code of
|
||||
Conduct may be permanently removed from the project team.
|
||||
|
||||
This Code of Conduct applies both within project spaces and in public spaces
|
||||
when an individual is representing the project or its community.
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project maintainer at mperham AT gmail.com. All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. Maintainers are
|
||||
obligated to maintain confidentiality with regard to the reporter of an
|
||||
incident.
|
||||
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 1.3.0, available at
|
||||
[http://contributor-covenant.org/version/1/3/0/][version]
|
||||
|
||||
[homepage]: http://contributor-covenant.org
|
||||
[version]: http://contributor-covenant.org/version/1/3/0/
|
|
@ -1,7 +1,7 @@
|
|||
# frozen_string_literal: true
|
||||
# encoding: utf-8
|
||||
require 'sidekiq/version'
|
||||
fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby 1.9." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.0.0'
|
||||
fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.0.0." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.0.0'
|
||||
|
||||
require 'sidekiq/logging'
|
||||
require 'sidekiq/client'
|
||||
|
@ -38,6 +38,14 @@ module Sidekiq
|
|||
'queue' => 'default'
|
||||
}
|
||||
|
||||
FAKE_INFO = {
|
||||
"redis_version" => "9.9.9",
|
||||
"uptime_in_days" => "9999",
|
||||
"connected_clients" => "9999",
|
||||
"used_memory_human" => "9P",
|
||||
"used_memory_peak_human" => "9P"
|
||||
}.freeze
|
||||
|
||||
def self.❨╯°□°❩╯︵┻━┻
|
||||
puts "Calm down, yo."
|
||||
end
|
||||
|
@ -45,7 +53,6 @@ module Sidekiq
|
|||
def self.options
|
||||
@options ||= DEFAULTS.dup
|
||||
end
|
||||
|
||||
def self.options=(opts)
|
||||
@options = opts
|
||||
end
|
||||
|
@ -92,6 +99,24 @@ module Sidekiq
|
|||
end
|
||||
end
|
||||
|
||||
def self.redis_info
|
||||
redis do |conn|
|
||||
begin
|
||||
# admin commands can't go through redis-namespace starting
|
||||
# in redis-namespace 2.0
|
||||
if conn.respond_to?(:namespace)
|
||||
conn.redis.info
|
||||
else
|
||||
conn.info
|
||||
end
|
||||
rescue Redis::CommandError => ex
|
||||
#2850 return fake version when INFO command has (probably) been renamed
|
||||
raise unless ex.message =~ /unknown command/
|
||||
FAKE_INFO
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.redis_pool
|
||||
@redis ||= Sidekiq::RedisConnection.create
|
||||
end
|
||||
|
@ -133,15 +158,24 @@ module Sidekiq
|
|||
def self.default_worker_options=(hash)
|
||||
@default_worker_options = default_worker_options.merge(hash.stringify_keys)
|
||||
end
|
||||
|
||||
def self.default_worker_options
|
||||
defined?(@default_worker_options) ? @default_worker_options : DEFAULT_WORKER_OPTIONS
|
||||
end
|
||||
|
||||
# Sidekiq.configure_server do |config|
|
||||
# config.default_retries_exhausted = -> (job, ex) do
|
||||
# end
|
||||
# end
|
||||
def self.default_retries_exhausted=(prok)
|
||||
@default_retries_exhausted = prok
|
||||
end
|
||||
def self.default_retries_exhausted
|
||||
@default_retries_exhausted
|
||||
end
|
||||
|
||||
def self.load_json(string)
|
||||
JSON.parse(string)
|
||||
end
|
||||
|
||||
def self.dump_json(object)
|
||||
JSON.generate(object)
|
||||
end
|
||||
|
@ -149,7 +183,6 @@ module Sidekiq
|
|||
def self.logger
|
||||
Sidekiq::Logging.logger
|
||||
end
|
||||
|
||||
def self.logger=(log)
|
||||
Sidekiq::Logging.logger = log
|
||||
end
|
||||
|
|
|
@ -192,8 +192,11 @@ module Sidekiq
|
|||
class Queue
|
||||
include Enumerable
|
||||
|
||||
##
|
||||
# Return all known queues within Redis.
|
||||
#
|
||||
def self.all
|
||||
Sidekiq.redis {|c| c.smembers('queues'.freeze) }.sort.map {|q| Sidekiq::Queue.new(q) }
|
||||
Sidekiq.redis { |c| c.smembers('queues'.freeze) }.sort.map { |q| Sidekiq::Queue.new(q) }
|
||||
end
|
||||
|
||||
attr_reader :name
|
||||
|
@ -212,6 +215,11 @@ module Sidekiq
|
|||
false
|
||||
end
|
||||
|
||||
##
|
||||
# Calculates this queue's latency, the difference in seconds since the oldest
|
||||
# job in the queue was enqueued.
|
||||
#
|
||||
# @return Float
|
||||
def latency
|
||||
entry = Sidekiq.redis do |conn|
|
||||
conn.lrange(@rname, -1, -1)
|
||||
|
@ -228,7 +236,7 @@ module Sidekiq
|
|||
|
||||
while true do
|
||||
range_start = page * page_size - deleted_size
|
||||
range_end = page * page_size - deleted_size + (page_size - 1)
|
||||
range_end = range_start + page_size - 1
|
||||
entries = Sidekiq.redis do |conn|
|
||||
conn.lrange @rname, range_start, range_end
|
||||
end
|
||||
|
@ -241,6 +249,11 @@ module Sidekiq
|
|||
end
|
||||
end
|
||||
|
||||
##
|
||||
# Find the job with the given JID within this queue.
|
||||
#
|
||||
# This is a slow, inefficient operation. Do not use under
|
||||
# normal conditions. Sidekiq Pro contains a faster version.
|
||||
def find_job(jid)
|
||||
detect { |j| j.jid == jid }
|
||||
end
|
||||
|
@ -265,6 +278,7 @@ module Sidekiq
|
|||
#
|
||||
class Job
|
||||
attr_reader :item
|
||||
attr_reader :value
|
||||
|
||||
def initialize(item, queue_name=nil)
|
||||
@value = item
|
||||
|
@ -350,7 +364,7 @@ module Sidekiq
|
|||
end
|
||||
|
||||
def [](name)
|
||||
@item.__send__(:[], name)
|
||||
@item[name]
|
||||
end
|
||||
|
||||
private
|
||||
|
@ -503,7 +517,7 @@ module Sidekiq
|
|||
|
||||
while true do
|
||||
range_start = page * page_size + offset_size
|
||||
range_end = page * page_size + offset_size + (page_size - 1)
|
||||
range_end = range_start + page_size - 1
|
||||
elements = Sidekiq.redis do |conn|
|
||||
conn.zrange name, range_start, range_end, with_scores: true
|
||||
end
|
||||
|
@ -532,6 +546,11 @@ module Sidekiq
|
|||
end
|
||||
end
|
||||
|
||||
##
|
||||
# Find the job with the given JID within this sorted set.
|
||||
#
|
||||
# This is a slow, inefficient operation. Do not use under
|
||||
# normal conditions. Sidekiq Pro contains a faster version.
|
||||
def find_job(jid)
|
||||
self.detect { |j| j.jid == jid }
|
||||
end
|
||||
|
@ -634,7 +653,6 @@ module Sidekiq
|
|||
#
|
||||
# Yields a Sidekiq::Process.
|
||||
#
|
||||
|
||||
class ProcessSet
|
||||
include Enumerable
|
||||
|
||||
|
@ -675,13 +693,13 @@ module Sidekiq
|
|||
# you'll be happier this way
|
||||
result = conn.pipelined do
|
||||
procs.each do |key|
|
||||
conn.hmget(key, 'info', 'busy', 'beat')
|
||||
conn.hmget(key, 'info', 'busy', 'beat', 'quiet')
|
||||
end
|
||||
end
|
||||
|
||||
result.each do |info, busy, at_s|
|
||||
result.each do |info, busy, at_s, quiet|
|
||||
hash = Sidekiq.load_json(info)
|
||||
yield Process.new(hash.merge('busy' => busy.to_i, 'beat' => at_s.to_f))
|
||||
yield Process.new(hash.merge('busy' => busy.to_i, 'beat' => at_s.to_f, 'quiet' => quiet))
|
||||
end
|
||||
end
|
||||
|
||||
|
@ -698,7 +716,8 @@ module Sidekiq
|
|||
end
|
||||
|
||||
#
|
||||
# Sidekiq::Process has a set of attributes which look like this:
|
||||
# Sidekiq::Process represents an active Sidekiq process talking with Redis.
|
||||
# Each process has a set of attributes which look like this:
|
||||
#
|
||||
# {
|
||||
# 'hostname' => 'app-1.example.com',
|
||||
|
@ -740,6 +759,10 @@ module Sidekiq
|
|||
signal('TTIN')
|
||||
end
|
||||
|
||||
def stopping?
|
||||
self['quiet'] == 'true'
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def signal(sig)
|
||||
|
@ -758,6 +781,7 @@ module Sidekiq
|
|||
end
|
||||
|
||||
##
|
||||
# A worker is a thread that is currently processing a job.
|
||||
# Programmatic access to the current active worker set.
|
||||
#
|
||||
# WARNING WARNING WARNING
|
||||
|
|
|
@ -66,20 +66,17 @@ module Sidekiq
|
|||
logger.info Sidekiq::LICENSE
|
||||
logger.info "Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org" unless defined?(::Sidekiq::Pro)
|
||||
|
||||
Sidekiq.redis do |conn|
|
||||
# touch the connection pool so it is created before we
|
||||
# fire startup and start multithreading.
|
||||
ver = conn.info['redis_version']
|
||||
raise "You are using Redis v#{ver}, Sidekiq requires Redis v2.8.0 or greater" if ver < '2.8'
|
||||
end
|
||||
# touch the connection pool so it is created before we
|
||||
# fire startup and start multithreading.
|
||||
ver = Sidekiq.redis_info['redis_version']
|
||||
raise "You are using Redis v#{ver}, Sidekiq requires Redis v2.8.0 or greater" if ver < '2.8'
|
||||
|
||||
# Before this point, the process is initializing with just the main thread.
|
||||
# Starting here the process will now have multiple threads running.
|
||||
fire_event(:startup)
|
||||
|
||||
logger.debug {
|
||||
"Middleware: #{Sidekiq.server_middleware.map(&:klass).join(', ')}"
|
||||
}
|
||||
logger.debug { "Client Middleware: #{Sidekiq.client_middleware.map(&:klass).join(', ')}" }
|
||||
logger.debug { "Server Middleware: #{Sidekiq.server_middleware.map(&:klass).join(', ')}" }
|
||||
|
||||
if !options[:daemon]
|
||||
logger.info 'Starting processing, hit Ctrl-C to stop'
|
||||
|
|
|
@ -36,10 +36,8 @@ module Sidekiq
|
|||
# Sidekiq::Client.new(ConnectionPool.new { Redis.new })
|
||||
#
|
||||
# Generally this is only needed for very large Sidekiq installs processing
|
||||
# more than thousands jobs per second. I do not recommend sharding unless
|
||||
# you truly cannot scale any other way (e.g. splitting your app into smaller apps).
|
||||
# Some features, like the API, do not support sharding: they are designed to work
|
||||
# against a single Redis instance only.
|
||||
# thousands of jobs per second. I don't recommend sharding unless you
|
||||
# cannot scale any other way (e.g. splitting your app into smaller apps).
|
||||
def initialize(redis_pool=nil)
|
||||
@redis_pool = redis_pool || Thread.current[:sidekiq_via_pool] || Sidekiq.redis_pool
|
||||
end
|
||||
|
@ -50,11 +48,12 @@ module Sidekiq
|
|||
# queue - the named queue to use, default 'default'
|
||||
# class - the worker class to call, required
|
||||
# args - an array of simple arguments to the perform method, must be JSON-serializable
|
||||
# retry - whether to retry this job if it fails, true or false, default true
|
||||
# retry - whether to retry this job if it fails, default true or an integer number of retries
|
||||
# backtrace - whether to save any error backtrace, default false
|
||||
#
|
||||
# All options must be strings, not symbols. NB: because we are serializing to JSON, all
|
||||
# symbols in 'args' will be converted to strings.
|
||||
# symbols in 'args' will be converted to strings. Note that +backtrace: true+ can take quite a bit of
|
||||
# space in Redis; a large volume of failing jobs can start Redis swapping if you aren't careful.
|
||||
#
|
||||
# Returns a unique Job ID. If middleware stops the job, nil will be returned instead.
|
||||
#
|
||||
|
@ -73,9 +72,8 @@ module Sidekiq
|
|||
|
||||
##
|
||||
# Push a large number of jobs to Redis. In practice this method is only
|
||||
# useful if you are pushing tens of thousands of jobs or more, or if you need
|
||||
# to ensure that a batch doesn't complete prematurely. This method
|
||||
# basically cuts down on the redis round trip latency.
|
||||
# useful if you are pushing thousands of jobs or more. This method
|
||||
# cuts out the redis network round trip latency.
|
||||
#
|
||||
# Takes the same arguments as #push except that args is expected to be
|
||||
# an Array of Arrays. All other keys are duplicated for each job. Each job
|
||||
|
@ -85,10 +83,14 @@ module Sidekiq
|
|||
# Returns an array of the of pushed jobs' jids. The number of jobs pushed can be less
|
||||
# than the number given if the middleware stopped processing for one or more jobs.
|
||||
def push_bulk(items)
|
||||
arg = items['args'].first
|
||||
raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !arg.is_a?(Array)
|
||||
|
||||
normed = normalize_item(items)
|
||||
payloads = items['args'].map do |args|
|
||||
raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !args.is_a?(Array)
|
||||
process_single(items['class'], normed.merge('args' => args, 'jid' => SecureRandom.hex(12), 'enqueued_at' => Time.now.to_f))
|
||||
copy = normed.merge('args' => args, 'jid' => SecureRandom.hex(12), 'enqueued_at' => Time.now.to_f)
|
||||
result = process_single(items['class'], copy)
|
||||
result ? result : nil
|
||||
end.compact
|
||||
|
||||
raw_push(payloads) if !payloads.empty?
|
||||
|
@ -105,13 +107,12 @@ module Sidekiq
|
|||
# end
|
||||
#
|
||||
# Generally this is only needed for very large Sidekiq installs processing
|
||||
# more than thousands jobs per second. I do not recommend sharding unless
|
||||
# you truly cannot scale any other way (e.g. splitting your app into smaller apps).
|
||||
# Some features, like the API, do not support sharding: they are designed to work
|
||||
# against a single Redis instance.
|
||||
# thousands of jobs per second. I do not recommend sharding unless
|
||||
# you cannot scale any other way (e.g. splitting your app into smaller apps).
|
||||
def self.via(pool)
|
||||
raise ArgumentError, "No pool given" if pool.nil?
|
||||
raise RuntimeError, "Sidekiq::Client.via is not re-entrant" if x = Thread.current[:sidekiq_via_pool] && x != pool
|
||||
current_sidekiq_pool = Thread.current[:sidekiq_via_pool]
|
||||
raise RuntimeError, "Sidekiq::Client.via is not re-entrant" if current_sidekiq_pool && current_sidekiq_pool != pool
|
||||
Thread.current[:sidekiq_via_pool] = pool
|
||||
yield
|
||||
ensure
|
||||
|
@ -210,6 +211,7 @@ module Sidekiq
|
|||
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.has_key?('class'.freeze) && item.has_key?('args'.freeze)
|
||||
raise(ArgumentError, "Job args must be an Array") unless item['args'].is_a?(Array)
|
||||
raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item['class'.freeze].is_a?(Class) || item['class'.freeze].is_a?(String)
|
||||
#raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
|
||||
|
||||
normalized_hash(item['class'.freeze])
|
||||
.each{ |key, value| item[key] = value if item[key].nil? }
|
||||
|
|
|
@ -6,7 +6,7 @@ module Sidekiq
|
|||
|
||||
class Logger
|
||||
def call(ex, ctxHash)
|
||||
Sidekiq.logger.warn(ctxHash) if !ctxHash.empty?
|
||||
Sidekiq.logger.warn(Sidekiq.dump_json(ctxHash)) if !ctxHash.empty?
|
||||
Sidekiq.logger.warn "#{ex.class.name}: #{ex.message}"
|
||||
Sidekiq.logger.warn ex.backtrace.join("\n") unless ex.backtrace.nil?
|
||||
end
|
||||
|
|
|
@ -96,7 +96,7 @@ module Sidekiq
|
|||
_, _, _, msg = Sidekiq.redis do |conn|
|
||||
conn.pipelined do
|
||||
conn.sadd('processes', key)
|
||||
conn.hmset(key, 'info', json, 'busy', Processor::WORKER_STATE.size, 'beat', Time.now.to_f)
|
||||
conn.hmset(key, 'info', json, 'busy', Processor::WORKER_STATE.size, 'beat', Time.now.to_f, 'quiet', @done)
|
||||
conn.expire(key, 60)
|
||||
conn.rpop("#{key}-signals")
|
||||
end
|
||||
|
|
|
@ -1,6 +1,7 @@
|
|||
# frozen_string_literal: true
|
||||
require 'time'
|
||||
require 'logger'
|
||||
require 'fcntl'
|
||||
|
||||
module Sidekiq
|
||||
module Logging
|
||||
|
@ -47,7 +48,7 @@ module Sidekiq
|
|||
end
|
||||
|
||||
def self.logger=(log)
|
||||
@logger = (log ? log : Logger.new('/dev/null'))
|
||||
@logger = (log ? log : Logger.new(File::NULL))
|
||||
end
|
||||
|
||||
# This reopens ALL logfiles in the process that have been rotated
|
||||
|
|
|
@ -56,6 +56,9 @@ module Sidekiq
|
|||
fire_event(:quiet, true)
|
||||
end
|
||||
|
||||
# hack for quicker development / testing environment #2774
|
||||
PAUSE_TIME = STDOUT.tty? ? 0.1 : 0.5
|
||||
|
||||
def stop(deadline)
|
||||
quiet
|
||||
fire_event(:shutdown, true)
|
||||
|
@ -63,14 +66,14 @@ module Sidekiq
|
|||
# some of the shutdown events can be async,
|
||||
# we don't have any way to know when they're done but
|
||||
# give them a little time to take effect
|
||||
sleep 0.5
|
||||
sleep PAUSE_TIME
|
||||
return if @workers.empty?
|
||||
|
||||
logger.info { "Pausing to allow workers to finish..." }
|
||||
remaining = deadline - Time.now
|
||||
while remaining > 0.5
|
||||
while remaining > PAUSE_TIME
|
||||
return if @workers.empty?
|
||||
sleep 0.5
|
||||
sleep PAUSE_TIME
|
||||
remaining = deadline - Time.now
|
||||
end
|
||||
return if @workers.empty?
|
||||
|
|
|
@ -8,14 +8,14 @@ module Sidekiq
|
|||
# Automatically retry jobs that fail in Sidekiq.
|
||||
# Sidekiq's retry support assumes a typical development lifecycle:
|
||||
#
|
||||
# 0. push some code changes with a bug in it
|
||||
# 1. bug causes job processing to fail, sidekiq's middleware captures
|
||||
# the job and pushes it onto a retry queue
|
||||
# 2. sidekiq retries jobs in the retry queue multiple times with
|
||||
# an exponential delay, the job continues to fail
|
||||
# 3. after a few days, a developer deploys a fix. the job is
|
||||
# 0. Push some code changes with a bug in it.
|
||||
# 1. Bug causes job processing to fail, Sidekiq's middleware captures
|
||||
# the job and pushes it onto a retry queue.
|
||||
# 2. Sidekiq retries jobs in the retry queue multiple times with
|
||||
# an exponential delay, the job continues to fail.
|
||||
# 3. After a few days, a developer deploys a fix. The job is
|
||||
# reprocessed successfully.
|
||||
# 4. once retries are exhausted, sidekiq will give up and move the
|
||||
# 4. Once retries are exhausted, Sidekiq will give up and move the
|
||||
# job to the Dead Job Queue (aka morgue) where it must be dealt with
|
||||
# manually in the Web UI.
|
||||
# 5. After 6 months on the DJQ, Sidekiq will discard the job.
|
||||
|
@ -130,18 +130,17 @@ module Sidekiq
|
|||
end
|
||||
else
|
||||
# Goodbye dear message, you (re)tried your best I'm sure.
|
||||
retries_exhausted(worker, msg)
|
||||
retries_exhausted(worker, msg, exception)
|
||||
end
|
||||
|
||||
raise exception
|
||||
end
|
||||
|
||||
def retries_exhausted(worker, msg)
|
||||
logger.debug { "Dropping message after hitting the retry maximum: #{msg}" }
|
||||
def retries_exhausted(worker, msg, exception)
|
||||
logger.debug { "Retries exhausted for job" }
|
||||
begin
|
||||
if worker.sidekiq_retries_exhausted_block?
|
||||
worker.sidekiq_retries_exhausted_block.call(msg)
|
||||
end
|
||||
block = worker.sidekiq_retries_exhausted_block || Sidekiq.default_retries_exhausted
|
||||
block.call(msg, exception) if block
|
||||
rescue => e
|
||||
handle_exception(e, { context: "Error calling retries_exhausted for #{worker.class}", job: msg })
|
||||
end
|
||||
|
|
|
@ -111,6 +111,7 @@ module Sidekiq
|
|||
end
|
||||
end
|
||||
sleep(1)
|
||||
nil
|
||||
end
|
||||
|
||||
def process(work)
|
||||
|
|
|
@ -45,7 +45,8 @@ module Sidekiq
|
|||
require 'redis/namespace'
|
||||
Redis::Namespace.new(namespace, :redis => client)
|
||||
rescue LoadError
|
||||
Sidekiq.logger.error("redis-namespace gem not included in Gemfile, cannot use namespace '#{namespace}'")
|
||||
Sidekiq.logger.error("Your Redis configuration use the namespace '#{namespace}' but the redis-namespace gem not included in Gemfile." \
|
||||
"Add the gem to your Gemfile in case you would like to keep using a namespace, otherwise remove the namespace parameter.")
|
||||
exit(-127)
|
||||
end
|
||||
else
|
||||
|
|
|
@ -78,7 +78,9 @@ module Sidekiq
|
|||
# Most likely a problem with redis networking.
|
||||
# Punt and try again at the next interval
|
||||
logger.error ex.message
|
||||
logger.error ex.backtrace.first
|
||||
ex.backtrace.each do |bt|
|
||||
logger.error(bt)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
|
@ -87,6 +89,13 @@ module Sidekiq
|
|||
def wait
|
||||
@sleeper.pop(random_poll_interval)
|
||||
rescue Timeout::Error
|
||||
# expected
|
||||
rescue => ex
|
||||
# if poll_interval_average hasn't been calculated yet, we can
|
||||
# raise an error trying to reach Redis.
|
||||
logger.error ex.message
|
||||
logger.error ex.backtrace.first
|
||||
sleep 5
|
||||
end
|
||||
|
||||
# Calculates a random interval that is ±50% the desired average.
|
||||
|
|
|
@ -171,7 +171,7 @@ module Sidekiq
|
|||
end
|
||||
|
||||
def delete_for(jid, queue, klass)
|
||||
jobs_by_queue[queue].delete_if { |job| job["jid"] == jid }
|
||||
jobs_by_queue[queue.to_s].delete_if { |job| job["jid"] == jid }
|
||||
jobs_by_worker[klass].delete_if { |job| job["jid"] == jid }
|
||||
end
|
||||
|
||||
|
|
|
@ -58,19 +58,5 @@ module Sidekiq
|
|||
end
|
||||
arr.clear
|
||||
end
|
||||
|
||||
def want_a_hertz_donut?
|
||||
# what's a hertz donut?
|
||||
# punch! Hurts, don't it?
|
||||
info = Sidekiq.redis {|c| c.info }
|
||||
if info['connected_clients'].to_i > 1000 && info['hz'].to_i >= 10
|
||||
Sidekiq.logger.warn { "Your Redis `hz` setting is too high at #{info['hz']}. See mperham/sidekiq#2431. Set it to 3 in #{info[:config_file]}" }
|
||||
true
|
||||
else
|
||||
Sidekiq.logger.debug { "Redis hz: #{info['hz']}. Client count: #{info['connected_clients']}" }
|
||||
false
|
||||
end
|
||||
end
|
||||
|
||||
end
|
||||
end
|
||||
|
|
|
@ -1,4 +1,4 @@
|
|||
# frozen_string_literal: true
|
||||
module Sidekiq
|
||||
VERSION = "4.0.2"
|
||||
VERSION = "4.1.1"
|
||||
end
|
||||
|
|
|
@ -18,6 +18,11 @@ module Sidekiq
|
|||
end
|
||||
end
|
||||
|
||||
def clear_caches
|
||||
@@strings = nil
|
||||
@@locale_files = nil
|
||||
end
|
||||
|
||||
def locale_files
|
||||
@@locale_files ||= settings.locales.flat_map do |path|
|
||||
Dir["#{path}/*.yml"]
|
||||
|
@ -118,15 +123,7 @@ module Sidekiq
|
|||
end
|
||||
|
||||
def redis_info
|
||||
Sidekiq.redis do |conn|
|
||||
# admin commands can't go through redis-namespace starting
|
||||
# in redis-namespace 2.0
|
||||
if conn.respond_to?(:namespace)
|
||||
conn.redis.info
|
||||
else
|
||||
conn.info
|
||||
end
|
||||
end
|
||||
Sidekiq.redis_info
|
||||
end
|
||||
|
||||
def root_path
|
||||
|
@ -170,7 +167,7 @@ module Sidekiq
|
|||
|
||||
def display_args(args, truncate_after_chars = 2000)
|
||||
args.map do |arg|
|
||||
h(truncate(to_display(arg)))
|
||||
h(truncate(to_display(arg), truncate_after_chars))
|
||||
end.join(", ")
|
||||
end
|
||||
|
||||
|
|
|
@ -51,6 +51,11 @@ module Sidekiq
|
|||
raise ArgumentError, "Do not call .delay_until on a Sidekiq::Worker class, call .perform_at"
|
||||
end
|
||||
|
||||
def set(options)
|
||||
Thread.current[:sidekiq_worker_set] = options
|
||||
self
|
||||
end
|
||||
|
||||
def perform_async(*args)
|
||||
client_push('class' => self, 'args' => args)
|
||||
end
|
||||
|
@ -75,11 +80,15 @@ module Sidekiq
|
|||
# Allows customization for this type of Worker.
|
||||
# Legal options:
|
||||
#
|
||||
# :queue - use a named queue for this Worker, default 'default'
|
||||
# :retry - enable the RetryJobs middleware for this Worker, default *true*
|
||||
# :backtrace - whether to save any error backtrace in the retry payload to display in web UI,
|
||||
# queue - use a named queue for this Worker, default 'default'
|
||||
# retry - enable the RetryJobs middleware for this Worker, *true* to use the default
|
||||
# or *Integer* count
|
||||
# backtrace - whether to save any error backtrace in the retry payload to display in web UI,
|
||||
# can be true, false or an integer number of lines to save, default *false*
|
||||
# :pool - use the given Redis connection pool to push this type of job to a given shard.
|
||||
# pool - use the given Redis connection pool to push this type of job to a given shard.
|
||||
#
|
||||
# In practice, any option is allowed. This is the main mechanism to configure the
|
||||
# options for a specific job.
|
||||
def sidekiq_options(opts={})
|
||||
self.sidekiq_options_hash = get_sidekiq_options.merge(opts.stringify_keys)
|
||||
end
|
||||
|
@ -98,7 +107,13 @@ module Sidekiq
|
|||
|
||||
def client_push(item) # :nodoc:
|
||||
pool = Thread.current[:sidekiq_via_pool] || get_sidekiq_options['pool'] || Sidekiq.redis_pool
|
||||
Sidekiq::Client.new(pool).push(item.stringify_keys)
|
||||
hash = if Thread.current[:sidekiq_worker_set]
|
||||
x, Thread.current[:sidekiq_worker_set] = Thread.current[:sidekiq_worker_set], nil
|
||||
x.stringify_keys.merge(item.stringify_keys)
|
||||
else
|
||||
item.stringify_keys
|
||||
end
|
||||
Sidekiq::Client.new(pool).push(hash)
|
||||
end
|
||||
|
||||
end
|
||||
|
|
|
@ -22,5 +22,6 @@ module Myapp
|
|||
|
||||
# Do not swallow errors in after_commit/after_rollback callbacks.
|
||||
config.active_record.raise_in_transactional_callbacks = true
|
||||
config.active_job.queue_adapter = :sidekiq
|
||||
end
|
||||
end
|
||||
|
|
|
@ -17,7 +17,6 @@ Gem::Specification.new do |gem|
|
|||
gem.version = Sidekiq::VERSION
|
||||
gem.add_dependency 'redis', '~> 3.2', '>= 3.2.1'
|
||||
gem.add_dependency 'connection_pool', '~> 2.2', '>= 2.2.0'
|
||||
gem.add_dependency 'json', '~> 1.0'
|
||||
gem.add_dependency 'concurrent-ruby', '~> 1.0'
|
||||
gem.add_development_dependency 'redis-namespace', '~> 1.5', '>= 1.5.2'
|
||||
gem.add_development_dependency 'sinatra', '~> 1.4', '>= 1.4.6'
|
||||
|
|
|
@ -310,18 +310,88 @@ class TestCli < Sidekiq::Test
|
|||
end
|
||||
|
||||
describe 'misc' do
|
||||
it 'handles interrupts' do
|
||||
cli = Sidekiq::CLI.new
|
||||
assert_raises Interrupt do
|
||||
cli.handle_signal('INT')
|
||||
end
|
||||
assert_raises Interrupt do
|
||||
cli.handle_signal('TERM')
|
||||
end
|
||||
cli.handle_signal('USR2')
|
||||
cli.handle_signal('TTIN')
|
||||
before do
|
||||
@cli = Sidekiq::CLI.new
|
||||
end
|
||||
|
||||
it 'handles interrupts' do
|
||||
assert_raises Interrupt do
|
||||
@cli.handle_signal('INT')
|
||||
end
|
||||
assert_raises Interrupt do
|
||||
@cli.handle_signal('TERM')
|
||||
end
|
||||
end
|
||||
|
||||
describe 'handles USR1 and USR2' do
|
||||
before do
|
||||
@tmp_log_path = '/tmp/sidekiq.log'
|
||||
@cli.parse(['sidekiq', '-L', @tmp_log_path, '-r', './test/fake_env.rb'])
|
||||
end
|
||||
|
||||
after do
|
||||
File.unlink @tmp_log_path if File.exists? @tmp_log_path
|
||||
end
|
||||
|
||||
it 'shuts down the worker' do
|
||||
count = 0
|
||||
Sidekiq.options[:lifecycle_events][:quiet] = [proc {
|
||||
count += 1
|
||||
}]
|
||||
@cli.launcher = Sidekiq::Launcher.new(Sidekiq.options)
|
||||
@cli.handle_signal('USR1')
|
||||
|
||||
assert_equal 1, count
|
||||
end
|
||||
|
||||
it 'reopens logs' do
|
||||
mock = MiniTest::Mock.new
|
||||
# reopen_logs returns number of files reopened so mock that
|
||||
mock.expect(:call, 1)
|
||||
|
||||
Sidekiq::Logging.stub(:reopen_logs, mock) do
|
||||
@cli.handle_signal('USR2')
|
||||
end
|
||||
mock.verify
|
||||
end
|
||||
end
|
||||
|
||||
describe 'handles TTIN' do
|
||||
before do
|
||||
@tmp_log_path = '/tmp/sidekiq.log'
|
||||
@cli.parse(['sidekiq', '-L', @tmp_log_path, '-r', './test/fake_env.rb'])
|
||||
@mock_thread = MiniTest::Mock.new
|
||||
@mock_thread.expect(:[], 'interrupt_test', ['label'])
|
||||
end
|
||||
|
||||
after do
|
||||
File.unlink @tmp_log_path if File.exists? @tmp_log_path
|
||||
end
|
||||
|
||||
describe 'with backtrace' do
|
||||
it 'logs backtrace' do
|
||||
2.times { @mock_thread.expect(:backtrace, ['something went wrong']) }
|
||||
|
||||
Thread.stub(:list, [@mock_thread]) do
|
||||
@cli.handle_signal('TTIN')
|
||||
assert_match(/something went wrong/, File.read(@tmp_log_path), "didn't include the log message")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
describe 'without backtrace' do
|
||||
it 'logs no backtrace available' do
|
||||
@mock_thread.expect(:backtrace, nil)
|
||||
|
||||
Thread.stub(:list, [@mock_thread]) do
|
||||
@cli.handle_signal('TTIN')
|
||||
assert_match(/no backtrace available/, File.read(@tmp_log_path), "didn't include the log message")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
|
||||
it 'can fire events' do
|
||||
count = 0
|
||||
Sidekiq.options[:lifecycle_events][:startup] = [proc {
|
||||
|
|
|
@ -33,6 +33,23 @@ class TestClient < Sidekiq::Test
|
|||
assert_equal 24, jid.size
|
||||
end
|
||||
|
||||
it 'allows middleware to stop bulk jobs' do
|
||||
mware = Class.new do
|
||||
def call(worker_klass,msg,q,r)
|
||||
msg['args'][0] == 1 ? yield : false
|
||||
end
|
||||
end
|
||||
client = Sidekiq::Client.new
|
||||
client.middleware do |chain|
|
||||
chain.add mware
|
||||
end
|
||||
q = Sidekiq::Queue.new
|
||||
q.clear
|
||||
result = client.push_bulk('class' => 'Blah', 'args' => [[1],[2],[3]])
|
||||
assert_equal 1, result.size
|
||||
assert_equal 1, q.size
|
||||
end
|
||||
|
||||
it 'allows local middleware modification' do
|
||||
$called = false
|
||||
mware = Class.new { def call(worker_klass,msg,q,r); $called = true; msg;end }
|
||||
|
@ -165,6 +182,18 @@ class TestClient < Sidekiq::Test
|
|||
conn.verify
|
||||
end
|
||||
|
||||
it 'allows #via to point to same Redi' do
|
||||
conn = MiniTest::Mock.new
|
||||
conn.expect(:multi, [0, 1])
|
||||
sharded_pool = ConnectionPool.new(size: 1) { conn }
|
||||
Sidekiq::Client.via(sharded_pool) do
|
||||
Sidekiq::Client.via(sharded_pool) do
|
||||
CWorker.perform_async(1,2,3)
|
||||
end
|
||||
end
|
||||
conn.verify
|
||||
end
|
||||
|
||||
it 'allows #via to point to different Redi' do
|
||||
conn = MiniTest::Mock.new
|
||||
conn.expect(:multi, [0, 1])
|
||||
|
@ -192,4 +221,42 @@ class TestClient < Sidekiq::Test
|
|||
conn.verify
|
||||
end
|
||||
end
|
||||
|
||||
describe 'Sidekiq::Worker#set' do
|
||||
class SetWorker
|
||||
include Sidekiq::Worker
|
||||
sidekiq_options :queue => :foo, 'retry' => 12
|
||||
end
|
||||
|
||||
def setup
|
||||
Sidekiq.redis {|c| c.flushdb }
|
||||
end
|
||||
|
||||
it 'allows option overrides' do
|
||||
q = Sidekiq::Queue.new('bar')
|
||||
assert_equal 0, q.size
|
||||
assert SetWorker.set(queue: :bar).perform_async(1)
|
||||
job = q.first
|
||||
assert_equal 'bar', job['queue']
|
||||
assert_equal 12, job['retry']
|
||||
end
|
||||
|
||||
it 'handles symbols and strings' do
|
||||
q = Sidekiq::Queue.new('bar')
|
||||
assert_equal 0, q.size
|
||||
assert SetWorker.set('queue' => 'bar', :retry => 11).perform_async(1)
|
||||
job = q.first
|
||||
assert_equal 'bar', job['queue']
|
||||
assert_equal 11, job['retry']
|
||||
|
||||
q.clear
|
||||
assert SetWorker.perform_async(1)
|
||||
assert_equal 0, q.size
|
||||
|
||||
q = Sidekiq::Queue.new('foo')
|
||||
job = q.first
|
||||
assert_equal 'foo', job['queue']
|
||||
assert_equal 12, job['retry']
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
|
@ -33,7 +33,7 @@ class TestExceptionHandler < Sidekiq::Test
|
|||
Component.new.invoke_exception(:a => 1)
|
||||
@str_logger.rewind
|
||||
log = @str_logger.readlines
|
||||
assert_match(/a=>1/, log[0], "didn't include the context")
|
||||
assert_match(/"a":1/, log[0], "didn't include the context")
|
||||
assert_match(/Something didn't work!/, log[1], "didn't include the exception message")
|
||||
assert_match(/test\/test_exception_handler.rb/, log[2], "didn't include the backtrace")
|
||||
end
|
||||
|
|
149
test/test_retry_exhausted.rb
Normal file
149
test/test_retry_exhausted.rb
Normal file
|
@ -0,0 +1,149 @@
|
|||
# encoding: utf-8
|
||||
require_relative 'helper'
|
||||
require 'sidekiq/middleware/server/retry_jobs'
|
||||
|
||||
class TestRetryExhausted < Sidekiq::Test
|
||||
describe 'sidekiq_retries_exhausted' do
|
||||
class NewWorker
|
||||
include Sidekiq::Worker
|
||||
|
||||
class_attribute :exhausted_called, :exhausted_job, :exhausted_exception
|
||||
|
||||
sidekiq_retries_exhausted do |job, e|
|
||||
self.exhausted_called = true
|
||||
self.exhausted_job = job
|
||||
self.exhausted_exception = e
|
||||
end
|
||||
end
|
||||
|
||||
class OldWorker
|
||||
include Sidekiq::Worker
|
||||
|
||||
class_attribute :exhausted_called, :exhausted_job, :exhausted_exception
|
||||
|
||||
sidekiq_retries_exhausted do |job|
|
||||
self.exhausted_called = true
|
||||
self.exhausted_job = job
|
||||
end
|
||||
end
|
||||
|
||||
def cleanup
|
||||
[NewWorker, OldWorker].each do |worker_class|
|
||||
worker_class.exhausted_called = nil
|
||||
worker_class.exhausted_job = nil
|
||||
worker_class.exhausted_exception = nil
|
||||
end
|
||||
end
|
||||
|
||||
before do
|
||||
cleanup
|
||||
end
|
||||
|
||||
after do
|
||||
cleanup
|
||||
end
|
||||
|
||||
def new_worker
|
||||
@new_worker ||= NewWorker.new
|
||||
end
|
||||
|
||||
def old_worker
|
||||
@old_worker ||= OldWorker.new
|
||||
end
|
||||
|
||||
def handler(options={})
|
||||
@handler ||= Sidekiq::Middleware::Server::RetryJobs.new(options)
|
||||
end
|
||||
|
||||
def job(options={})
|
||||
@job ||= {'class' => 'Bob', 'args' => [1, 2, 'foo']}.merge(options)
|
||||
end
|
||||
|
||||
it 'does not run exhausted block when job successful on first run' do
|
||||
handler.call(new_worker, job('retry' => 2), 'default') do
|
||||
# successful
|
||||
end
|
||||
|
||||
refute NewWorker.exhausted_called?
|
||||
end
|
||||
|
||||
it 'does not run exhausted block when job successful on last retry' do
|
||||
handler.call(new_worker, job('retry_count' => 0, 'retry' => 1), 'default') do
|
||||
# successful
|
||||
end
|
||||
|
||||
refute NewWorker.exhausted_called?
|
||||
end
|
||||
|
||||
it 'does not run exhausted block when retries not exhausted yet' do
|
||||
assert_raises RuntimeError do
|
||||
handler.call(new_worker, job('retry' => 1), 'default') do
|
||||
raise 'kerblammo!'
|
||||
end
|
||||
end
|
||||
|
||||
refute NewWorker.exhausted_called?
|
||||
end
|
||||
|
||||
it 'runs exhausted block when retries exhausted' do
|
||||
assert_raises RuntimeError do
|
||||
handler.call(new_worker, job('retry_count' => 0, 'retry' => 1), 'default') do
|
||||
raise 'kerblammo!'
|
||||
end
|
||||
end
|
||||
|
||||
assert NewWorker.exhausted_called?
|
||||
end
|
||||
|
||||
|
||||
it 'passes job and exception to retries exhausted block' do
|
||||
raised_error = assert_raises RuntimeError do
|
||||
handler.call(new_worker, job('retry_count' => 0, 'retry' => 1), 'default') do
|
||||
raise 'kerblammo!'
|
||||
end
|
||||
end
|
||||
|
||||
assert new_worker.exhausted_called?
|
||||
assert_equal raised_error.message, new_worker.exhausted_job['error_message']
|
||||
assert_equal raised_error, new_worker.exhausted_exception
|
||||
end
|
||||
|
||||
it 'passes job to retries exhausted block' do
|
||||
raised_error = assert_raises RuntimeError do
|
||||
handler.call(old_worker, job('retry_count' => 0, 'retry' => 1), 'default') do
|
||||
raise 'kerblammo!'
|
||||
end
|
||||
end
|
||||
|
||||
assert old_worker.exhausted_called?
|
||||
assert_equal raised_error.message, old_worker.exhausted_job['error_message']
|
||||
assert_equal nil, new_worker.exhausted_exception
|
||||
end
|
||||
|
||||
it 'allows a global default handler' do
|
||||
begin
|
||||
class Foobar
|
||||
include Sidekiq::Worker
|
||||
end
|
||||
|
||||
exhausted_job = nil
|
||||
exhausted_exception = nil
|
||||
Sidekiq.default_retries_exhausted = lambda do |job, ex|
|
||||
exhausted_job = job
|
||||
exhausted_exception = ex
|
||||
end
|
||||
f = Foobar.new
|
||||
raised_error = assert_raises RuntimeError do
|
||||
handler.call(f, job('retry_count' => 0, 'retry' => 1), 'default') do
|
||||
raise 'kerblammo!'
|
||||
end
|
||||
end
|
||||
|
||||
assert exhausted_job
|
||||
assert_equal raised_error, exhausted_exception
|
||||
ensure
|
||||
Sidekiq.default_retries_exhausted = nil
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -91,7 +91,7 @@ class TestScheduled < Sidekiq::Test
|
|||
it 'generates random intervals that target a configured average' do
|
||||
with_sidekiq_option(:poll_interval_average, 10) do
|
||||
i = 500
|
||||
intervals = i.times.map{ @poller.send(:random_poll_interval) }
|
||||
intervals = Array.new(i){ @poller.send(:random_poll_interval) }
|
||||
|
||||
assert intervals.all?{|x| x >= 5}
|
||||
assert intervals.all?{|x| x <= 15}
|
||||
|
|
|
@ -97,4 +97,11 @@ class TestSidekiq < Sidekiq::Test
|
|||
assert_equal counts[0] + 1, counts[1]
|
||||
end
|
||||
end
|
||||
|
||||
describe 'redis info' do
|
||||
it 'calls the INFO command which returns at least redis_version' do
|
||||
output = Sidekiq.redis_info
|
||||
assert_includes output.keys, "redis_version"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
|
@ -262,6 +262,16 @@ class TestTesting < Sidekiq::Test
|
|||
assert_equal 1, SecondWorker.count
|
||||
end
|
||||
|
||||
it 'drains jobs of workers with symbolized queue names' do
|
||||
Sidekiq::Worker.jobs.clear
|
||||
|
||||
AltQueueWorker.perform_async(5,6)
|
||||
assert_equal 1, AltQueueWorker.jobs.size
|
||||
|
||||
Sidekiq::Worker.drain_all
|
||||
assert_equal 0, AltQueueWorker.jobs.size
|
||||
end
|
||||
|
||||
it 'can execute a job' do
|
||||
DirectWorker.execute_job(DirectWorker.new, [2, 3])
|
||||
end
|
||||
|
|
|
@ -87,7 +87,7 @@ class TestInline < Sidekiq::Test
|
|||
end
|
||||
|
||||
it 'should relay parameters through json' do
|
||||
assert Sidekiq::Client.enqueue(InlineWorkerWithTimeParam, Time.now)
|
||||
assert Sidekiq::Client.enqueue(InlineWorkerWithTimeParam, Time.now.to_f)
|
||||
end
|
||||
|
||||
end
|
||||
|
|
|
@ -7,11 +7,7 @@ class TestUtil < Sidekiq::Test
|
|||
include Sidekiq::Util
|
||||
end
|
||||
|
||||
def test_hertz_donut
|
||||
obj = Helpers.new
|
||||
output = capture_logging(Logger::DEBUG) do
|
||||
assert_equal false, obj.want_a_hertz_donut?
|
||||
end
|
||||
assert_includes output, "hz: 10"
|
||||
def test_nothing_atm
|
||||
assert true
|
||||
end
|
||||
end
|
||||
|
|
|
@ -370,19 +370,24 @@ class TestWeb < Sidekiq::Test
|
|||
assert_equal 200, last_response.status
|
||||
end
|
||||
|
||||
Sidekiq::Web.settings.locales << File.join(File.dirname(__FILE__), "fixtures")
|
||||
it 'can show user defined tab with custom locales' do
|
||||
begin
|
||||
describe 'custom locales' do
|
||||
before do
|
||||
Sidekiq::Web.settings.locales << File.join(File.dirname(__FILE__), "fixtures")
|
||||
Sidekiq::Web.tabs['Custom Tab'] = '/custom'
|
||||
Sidekiq::Web.get('/custom') do
|
||||
clear_caches # ugly hack since I can't figure out how to access WebHelpers outside of this context
|
||||
t('translated_text')
|
||||
end
|
||||
end
|
||||
|
||||
after do
|
||||
Sidekiq::Web.tabs.delete 'Custom Tab'
|
||||
Sidekiq::Web.settings.locales.pop
|
||||
end
|
||||
|
||||
it 'can show user defined tab with custom locales' do
|
||||
get '/custom'
|
||||
assert_match(/Changed text/, last_response.body)
|
||||
|
||||
ensure
|
||||
Sidekiq::Web.tabs.delete 'Custom Tab'
|
||||
end
|
||||
end
|
||||
|
||||
|
|
|
@ -33,6 +33,7 @@ cs:
|
|||
NextRetry: Další opakování
|
||||
RetryCount: Počet opakování
|
||||
RetryNow: Opakovat teď
|
||||
Kill: Zabít
|
||||
LastRetry: Poslední opakování
|
||||
OriginallyFailed: Původně se nezdařilo
|
||||
AreYouSure: Jste si jisti?
|
||||
|
@ -61,8 +62,17 @@ cs:
|
|||
Failures: Selhání
|
||||
DeadJobs: Mrtvé úkoly
|
||||
NoDeadJobsFound: Nebyly nalezeny žádné mrtvé úkoly
|
||||
Dead: Mrtvý
|
||||
Dead: Mrtvé
|
||||
Processes: Procesy
|
||||
Thread: Vlákno
|
||||
Threads: Vlákna
|
||||
Jobs: Úkoly
|
||||
Paused: Pozastavené
|
||||
Stop: Zastavit
|
||||
Quiet: Ztišit
|
||||
StopAll: Zastavit vše
|
||||
QuietAll: Ztišit vše
|
||||
PollingInterval: Interval obnovení
|
||||
Plugins: Doplňky
|
||||
NotYetEnqueued: Ještě nezařazeno
|
||||
CreatedAt: Vytvořeno
|
||||
|
|
|
@ -6,12 +6,12 @@ fr:
|
|||
Namespace: Namespace
|
||||
Realtime: Temps réel
|
||||
History: Historique
|
||||
Busy: Occupées
|
||||
Busy: En cours
|
||||
Processed: Traitées
|
||||
Failed: Échouées
|
||||
Scheduled: Planifiée
|
||||
Scheduled: Planifiées
|
||||
Retries: Tentatives
|
||||
Enqueued: En queue
|
||||
Enqueued: En attente
|
||||
Worker: Travailleur
|
||||
LivePoll: Temps réel
|
||||
StopPolling: Arrêt du temps réel
|
||||
|
@ -20,8 +20,8 @@ fr:
|
|||
Job: Tâche
|
||||
Arguments: Arguments
|
||||
Extras: Extras
|
||||
Started: Démarrées
|
||||
ShowAll: Montrer tout
|
||||
Started: Démarrée
|
||||
ShowAll: Tout montrer
|
||||
CurrentMessagesInQueue: Messages actuellement dans <span class='title'>%{queue}</span>
|
||||
Delete: Supprimer
|
||||
AddToQueue: Ajouter à la queue
|
||||
|
@ -35,7 +35,7 @@ fr:
|
|||
RetryNow: Réessayer maintenant
|
||||
Kill: Tuer
|
||||
LastRetry: Dernier essai
|
||||
OriginallyFailed: Échec originel
|
||||
OriginallyFailed: Échec initial
|
||||
AreYouSure: Êtes-vous certain ?
|
||||
DeleteAll: Tout supprimer
|
||||
RetryAll: Tout réessayer
|
||||
|
@ -45,11 +45,11 @@ fr:
|
|||
ErrorMessage: Message d’erreur
|
||||
ErrorBacktrace: Backtrace d’erreur
|
||||
GoBack: ← Retour
|
||||
NoScheduledFound: Pas de tâches planifiées trouvées
|
||||
NoScheduledFound: Aucune tâche planifiée n'a été trouvée
|
||||
When: Quand
|
||||
ScheduledJobs: Tâches planifiées
|
||||
idle: en attente
|
||||
active: actives
|
||||
idle: inactif
|
||||
active: actif
|
||||
Version: Version
|
||||
Connections: Connexions
|
||||
MemoryUsage: Mémoire utilisée
|
||||
|
@ -62,8 +62,17 @@ fr:
|
|||
Failures: Echecs
|
||||
DeadJobs: Tâches mortes
|
||||
NoDeadJobsFound: Aucune tâche morte n'a été trouvée
|
||||
Dead: Morte
|
||||
Dead: Mortes
|
||||
Processes: Processus
|
||||
Thread: Thread
|
||||
Threads: Threads
|
||||
Jobs: Tâches
|
||||
Paused: Mise en pause
|
||||
Stop: Arrêter
|
||||
Quiet: Clôturer
|
||||
StopAll: Tout arrêter
|
||||
QuietAll: Tout clôturer
|
||||
PollingInterval: Interval de rafraîchissement
|
||||
Plugins: Plugins
|
||||
NotYetEnqueued: Pas encore en file d'attente
|
||||
CreatedAt: Créée le
|
||||
|
|
|
@ -6,8 +6,8 @@ ja:
|
|||
Namespace: ネームスペース
|
||||
Realtime: リアルタイム
|
||||
History: 履歴
|
||||
Busy: ビジー
|
||||
Processed: 処理完了
|
||||
Busy: 実行中
|
||||
Processed: 完了
|
||||
Failed: 失敗
|
||||
Scheduled: 予定
|
||||
Retries: 再試行
|
||||
|
@ -35,25 +35,25 @@ ja:
|
|||
RetryNow: 今すぐ再試行
|
||||
LastRetry: 再試行履歴
|
||||
OriginallyFailed: 失敗
|
||||
AreYouSure: いいですか?
|
||||
AreYouSure: よろしいですか?
|
||||
DeleteAll: 全て削除
|
||||
RetryAll: 全て再試行
|
||||
NoRetriesFound: 再試行できません
|
||||
NoRetriesFound: 再試行するジョブはありません
|
||||
Error: エラー
|
||||
ErrorClass: クラスエラー
|
||||
ErrorClass: エラークラス
|
||||
ErrorMessage: エラーメッセージ
|
||||
ErrorBacktrace: エラーバックトレース
|
||||
GoBack: ← 戻る
|
||||
NoScheduledFound: 予定したジョブはありません
|
||||
NoScheduledFound: 予定されたジョブはありません
|
||||
When: いつ
|
||||
ScheduledJobs: 予定したジョブ
|
||||
ScheduledJobs: 予定されたジョブ
|
||||
idle: アイドル
|
||||
active: アクティブ
|
||||
Version: バージョン
|
||||
Connections: 接続
|
||||
MemoryUsage: メモリー容量
|
||||
PeakMemoryUsage: 最大メモリー容量
|
||||
Uptime: Uptime (days)
|
||||
MemoryUsage: メモリー使用量
|
||||
PeakMemoryUsage: 最大メモリー使用量
|
||||
Uptime: 連続稼働時間 (日)
|
||||
OneWeek: 1 週
|
||||
OneMonth: 1 ヶ月
|
||||
ThreeMonths: 3 ヶ月
|
||||
|
@ -67,3 +67,12 @@ ja:
|
|||
Thread: スレッド
|
||||
Threads: スレッド
|
||||
Jobs: ジョブ
|
||||
Paused: 一時停止中
|
||||
Stop: 停止
|
||||
Quiet: 処理終了
|
||||
StopAll: すべて停止
|
||||
QuietAll: すべて処理終了
|
||||
PollingInterval: ポーリング間隔
|
||||
Plugins: プラグイン
|
||||
NotYetEnqueued: キューに入っていません
|
||||
CreatedAt: 作成日時
|
||||
|
|
|
@ -6,8 +6,8 @@
|
|||
<form method="POST" style="margin-top: 20px; margin-bottom: 10px;">
|
||||
<%= csrf_tag %>
|
||||
<div class="btn-group pull-right">
|
||||
<button class="btn btn-warn" type="submit" name="quiet" value="1"><%= t('QuietAll') %></button>
|
||||
<button class="btn btn-danger" type="submit" name="stop" value="1"><%= t('StopAll') %></button>
|
||||
<button class="btn btn-warn" type="submit" name="quiet" value="1" data-confirm="<%= t('AreYouSure') %>"><%= t('QuietAll') %></button>
|
||||
<button class="btn btn-danger" type="submit" name="stop" value="1" data-confirm="<%= t('AreYouSure') %>"><%= t('StopAll') %></button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
@ -30,6 +30,9 @@
|
|||
<% process.labels.each do |label| %>
|
||||
<span class="label label-info"><%= label %></span>
|
||||
<% end %>
|
||||
<% if process.stopping? %>
|
||||
<span class="label label-danger">Quiet</span>
|
||||
<% end %>
|
||||
<br>
|
||||
<b><%= "#{t('Queues')}: " %></b>
|
||||
<%= process['queues'] * ", " %>
|
||||
|
@ -42,7 +45,9 @@
|
|||
<form method="POST">
|
||||
<%= csrf_tag %>
|
||||
<input type="hidden" name="identity" value="<%= process['identity'] %>"/>
|
||||
<button class="btn btn-warn" type="submit" name="quiet" value="1"><%= t('Quiet') %></button>
|
||||
<% unless process.stopping? %>
|
||||
<button class="btn btn-warn" type="submit" name="quiet" value="1"><%= t('Quiet') %></button>
|
||||
<% end %>
|
||||
<button class="btn btn-danger" type="submit" name="stop" value="1"><%= t('Stop') %></button>
|
||||
</form>
|
||||
</div>
|
||||
|
|
|
@ -7,7 +7,7 @@
|
|||
<tr>
|
||||
<th><%= t('ErrorClass') %></th>
|
||||
<td>
|
||||
<code><%= @retry['error_class'] %></code>
|
||||
<code><%= h @retry.display_class %></code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
|
Loading…
Add table
Reference in a new issue