1
0
Fork 0
mirror of https://github.com/mperham/sidekiq.git synced 2022-11-09 13:52:34 -05:00

merge master

This commit is contained in:
Mike Perham 2016-06-15 16:48:40 -07:00
commit a67c9b1d90
70 changed files with 166 additions and 36 deletions

View file

@ -25,6 +25,9 @@ gem 'redis-namespace'
`concurrency + 2` connections in your pool or Sidekiq will exit.
When in doubt, let Sidekiq size the connection pool for you.
* Worker data is no longer updated in real-time but rather upon every
heartbeat. Don't expect the `Sidekiq::Workers` API to be millisecond-precise.
* There's a new testing API based off the `Sidekiq::Queues` namespace. All
assertions made against the Worker class still work as expected.
```ruby

View file

@ -1,8 +1,37 @@
# Sidekiq Changes
HEAD
-----------
- Fixed race condition in heartbeat which could rarely lead to lingering
processes on the Busy tab. [#2982]
```ruby
# to clean up lingering processes, modify this as necessary to connect to your Redis.
# after 60 seconds, lingering processes should disappear from the Busy page.
require 'redis'
r = Redis.new(url: "redis://localhost:6379/0")
# uncomment if you need a namespace
#require 'redis-namespace'
#r = Redis::Namespace.new("foo", r)
r.smembers("processes").each do |pro|
r.expire(pro, 60)
r.expire("#{pro}:workers", 60)
end
```
4.1.2
-----------
- Fix Redis data leak with worker data when a busy Sidekiq process
crashes. You can find and expire leaked data in Redis with this
script:
```bash
$ redis-cli keys "*:workers" | while read LINE ; do TTL=`redis-cli expire "$LINE" 60`; echo "$LINE"; done;
```
Please note that `keys` can be dangerous to run on a large, busy Redis. Caveat runner.
- Freeze all string literals with Ruby 2.3. [#2741]
- Client middleware can now stop bulk job push. [#2887]
4.1.1

View file

@ -3,6 +3,12 @@ Sidekiq Enterprise Changelog
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
HEAD
-------------
- Add API to check if a unique lock is present. See [#2932] for details.
- Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
1.2.1
-------------

View file

@ -1,9 +1,10 @@
source 'https://rubygems.org'
gemspec
gem 'rails', '5.0.0.beta2'
gem 'rack', '2.0.0.alpha'
gem 'rails', '5.0.0.rc1'
gem 'rack', '2.0.0.rc1'
gem 'sinatra', github: 'sinatra/sinatra'
gem "hiredis"
gem 'simplecov'
gem 'minitest'
gem 'minitest-utils'

View file

@ -17,17 +17,15 @@ Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
* Reliable fetch has been re-implemented due to the fetch changes in
Sidekiq 4.0.
* Support for platforms without persistent hostnames. Since reliable fetch
normally requires a persistent hostname, you may disable hostname usage on
platforms like Heroku and Docker:
* Support for platforms without persistent hostnames. Since the reliable\_fetch
algorithm requires a persistent hostname, an alternative reliability
algorithm is now available for platforms like Heroku and Docker:
```ruby
Sidekiq.configure_server do |config|
config.options[:ephemeral_hostname] = true
config.reliable_fetch!
config.timed_fetch!
end
```
This option is enabled automatically if Heroku's DYNO environment variable is present.
Without a persistent hostname, each Sidekiq process **must** have its own unique index.
The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
* The old 'sidekiq/notifications' features have been removed.

View file

@ -3,6 +3,20 @@ Sidekiq Pro Changelog
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
HEAD
---------
- Don't permanently delete batches immediately upon success [#3011]
- New `Sidekiq::PendingSet#destroy(jid)` API to remove poison pill jobs [#3015]
3.2.2
---------
- A default value for -i is only set in development now, staging or
other environments must set an index if you wish to use reliable\_fetch. [#2971]
- Fix nil dereference when checking for jobs over timeout in timed\_fetch
3.2.1
---------

View file

@ -18,7 +18,8 @@ message format as Resque so it can integrate into an existing Resque processing
You can have Sidekiq and Resque run side-by-side at the same time and
use the Resque client to enqueue jobs in Redis to be processed by Sidekiq.
Sidekiq is fast.
Performance
---------------
Version | Latency | Garbage created for 10,000 jobs | Time to process 100,000 jobs | Throughput
-----------------|------|---------|---------|------------------------

View file

@ -26,8 +26,8 @@ Toxiproxy.populate([{
Sidekiq.configure_server do |config|
config.redis = { db: 13, port: 6380 }
#config.redis = { db: 13 }
#config.options[:concurrency] = 1
config.redis = { driver: :hiredis, db: 13, port: 6380 }
config.options[:queues] << 'default'
config.logger.level = Logger::ERROR
config.average_scheduled_poll_interval = 2

View file

@ -42,6 +42,10 @@ normal exit 0 TERM
# this commented out.
reload signal USR1
# Upstart waits 5 seconds by default to kill the a process. Increase timeout to
# give sidekiq process enough time to exit.
kill timeout 15
instance $index
script

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require 'sidekiq/version'
fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.0.0." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.0.0'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require 'sidekiq'
@ -195,7 +196,7 @@ module Sidekiq
# Return all known queues within Redis.
#
def self.all
Sidekiq.redis {|c| c.smembers('queues'.freeze) }.sort.map {|q| Sidekiq::Queue.new(q) }
Sidekiq.redis { |c| c.smembers('queues'.freeze) }.sort.map { |q| Sidekiq::Queue.new(q) }
end
attr_reader :name

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
$stdout.sync = true

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'securerandom'
require 'sidekiq/middleware/chain'
@ -83,6 +84,7 @@ module Sidekiq
# than the number given if the middleware stopped processing for one or more jobs.
def push_bulk(items)
arg = items['args'].first
return [] unless arg # no jobs to push
raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !arg.is_a?(Array)
normed = normalize_item(items)

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
begin
require 'active_support/core_ext/class/attribute'
rescue LoadError

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq'
module Sidekiq
@ -5,7 +6,7 @@ module Sidekiq
class Logger
def call(ex, ctxHash)
Sidekiq.logger.warn(ctxHash) if !ctxHash.empty?
Sidekiq.logger.warn(Sidekiq.dump_json(ctxHash)) if !ctxHash.empty?
Sidekiq.logger.warn "#{ex.class.name}: #{ex.message}"
Sidekiq.logger.warn ex.backtrace.join("\n") unless ex.backtrace.nil?
end

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/extensions/generic_proxy'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/extensions/generic_proxy'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/extensions/generic_proxy'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'yaml'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require 'sidekiq/manager'
require 'sidekiq/fetch'
@ -79,7 +80,7 @@ module Sidekiq
workers_key = "#{key}:workers".freeze
nowdate = Time.now.utc.strftime("%Y-%m-%d".freeze)
Sidekiq.redis do |conn|
conn.pipelined do
conn.multi do
conn.incrby("stat:processed".freeze, procd)
conn.incrby("stat:processed:#{nowdate}", procd)
conn.incrby("stat:failed".freeze, fails)
@ -88,12 +89,13 @@ module Sidekiq
Processor::WORKER_STATE.each_pair do |tid, hash|
conn.hset(workers_key, tid, Sidekiq.dump_json(hash))
end
conn.expire(workers_key, 60)
end
end
fails = procd = 0
_, _, _, msg = Sidekiq.redis do |conn|
conn.pipelined do
conn.multi do
conn.sadd('processes', key)
conn.hmset(key, 'info', json, 'busy', Processor::WORKER_STATE.size, 'beat', Time.now.to_f, 'quiet', @done)
conn.expire(key, 60)

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'time'
require 'logger'
require 'fcntl'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require 'sidekiq/util'
require 'sidekiq/processor'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
module Sidekiq
# Middleware is code configured to run before/after
# a message is processed. It is patterned after Rack

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
#
# Simple middleware to save the current locale and restore it when the job executes.
# Use it by requiring it in your initializer:

View file

@ -8,14 +8,14 @@ module Sidekiq
# Automatically retry jobs that fail in Sidekiq.
# Sidekiq's retry support assumes a typical development lifecycle:
#
# 0. push some code changes with a bug in it
# 1. bug causes job processing to fail, sidekiq's middleware captures
# the job and pushes it onto a retry queue
# 2. sidekiq retries jobs in the retry queue multiple times with
# an exponential delay, the job continues to fail
# 3. after a few days, a developer deploys a fix. the job is
# 0. Push some code changes with a bug in it.
# 1. Bug causes job processing to fail, Sidekiq's middleware captures
# the job and pushes it onto a retry queue.
# 2. Sidekiq retries jobs in the retry queue multiple times with
# an exponential delay, the job continues to fail.
# 3. After a few days, a developer deploys a fix. The job is
# reprocessed successfully.
# 4. once retries are exhausted, sidekiq will give up and move the
# 4. Once retries are exhausted, Sidekiq will give up and move the
# job to the Dead Job Queue (aka morgue) where it must be dealt with
# manually in the Web UI.
# 5. After 6 months on the DJQ, Sidekiq will discard the job.

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
module Sidekiq
module Paginator

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/util'
require 'sidekiq/fetch'
require 'thread'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
module Sidekiq
def self.hook_rails!
return if defined?(@delay_removed)

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'connection_pool'
require 'redis'
require 'uri'
@ -7,6 +8,8 @@ module Sidekiq
class << self
def create(options={})
options = options.symbolize_keys
options[:url] ||= determine_redis_provider
size = options[:size] || (Sidekiq.server? ? (Sidekiq.options[:concurrency] + 5) : 5)
@ -32,7 +35,7 @@ module Sidekiq
# - enterprise's leader election
# - enterprise's cron support
def verify_sizing(size, concurrency)
raise ArgumentError, "Your Redis connection pool is too small for Sidekiq to work, your pool has #{size} connections but really needs to have at least #{concurrency + 2}" if size <= concurrency
raise ArgumentError, "Your Redis connection pool is too small for Sidekiq to work. Your pool has #{size} connections but really needs to have at least #{concurrency + 2}" if size <= concurrency
end
def build_client(options)
@ -44,8 +47,8 @@ module Sidekiq
require 'redis/namespace'
Redis::Namespace.new(namespace, :redis => client)
rescue LoadError
Sidekiq.logger.error("Your Redis configuration use the namespace '#{namespace}' but the redis-namespace gem not included in Gemfile." \
"Add the gem to your Gemfile in case you would like to keep using a namespace, otherwise remove the namespace parameter.")
Sidekiq.logger.error("Your Redis configuration uses the namespace '#{namespace}' but the redis-namespace gem is not included in the Gemfile." \
"Add the gem to your Gemfile to continue using a namespace. Otherwise, remove the namespace parameter.")
exit(-127)
end
else

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq'
require 'sidekiq/util'
require 'sidekiq/api'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'securerandom'
require 'sidekiq'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/testing'
##

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'socket'
require 'securerandom'
require 'sidekiq/exception_handler'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
module Sidekiq
VERSION = "4.1.1"
VERSION = "4.1.2"
end

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'erb'
require 'yaml'
require 'sinatra/base'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'uri'
module Sidekiq

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require 'sidekiq/client'
require 'sidekiq/core_ext'

View file

@ -2,8 +2,8 @@ source 'https://rubygems.org'
gem 'pry'
gem 'sidekiq', :path => '..'
gem 'rails', '5.0.0.beta4'
gem 'rack', '2.0.0.alpha'
gem 'rails', '5.0.0.rc1'
gem 'rack', '2.0.0.rc1'
gem 'sinatra', github: 'sinatra/sinatra'
platforms :ruby do

View file

@ -1,8 +1,7 @@
Sidekiq.configure_client do |config|
config.redis = { :size => 2, :namespace => 'foo' }
config.redis = { :size => 2 }
end
Sidekiq.configure_server do |config|
config.redis = { :namespace => 'foo' }
config.on(:startup) { }
config.on(:quiet) { }
config.on(:shutdown) do

View file

@ -0,0 +1 @@
# frozen_string_literal: true

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
$TESTING = true
# disable minitest/parallel threads
ENV["N"] = "0"

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/cli'
require 'sidekiq/fetch'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/api'
require 'active_job'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/cli'
require 'tempfile'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
class TestClient < Sidekiq::Test
@ -125,6 +126,10 @@ class TestClient < Sidekiq::Test
assert_match(/[0-9a-f]{12}/, jid)
end
end
it 'handles no jobs' do
result = Sidekiq::Client.push_bulk('class' => 'QueuedWorker', 'args' => [])
assert_equal 0, result.size
end
end
class BaseWorker

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/exception_handler'
require 'stringio'
@ -32,7 +33,7 @@ class TestExceptionHandler < Sidekiq::Test
Component.new.invoke_exception(:a => 1)
@str_logger.rewind
log = @str_logger.readlines
assert_match(/a=>1/, log[0], "didn't include the context")
assert_match(/"a":1/, log[0], "didn't include the context")
assert_match(/Something didn't work!/, log[1], "didn't include the exception message")
assert_match(/test\/test_exception_handler.rb/, log[2], "didn't include the backtrace")
end

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq'
require 'active_record'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/fetch'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/launcher'
@ -55,6 +56,10 @@ class TestLauncher < Sidekiq::Test
@launcher.heartbeat('identity', heartbeat_data, Sidekiq.dump_json(heartbeat_data))
end
#after do
#puts system('redis-cli -n 15 keys "*" | while read LINE ; do TTL=`redis-cli -n 15 ttl "$LINE"`; if [ "$TTL" -eq -1 ]; then echo "$LINE"; fi; done;')
#end
it 'indicates stopping status in proctitle' do
assert_equal "sidekiq #{Sidekiq::VERSION} myapp [1 of 3 busy] stopping", $0
end

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/logging'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/manager'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/middleware/chain'
require 'sidekiq/processor'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/fetch'
require 'sidekiq/cli'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
$HAS_AJ = true

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
class TestRedisConnection < Sidekiq::Test
@ -26,11 +27,16 @@ class TestRedisConnection < Sidekiq::Test
end
describe "namespace" do
it "uses a given :namespace" do
it "uses a given :namespace set by a symbol key" do
pool = Sidekiq::RedisConnection.create(:namespace => "xxx")
assert_equal "xxx", pool.checkout.namespace
end
it "uses a given :namespace set by a string key" do
pool = Sidekiq::RedisConnection.create("namespace" => "xxx")
assert_equal "xxx", pool.checkout.namespace
end
it "uses given :namespace over :namespace from Sidekiq.options" do
Sidekiq.options[:namespace] = "xxx"
pool = Sidekiq::RedisConnection.create(:namespace => "yyy")

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require_relative 'helper'
require 'sidekiq/scheduled'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/scheduled'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'sidekiq/scheduled'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require_relative 'helper'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'active_record'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'active_record'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
require 'active_record'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
class TestUtil < Sidekiq::Test

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
# encoding: utf-8
require_relative 'helper'
require 'sidekiq/web'

View file

@ -1,3 +1,4 @@
# frozen_string_literal: true
require_relative 'helper'
class TestWebHelpers < Sidekiq::Test

View file

@ -76,3 +76,4 @@ en: # <---- change this to your locale code
Plugins: Plugins
NotYetEnqueued: Not yet enqueued
CreatedAt: Created At
BackToApp: Back to App

View file

@ -73,3 +73,6 @@ ru:
QuietAll: Отдыхать всем
PollingInterval: Интервал опроса
Plugins: Плагины
NotYetEnqueued: Пока не в очереди
CreatedAt: Создан
BackToApp: Назад

View file

@ -56,7 +56,7 @@
<div class="poll-wrapper pull-right">
<%= erb :_poll_link %>
<% if Sidekiq::Web.app_url %>
<a class="btn btn-inverse" href="<%= Sidekiq::Web.app_url %>">Back to App</a>
<a class="btn btn-inverse" href="<%= Sidekiq::Web.app_url %>"><%= t('BackToApp') %></a>
<% end %>
</div>
</li>

View file

@ -6,8 +6,8 @@
<form method="POST" style="margin-top: 20px; margin-bottom: 10px;">
<%= csrf_tag %>
<div class="btn-group pull-right">
<button class="btn btn-warn" type="submit" name="quiet" value="1"><%= t('QuietAll') %></button>
<button class="btn btn-danger" type="submit" name="stop" value="1"><%= t('StopAll') %></button>
<button class="btn btn-warn" type="submit" name="quiet" value="1" data-confirm="<%= t('AreYouSure') %>"><%= t('QuietAll') %></button>
<button class="btn btn-danger" type="submit" name="stop" value="1" data-confirm="<%= t('AreYouSure') %>"><%= t('StopAll') %></button>
</div>
</form>
</div>