1
0
Fork 0
mirror of https://github.com/mperham/sidekiq.git synced 2022-11-09 13:52:34 -05:00
mperham--sidekiq/lib/sidekiq/job_util.rb

72 lines
3 KiB
Ruby
Raw Normal View History

require "securerandom"
require "time"
module Sidekiq
module JobUtil
# These functions encapsulate various job utilities.
2022-05-06 16:52:38 -04:00
TRANSIENT_ATTRIBUTES = %w[]
def validate(item)
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
end
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
def verify_json(item)
job_class = item["wrapped"] || item["class"]
if Sidekiq[:on_complex_arguments] == :raise
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
msg = <<~EOM
Job arguments to #{job_class} must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices.
2022-01-05 22:54:12 -05:00
To disable this error, remove `Sidekiq.strict_args!` from your initializer.
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
EOM
raise(ArgumentError, msg) unless json_safe?(item)
elsif Sidekiq[:on_complex_arguments] == :warn
Sidekiq.logger.warn <<~EOM unless json_safe?(item)
Job arguments to #{job_class} do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
2022-01-05 22:54:12 -05:00
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
EOM
end
end
def normalize_item(item)
validate(item)
# merge in the default sidekiq_options for the item's class and/or wrapped element
# this allows ActiveJobs to control sidekiq_options too.
defaults = normalized_hash(item["class"])
2022-01-06 00:01:32 -05:00
defaults = defaults.merge(item["wrapped"].get_sidekiq_options) if item["wrapped"].respond_to?(:get_sidekiq_options)
item = defaults.merge(item)
raise(ArgumentError, "Job must include a valid queue name") if item["queue"].nil? || item["queue"] == ""
2022-05-06 16:52:38 -04:00
# remove job attributes which aren't necessary to persist into Redis
TRANSIENT_ATTRIBUTES.each { |key| item.delete(key) }
item["jid"] ||= SecureRandom.hex(12)
item["class"] = item["class"].to_s
item["queue"] = item["queue"].to_s
item["created_at"] ||= Time.now.to_f
item
end
def normalized_hash(item_class)
if item_class.is_a?(Class)
raise(ArgumentError, "Message must include a Sidekiq::Job class, not class name: #{item_class.ancestors.inspect}") unless item_class.respond_to?(:get_sidekiq_options)
item_class.get_sidekiq_options
else
Sidekiq.default_job_options
end
end
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
private
def json_safe?(item)
JSON.parse(JSON.dump(item["args"])) == item["args"]
Implement strict argument checking (#5071) * Add the outline of failing tests * Implement argument checking * Move argument checking into Sidekiq::JobUtil#validate * Refactor acceptable class definition into a constant to cut down on array allocations * Improve error message, match raise call formatting to other errors in the class * Address feedback in the Pull Request to use the JSON round-trip method of confirming the safety of job argument payloads. Cleanup commented-out code from a few years back. Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Swap out JSON.load for JSON.parse per the security CI check Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a few more tests cases to build up confidence around our JSON.parse/dump approach and deep structures Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Improve test case description * Warn when job arguments do not serialize safely and point folks toward how to enable strict_mode and the best practice * Reconfigure the options-hash based approach to a global Sidekiq.strict_mode! method * Add a note in the raised error on how to disable the error * Let the error message breathe a little bit * Toggle strict_mode! off to suss out a test flake * Capitalize the start of a sentence * Rename job_is_json_safe to json_safe? Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Refactor a few tests to test a single argument at a time Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Break out test cases to exercise each individual intersting case instead of all at once Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Change formatting to be more consistent, tighter when arguments are simple Co-authored-by: Eda Zhou <eda.zhou@gusto.com> * Add a flag to disable the warning message for development warning messages Co-authored-by: Eda Zhou <eda.zhou@gusto.com> Co-authored-by: Eda Zhou <eda.zhou@gusto.com>
2021-12-07 16:20:20 -05:00
end
end
end