Sidekiq 101

The latest stable versions as May 26, 2022:

• Sidekiq 6.5.7
• Sidekiq Pro 5.5.5
• Sidekiq Enterprise 2.5.3

The creator, Mike Perham, has prolific blog: https://www.mikeperham.com/.

In particular you should read this post: A Tour of the Sidekiq API.

Sidekiq Documentation and the Discussions for questions and ideas.

Sidekiq Concepts

bin/sidekiq --config ./config/sidekiq.yml --concurrency 1 starts a Sidekiq Process.

Sidekiq Job is a unit of work.
Perform a Sidekiq Job pushes a job to Sidekiq Queue.
Sidekiq Queue is a list of jobs ready to execute now.
Sidekiq Process uses Ruby Threads that to execute jobs from Sidekiq Queues.

Sidekiq Client is a process pushes Jobs to Redis.
Sidekiq Server is a process that boots Rails. Pulls Jobs from Redis to perform with arguments.

Sidekiq arguments is a Ruby Hash that serializes to JSON. So they must be simple JSON datatypes (integer, float, string, boolean, array, and hash).

Sidekiq Concurrency

Usable concurrency is limited by Ruby's GVL meaning increase concurrency will increase CPU.

Next are some common tasks in Sidekiq.

Create a Job

class SomeJob
  include Sidekiq::Job
end

Perform a Job

Immediately

SomeJob.perform_inline

Asynchronously

SomeJob.perform_async

Bulk Perform a Job

SomeJob.perform_bulk [[args_for_somejob], [args_for_somejob], [args_for_somejob]]

Find a Queue

Sidekiq::Queue.new("queuename")

Delete Jobs from Sidekiq Retries

retry_set = Sidekiq::RetrySet.new

worker_names = [
  "YourJobClassA",
  "YourJobClassB",
]

worker_names.each do |name|
  count = retry_set.scan(name).count { |job| job.display_class == name }
  puts "Deleting #{count} jobs of #{name} from Sidekiq Retries."
  retry_set.scan(name).select { |job| job.display_class == name }.map(&:delete)
  puts "Done."
end

Based on A Tour of the Sidekiq API.

Migrate Job from Queue to Another Queue

# Migrate ProcessPaymentsWorker from default to priority queue in RetrySet.
#
job_names = {
  "ProcessPaymentsWorker" => ProcessPaymentsWorker
]
from_queue = "default"
to_queue = "priority"

queue = Sidekiq::RetrySet.new
job_names.each do |name, job_class|
  queue.scan(name).each do |job|
    if job.display_class == name && job.queue == from_queue
      job_class.set(queue: to_queue).perform_async(*job.args)
      job.delete
    end
  end
end

queue.scan(name) is important here to avoid CPU spike.

Sidekiq Retry

Sidekiq’s default Retry formula:

(count**4) + 15 + jitter

and you can custom this retry using sidekiq_retry_in block:

class SidekiqWorker
  include Sidekiq::Job
  sidekiq_options retry: 5 # default is 25

  sidekiq_retry_in do |count, exception|
    case exception
    when SpecialException
      10 * (count + 1) # (i.e. 10, 20, 30, 40, 50)
    end
  end

  def perform
    ...
  end
end

Enqueue job every minute

Requires Sidekiq Enterprise Unique Jobs feature.

class MinuteWorker
  include Sidekiq::Job

  sidekiq_options queue: "critical", retry: false
  sidekiq_options unique_for: 59.seconds, unique_until: :start
end

# config/initializers/sidekiq.rb
config.periodic do |periodic|
  periodic.register "* * * * *", "MinuteWorker"
end

Fun

Sidekiq’s heartbeat implementation: ❤

Sidekiq.❨╯°□°❩╯︵┻━┻