The creator has prolific blog: https://www.mikeperham.com/
In particular this post is good: A Tour of the Sidekiq API.
Sidekiq Documentation and the Discussions
for questions and ideas.
Some common task in Sidekiq.
Delete Jobs from Sidekiq Retries
retry_set = Sidekiq::RetrySet.new
worker_names = [
"YourJobClassA",
"YourJobClassB",
]
worker_names.each do |name|
count = retry_set.scan(name).count { |job| job.display_class == name }
puts "Deleting #{count} jobs of #{name} from Sidekiq Retries."
retry_set.scan(name).select { |job| job.display_class == name }.map(&:delete)
puts "Done."
end
Based on A Tour of the Sidekiq API.
Sidekiq Retry
Sidekiq’s default Retry formula:
(count**4) + 15 + jitter
and you can custom this retry using sidekiq_retry_in
block:
class SidekiqWorker
include Sidekiq::Worker
sidekiq_options retry: 5 # default is 25
sidekiq_retry_in do |count, exception|
case exception
when SpecialException
10 * (count + 1) # (i.e. 10, 20, 30, 40, 50)
end
end
def perform
...
end
end
Sidekiq with Redis Cluster
Cluster is for caching and similar workloads that can scale horizontally. That is not Sidekiq.
https://github.com/mperham/sidekiq/issues/4012
Enterprise Feautres: Limiter
Usage: Limit calls to 3rd-party API. Can use everywhere, not just Sidekiq Jobs.
Sidekiq reschedule overlimit job with linear backoff (try again in 5mins, try again in 10mins, try again in 15mins) up to 20 times (~ a day). Then mark the failed job as retry.
Sidekiq::Limiter
instance has a within_limit
block of work you want to apply limit.
-
wait_timeout
— time to work for a lock -
lock_timeout
— time to release lock -
raise
— raiseOverLimit
if cannot get a lock withinwait_timeout
-
ignore
— ignore if cannot get a lock withinwait_timeout
Note limiter is not throttling. For throttling, use Sidekiq::Throttled. With Sidekiq::Throttled, you also get limiter feature.
concurrent
Limit to 1 op at a time:
limiter = Sidekiq::Limiter.concurrent("SingleTransaction", 1, wait_timeout: 10)
limiter.within_limit do
# work...
end
bucket
Each time interval is a bucket. Limit how many operations can do within the bucket. Reset in the next bucket.
E.g. Limit up to 600 calls each hour.
[0900-1000 bucket - Remaining: 600]
09:00 — 100 calls
09:01 — 100 calls
09:02 — 100 calls
09:03 — 100 calls
09:04 — 100 calls
09:05 — 100 calls
09:06 — Cant make calls
...
09:59 — Cant make calls
[1000-1100 bucket - Remaining: 500]
# 250 requests per minute — 15,000 per hour.
def github_api_call
github_api_limiter = Sidekiq::Limiter.bucket("github-api-for-#{organization_id}", 250, :minute, wait_timeout: 5)
github_api_limiter.within_limit do
# github api call
end
rescue Sidekiq::Limiter::OverLimit
end
Leaky bucket
v2.2 added leaky bucket
Sidekiq::Limiter.leaky("key", 600, :hour)
Each time interval is a bucket. Limit how many operations can do within the bucket. After reached the limit, every second. Reset in next bucket.
E.g. Limit up to 600 calls each hour. +1 every 2 second.
[0900-1000 bucket - Remaining: 600]
09:00 — 100 calls
09:01 — 100 calls
09:02 — 100 calls
09:03 — 100 calls
09:04 — 100 calls
09:05 — 100 calls
[0900-1000 bucket - Remaining: 0]
09:06
[0900-1000 bucket - Remaining: 1]
09:07
[1000-1100 bucket - Remaining: 600]
Window
Sidekiq::Limiter.window("key", 600, :hour)
Each time interval is a sliding window. Limit how many operations can do within the sliding window. Most Strict.
E.g. Limit up to 600 calls every hour. +1 call every 2 minutes.
[0900-1000 bucket - Remaining: 400]
09:00 — 100 calls
[0901-1001 bucket - Remaining: 300]
09:01 — 100 calls
[0902-1002 bucket - Remaining: 202]
09:02 — 100 calls
[0903-1003 bucket - Remaining: 102]
09:03 — 100 calls
[0904-1004 bucket - Remaining: 43]
09:04 — 100 calls
[0905-1005 bucket - Remaining: 0]
09:05 — 43 calls
[0906-1006 bucket - Remaining: 1]
[0907-1007 bucket - Remaining: 1]
[0908-1008 bucket - Remaining: 2]
...
[1008-1108 bucket - Remaining: 600]
Performance: Leaky Bucket > Bucket > Window
Enqueue job every minute
Requires Sidekiq Enterprise Unique Jobs feature.
class MinuteWorker
include Sidekiq::Job
sidekiq_options queue: "critical", retry: false
sidekiq_options unique_for: 1.minute, until: :start
end
# config/initializers/sidekiq.rb
config.periodic do |periodic|
periodic.register "* * * * *", "MinuteWorker"
end