Enough Sidekiq

Delete Jobs from Sidekiq Retries

retry_set = Sidekiq::RetrySet.new

worker_names = [
  "YourJobClassA",
  "YourJobClassB",
]

worker_names.each do |name|
  count = retry_set.scan(name).count { |job| job.display_class == name }
  puts "Deleting #{count} jobs of #{name} from Sidekiq Retries."
  retry_set.scan(name).select { |job| job.display_class == name }.map(&:delete)
  puts "Done."
end

Based on A Tour of the Sidekiq API.

Enterprise Feautres: Limiter

Usage: Limit calls to 3rd-party API. Can use everywhere, not just Sidekiq Jobs.

Sidekiq reschedule overlimit job with linear backoff (try again in 5mins, try again in 10mins, try again in 15mins) up to 20 times (~ a day). Then mark the failed job as retry.

Sidekiq::Limiter instance has a within_limit block of work you want to apply limit.

wait_timeout — time to work for a lock
lock_timeout — time to release lock
raise — raise OverLimit if cannot get a lock within wait_timeout
ignore — ignore if cannot get a lock within wait_timeout

concurrent

Limit to 1 op at a time:

limiter = Sidekiq::Limiter.concurrent("SingleTransaction", 1, wait_timeout: 10)
limiter.within_limit do
  # work...
end

bucket

Each time interval is a bucket. Limit how many operations can do within the bucket. Reset in the next bucket.

E.g. Limit up to 600 calls each hour.

[0900-1000 bucket - Remaining: 600]
09:00 — 100 calls
09:01 — 100 calls
09:02 — 100 calls
09:03 — 100 calls
09:04 — 100 calls
09:05 — 100 calls
09:06 — Cant make calls
...
09:59 — Cant make calls

[1000-1100 bucket - Remaining: 500]

# 250 requests per minute — 15,000 per hour.
def github_api_call
  github_api_limiter = Sidekiq::Limiter.bucket("github-api-for-#{organization_id}", 250, :minute, wait_timeout: 5)
  github_api_limiter.within_limit do
    # github api call
  end
rescue Sidekiq::Limiter::OverLimit

end

Leaky bucket

v2.2 added leaky bucket

Implementation Story

Sidekiq::Limiter.leaky("key", 600, :hour)

Each time interval is a bucket. Limit how many operations can do within the bucket. After reached the limit, every second. Reset in next bucket.

E.g. Limit up to 600 calls each hour. +1 every 2 second.

[0900-1000 bucket - Remaining: 600]
09:00 — 100 calls
09:01 — 100 calls
09:02 — 100 calls
09:03 — 100 calls
09:04 — 100 calls
09:05 — 100 calls

[0900-1000 bucket - Remaining: 0]
09:06

[0900-1000 bucket - Remaining: 1]
09:07

[1000-1100 bucket - Remaining: 600]

Window

Sidekiq::Limiter.window("key", 600, :hour)

Each time interval is a sliding window. Limit how many operations can do within the sliding window. Most Strict.

E.g. Limit up to 600 calls every hour. +1 call every 2 minutes.

[0900-1000 bucket - Remaining: 400]
09:00 — 100 calls

[0901-1001 bucket - Remaining: 300]
09:01 — 100 calls

[0902-1002 bucket - Remaining: 202]
09:02 — 100 calls

[0903-1003 bucket - Remaining: 102]
09:03 — 100 calls

[0904-1004 bucket - Remaining: 43]
09:04 — 100 calls

[0905-1005 bucket - Remaining: 0]
09:05 — 43 calls

[0906-1006 bucket - Remaining: 1]
[0907-1007 bucket - Remaining: 1]
[0908-1008 bucket - Remaining: 2]
...
[1008-1108 bucket - Remaining: 600]

Performance: Leaky Bucket > Bucket > Window