Heroku, Puma, Redis, Sidekiq and Connection Limits

Updated 2015-08-07: Sidekiq client processes can use connection pooling rather than requiring one connection per thread.

Updated 2015-08-08: Added an additional considerations section for other things to look out for.

When deploying a Rails app with Sidekiq to Heroku, it can be confusing to figure out how many of your limited connections to Redis and the database you have available when setting your connection pool sizes in your configs.

The free tiers of the various Redis providers on Heroku, which are generally large enough to handle a job queue at first, have limits of 10-30 connections, before you start getting connection refused errors. How do we calculate our optimal connection pool size with web dynos, clustered puma threads, and job/worker dynos?

The Client Math

Sidekiq.configure_client do |config|  
  config.redis = { size: 1, url: ENV["REDIS_URL"], namespace: "your-app" }
end  

Updated:

Sidekiq uses a connection pool by default per process, so Puma's threads can share a limited number of connections for each web dyno.

It's unlikely your app will need a connection per thread, since the client's only job is to push to Redis, which is very fast. We can then share a reasonable amount of connections between all the Puma threads to minimize blocking and idle connections.

What is reasonable? Depends on your application, but as a rule of thumb, try half the number of Puma threads per worker. Heroku recommends a max thread count of 5 to work on their single dyno, so a client size of 2 or 3 will suffice per web dyno process.

Sidekiq.configure_client do |config|  
  config.redis = { size: 3, url: ENV["REDIS_URL"], namespace: "your-app" }
end  

So, how many connections will we be using:

Puma Workers * (Puma Max Threads / 2) * Heroku Web Dynos  

Again, Heroku recommends 2 workers per web dyno. If we have 2 web dynos, we'll have something like:

2 workers * 3 shared connections * 2 web dynos = 12 connections to redis.

The Server Math

The Heroku Redis Hobby-dev tier has a max connection limit of 20, so we have 8 connections left for our server connection pool. The number of connections the server uses is:

Heroku Job Dynos * Sidekiq Concurrency Count + 2 (reserved for internal Sidekiq stuff)  

We already know this has to come out to 8, the number of connections we have left, so if we only have one worker dyno:

1 * x + 2 = 8  
x = 6  

We can set the concurrency option in sidekiq.yml with a value of 6 and you don't have to worry about setting the :size parameter in the Sidekiq.configure_server block since Sidekiq will default to concurrency + 2.

Now you're taking advantage of all your allowed connections.

Other Considerations

Other processes such as the Heroku Scheduler or Rails Console that push to or pull from jobs on the Sidekiq queue will act as another client and initialize another set of connections. Even if the connections are only temporary, ensure you consider these in your client size total and treat them as an additional client "dyno".

Additional add-ons can also use connections of their own. For example, the RedisMonitor add-on uses 1 or 2 connections to monitor the Redis server. Be sure to take these into account as well.

TL;DR

Client Size = Puma Workers * (Puma Threads / 2) * Heroku Web Dynos  
Server Size = (Redis Connection Limit - Client Size - 2) / Heroku Job Dynos  

Thanks to Mike Perham and Alex Ostleitner for some pointers.