RightAws::S3 does not support the :multi_thread parameter

I’ve just emerged from a deep development cycle and am finding a few things that bear sharing with the rest of the world.  If you are using RightAws::S3 or RightAws::S3Interface you may want to re-evaluate your use in the context of Ruby Threads.

While earlier versions had an optional “:multi_thread => true” flag, in the most recent right_aws gem (version 2.0.0), the :multi_thread parameter is simply ignored.  After some digging around I found this note from one of the developers on github: https://github.com/rightscale/right_aws/issues/24



Right_aws does not support multi-threading (and we don’t have that option any more). If you need multiple threads then you must have RightAws::S3 or RightAws::S3Interface instance per thread. Once created the RightAws::S3 instance must be used in the thread if was created.

Plz make sure you do this and you do not access one RightAws::S3 instance from different threads

The RightScale http_connection gem wraps some retry logic around the basic Ruby Net::Http object, giving the abstraction of a more reliable HTTP connection.  RightAws::S3 is built on top of that.  In earlier versions, RightAws::S3 attempted to mask multithreaded execution by managing connections amongst your threads automatically.

It seems that this probably didn’t work correctly in some cases.  I’m following the suggestion above and am observing reliable execution.

A Postscript

I have become suspicious of anything that uses the words “threads”, “gem” and “automatic” together.  In the world of Ruby, your best bet is to manage TCP connections yourself in the context of threads.  It’s not that hard, and anything claiming to do it for you automatically probably wasn’t written with your particular set of constraints in mind.  Do it yourself.

One thought on “RightAws::S3 does not support the :multi_thread parameter”

Leave a Reply

Your email address will not be published. Required fields are marked *