RightAws::S3 does not support the :multi_thread parameter

I’ve just emerged from a deep development cycle and am finding a few things that bear sharing with the rest of the world.  If you are using RightAws::S3 or RightAws::S3Interface you may want to re-evaluate your use in the context of Ruby Threads.

While earlier versions had an optional “:multi_thread => true” flag, in the most recent right_aws gem (version 2.0.0), the :multi_thread parameter is simply ignored.  After some digging around I found this note from one of the developers on github: https://github.com/rightscale/right_aws/issues/24



Right_aws does not support multi-threading (and we don’t have that option any more). If you need multiple threads then you must have RightAws::S3 or RightAws::S3Interface instance per thread. Once created the RightAws::S3 instance must be used in the thread if was created.

Plz make sure you do this and you do not access one RightAws::S3 instance from different threads

The RightScale http_connection gem wraps some retry logic around the basic Ruby Net::Http object, giving the abstraction of a more reliable HTTP connection.  RightAws::S3 is built on top of that.  In earlier versions, RightAws::S3 attempted to mask multithreaded execution by managing connections amongst your threads automatically.

It seems that this probably didn’t work correctly in some cases.  I’m following the suggestion above and am observing reliable execution.

A Postscript

I have become suspicious of anything that uses the words “threads”, “gem” and “automatic” together.  In the world of Ruby, your best bet is to manage TCP connections yourself in the context of threads.  It’s not that hard, and anything claiming to do it for you automatically probably wasn’t written with your particular set of constraints in mind.  Do it yourself.

Ruby 1.9.2 becomes corrupted on Centos

We recently completed a fresh compile of Ruby 1.9.2 from sources on a Centos machine.   After a few days, the Ruby interpreter itself seemed to get corrupted.  The problem manifested itself as the interpreter failing to load rubygems.  See the Output Trace at the end of this article for what we saw.

Two articles on the web suggested that the Linux prelink optimizer was actually corrupting the Ruby binary.  The solution is to tell the system to BAN prelinking on the Ruby binary.  After adding the “-b /usr/bin/ruby” line to the /etc/prelink.conf file as suggested, the problem seems to have gone away.

I’m reposting the links to these articles to help others in a similar situation find this information.  These saved us a lot of time diagnosing a very mysterious problem.

The Referenced Articles



The Output Trace

<internal:lib/rubygems/custom_require>:29:in `require': closed stream(IOError)  
from <internal:lib/rubygems/custom_require>:29:in `require'  
from /usr/lib/ruby/1.9.1/irb.rb:13:in `<top (required)>'  
from <internal:lib/rubygems/custom_require>:29:in `require'  
from <internal:lib/rubygems/custom_require>:29:in `require'  from /usr/bin/irb:9:in `<main>'