[Backgroundrb-devel] Creating another database connection for large mysql import?

Ezra Zygmuntowicz ezmobius at gmail.com
Mon Dec 18 19:06:41 EST 2006


On Dec 18, 2006, at 2:48 PM, Mike Garey wrote:

> I'm using backgroundrb to periodically download a large file via  
> ftp from a remote location and then import it into the database.   
> To perform the import, I was using the following:
>
> ActiveRecord::Base.connection.execute (%{load data infile ...;})
>
> although on a file with 2.5 million records, this can take 5  
> minutes, which seems to tie up my rails application while this  
> executes (even though it's being run from a backgroundrb process).   
> I figured this was happening because I'm using the same database  
> connection that the rest of my rails app is running on.. I then  
> tried using the following:
>
> Importer.establish_connection(:development)
> Importer.connection.execute(%{load data infile ... ;})
>
> where Importer is a dummy class I created just so I could try to  
> establish a database connection separate from the rest of my rails  
> classes.  However, this seems to produce the same behavior as when  
> I use ActiveRecord:: Base.connection.execute..
>
> Does anyone have any suggestions on how I might be able to resolve  
> this?
>
> Also, I read a message on the list that an upcoming version of  
> backgroundrb will have the ability to automatically reload a worker  
> if running in development mode and the worker file changes.. Just  
> wondering how to enable this, since I've been stopping/restarting  
> the server to get the file reloaded.
>
> Thanks for any help, and also a big thanks to Ezra (and any other  
> contributors) for all the hard work that's been put into  
> backgroundrb, it's truly appreciated!
>
> Mike


Hey Mike-

	Can you show me exactly how you are calling the worker method that  
does the 5 minute task? The do_work method is run asyncronously when  
the worker is first created but other methods you define in your  
worker will block rails if you call them directly.  In the 0.2.1  
release you can use work_thread to call your worker method in a  
thread so it spins off and works in the background while rails can  
continue on its way.

so
worker = MiddleMan.worker(session[:key)
worker.work_thread(:some_method)


Cheers-
-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)




More information about the Backgroundrb-devel mailing list