[Backgroundrb-devel] UTF8 postgres args saving issue

Justin Wood justin.wood at trifectagis.com
Fri Mar 27 00:24:39 EDT 2009


Hi, All

I have an encountered an issue where the args field is not saved correctly
to the database.  I encounter an error like this:

ActiveRecord::StatementInvalid (RuntimeError: ERROR    C22021    M invalid
byte sequence for encoding "UTF8": 0xcb3a    H This error can also happen if
the byte sequence does not match the encoding expected by the server, which
is controlled by "client_encoding".

Here's the SQL that gets send to the database (note that the dump-ed args
are written as a string):

F.\src\backend\utils\mb\wchar.c    L1545    Rreport_invalid_encoding: INSERT
INTO "bdrb_job_queues" ("args", "job_key", "taken", "worker_key",
"worker_method", "priority", "finished_at", "tag", "worker_name", "timeout",
"submitted_at", "finished", "runner_info", "submitter_info", "archived_at",
"scheduled_at", "started_at")
VALUES(E'{:car_idiË:inspection_report_name"First week inspection',
E'66', 0, E'', E'send_warranty_notice', NULL, NULL, NULL,
E'notification_worker', NULL, '2009-03-27 03:53:14.036000', 0, NULL, NULL,
NULL, '2009-04-03 03:53:13.917000', NULL) RETURNING "id"):

This won't happen all the time ... only when the array of bytes from the
dump (represented as a string) have combinations or bytes that can't be
interpreted as UTF8.

This could be fixed in a couple of ways I suppose the most obvious being how
the adapter saves bytea fields and encoding the dump as UTF8 ... but I
wasn't sure if it would un-encode.  So I implemented the fool proof option
of Base64 encoding the data, which means never having to worry about
encoding again (because this is not the first time I've had a character
encoding issue)

Here's the bulletproof hack that I added to my BdrbJobQueue

...
  #these accessors get around any possible character encoding issues with
the database
  def args=(args)
    write_attribute(:args, Base64.b64encode(args))
  end
  def args
    Base64.decode64(read_attribute(:args))
  end
...

Hope that helps someone.  It will help anyone who has the problem referred
to here
http://rubyforge.org/pipermail/backgroundrb-devel/2009-March/002325.html.
Note, to the best of my knowledge all my other UTF8 settings are correct.

Cheers
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20090327/3e05a7d9/attachment.html>


More information about the Backgroundrb-devel mailing list