David,<br><br>I was running on Windows, and when I moved to linux the problem disappeared (I'm assuming because linux automatically uses cferret?). Thanks for the help!<br><br>Justin<br><br><br><div><span class="gmail_quote">
On 6/16/06, <b class="gmail_sendername">David Balmain</b> <<a href="mailto:dbalmain.ml@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
dbalmain.ml@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">On 6/17/06, Justin Kan <<a href="mailto:justin.kan@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
justin.kan@gmail.com</a>> wrote:<br>> Hi,<br>><br>> I'm using the StandardAnalyzer to build an index, and passing in Documents<br>> that have Fields that contain large tokens (22+ characters) interpersed with
<br>> normal English words. This seems to cause the IndexWriter to slow to a<br>> crawl. Is this a known issue, or am I doing something wrong?<br><br>Hi Justin,<br><br>I haven't come accross this problem? Are you on Windows by any chance?
<br>Currently Ferret is just generally slow on Windows because it is pure<br>Ruby code. One problem large tokens may cause is the general increase<br>in the number of terms in the index which can slow down indexing a<br>
little but it would surprise me if it was making a huge difference
<br>unless there was a particularly large number of them.<br><br>> If this is a known issue I don't have any problem just not indexing tokens<br>> longer than a certain length, but what's the best way to eliminate them?
<br>> Using a TokenFilter on my own Analyzer? Sorry for the newbish questions, I'm<br>> new to ferret having never used lucene. Thanks in advance,<br><br>Yes, using a token filter will do the job. Have a look in the analysis
<br>module of Ferret for some examples. I'd be interested to hear if it<br>makes any difference.<br><br>Cheers,<br>Dave<br>_______________________________________________<br>Ferret-talk mailing list<br><a href="mailto:Ferret-talk@rubyforge.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Ferret-talk@rubyforge.org</a><br><a href="http://rubyforge.org/mailman/listinfo/ferret-talk" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://rubyforge.org/mailman/listinfo/ferret-talk</a><br>
</blockquote></div><br><br><a href="mailto:justin@kiko.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"></a>