[Dxspider-support] Followup - Hang hang and some more hang...

[Dirk Koopman]

On Sun, 2005-11-13 at 14:42 -0500, charlie carroll wrote:
> An observation.  Kelly sent me his load test software.  The test robots 
> issue a series of commands once logged in.  One command causes my node 
> to noticably slow down....   sh/dx/10 15.  This means show me 10 entries 
> from the spot database each which contains the text "15"  As a result, 
> my node has to open and search every spot log searching for the text 
> "15"  This takes a noticable amount of time.  If I have several robots 
> connected doing the same thing, my node seemingly grinds to a halt until 
> each of these searches is processed.  One other observation... a search 
> such as  sh/dx/10 3d2 completes a lot quicker.
> 
> Dirk can probably explain why the search for 3d2 completes faster than 
> the search for the text 15.
> 

Ok. You have put your finger on (just) one of the reasons why I want to
start looking at something a bit more sophisticated than flat files (for
searching spots) on the one hand and more reliable than DB_File (for
things like user file access) on the other.

The sh/dx query (as someone else has pointed out) is wrong. However, you
could argue that I could be a bit cleverer and catch that sort of thing.
Having said that, what is happening is that the failure case for a
search (ie none, or less than the total number, of the required prefix)
will search the last 100 days worth of spots. Linearly. This is a
comparatively slow operation. 

This is not a problem in normal use. But doing several failed searches
will slow it down to a crawl, because it isn't a multithreaded process.
Again, the tradeoff is simplicity for writing, ease (and consistency) of
debugging versus possible performance gains. If I had wanted something
that was blindingly fast, I would have written it in C - if I had had
time. The usual problem, you can have two of: fast, good or cheap (read
easy to program/modify)[be portable and easy to update].

Using a simple database system like SQLite means that this sort of thing
would not happen. A query would complete very quickly (given that the
indexing is sane). It is also a module written in C and so is inherently
quicker for *its* (completely different) failure cases (there is always
at least one for tasks like these).

I must say that I am really curious as to why certain people only manage
a "few" (say 60 or so connections) and others (including me BTW) seem
not to have any limitations at all. I believe Angel regularly has 150 or
more on his box. 

I muse gently that, since there are no limitations in the code as to the
number of connects, whether we should be looking somewhere else. And
since you have had essentially the same problems with linux and windows
(no more than 60 active connections), I wonder whether we are looking at
a (stateful) firewall "active table" resource filling up problem? 

Dirk G1TLH