[Dxspider-support] How far back does sh/dx go?

[Dirk Koopman]

I have been away for the last several weeks but I am now back.

Sh/dx, by default, only searches the last 100 days worth of spots 
($Spot::maxdays controls this). This value is a compromise between those 
users who are just looking for "recent" spots (99+%) and those others 
that might want go further back. Sh/dx does a linear search, it is not 
indexed in any way, therefore the larger you make $maxdays, the more 
spot files will searched for that elusive spot.

'Help sh/dx" is your friend here, but it may not do exactly what you want.

Amongst the help text is:

  day <number>    - starting <number> days ago
  day <from>-<to> - <from> days <to> days ago
      <from>/<to>

I have just noticed a small feature in the code such that the 'day 
<from>/<to>' will only search (a maximum of) $maxdays starting from the 
<from> day. I've added that to the list.

Please don't load all your spots into cache. It is really only designed 
to deal with startup scripts either setup in DXSpider or a client 
program that can't cope with the fact that sh/dx will start an 
asynchronous process so that the other users of your node don't have to 
wait for your search for a unique spot from three years ago. That 3GB 
will start to bite after a month or so of running with many users and 
will slow everything down for them as well.

73 Dirk G1TLH

On 23/05/2024 09:29, Keith Maton via Dxspider-support wrote:
> Answering myself because I’m experimenting ;-)
>
> I added the following variable into /spider/scripts/startup
>
> set/var $Spot::spotcachedays = 365
>
> As the node loaded, I saw the following appear in the debug log
>
> ...
> Spot::init read 10992 spots from 2023|246
> Spot::init read 14019 spots from 2023|245
> Spot::init 365 files of spots read into cache in 16493mS
>
> So what I think was happening there was the last 365 days (I don’t quite have a years worth of history yet) of spot data were loaded into cache, probably to speed up searching.  Looking at the memory usage on the host, I was up at over 3Gb of RAM in use, so this makes sense.
>
> However, an sh/dx still doesn’t search all the way back so this isn’t the correct variable to change.
>
> When I removed that line and restarted, the memory usage dropped back to normal and I could see from debug that by default it loads two days of old spots into cache.
>
> So I’m learning by experimentation but I still don’t know how far back sh/dx searches by default and whether it’s possible to change it.
>
> 73 Keith.
>
>
>> On 21 May 2024, at 11:59, Keith Maton via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
>>
>> I see there’s a variable in /spider/cmd/show/dx.pl called spotcachedays which I assume is the period that spots are cached.
>>
>> Does anyone know where this variable is defined and can I change it?
>>
>> Thanks,
>>
>> 73 Keith.
>>
>>> On 16 May 2024, at 16:29, Keith Maton via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
>>>
>>> Hi all,
>>>
>>> A while ago, I increased the amount of time I keep old spot files in /spider/local_data/spots up to one year, I did this by tweaking the cleanup entry I have in my crontab.
>>>
>>> I wanted to check the date an expedition was last spotted and I just used sh/dx but they weren’t found.   It was definitely within the last year.
>>>
>>> How far back does sh/dx actually search by default please and can this be changed?
>>>
>>> Thanks,
>>>
>>> 73 Keith
>>>
>>>
>>> _______________________________________________
>>> Dxspider-support mailing list
>>> Dxspider-support at tobit.co.uk
>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>> _______________________________________________
>> Dxspider-support mailing list
>> Dxspider-support at tobit.co.uk
>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support