[Dxspider-support] How far back does sh/dx go?

[Keith Maton]

Thank you Dirk, that’s exactly the information I needed and the variable name I was looking for.

I had the cache increased for about twenty minutes before I put it back because I did some quick testing and it didn’t seem to make any difference whatsoever apart from fill up the physical memory.  I couldn’t see any benefit so I just removed the tweak.

73 Keith


> On 28 May 2024, at 15:37, Dirk Koopman via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
> 
> I have been away for the last several weeks but I am now back.
> 
> Sh/dx, by default, only searches the last 100 days worth of spots ($Spot::maxdays controls this). This value is a compromise between those users who are just looking for "recent" spots (99+%) and those others that might want go further back. Sh/dx does a linear search, it is not indexed in any way, therefore the larger you make $maxdays, the more spot files will searched for that elusive spot.
> 
> 'Help sh/dx" is your friend here, but it may not do exactly what you want.
> 
> Amongst the help text is:
> 
>  day <number>    - starting <number> days ago
>  day <from>-<to> - <from> days <to> days ago
>      <from>/<to>
> 
> I have just noticed a small feature in the code such that the 'day <from>/<to>' will only search (a maximum of) $maxdays starting from the <from> day. I've added that to the list.
> 
> Please don't load all your spots into cache. It is really only designed to deal with startup scripts either setup in DXSpider or a client program that can't cope with the fact that sh/dx will start an asynchronous process so that the other users of your node don't have to wait for your search for a unique spot from three years ago. That 3GB will start to bite after a month or so of running with many users and will slow everything down for them as well.
> 
> 73 Dirk G1TLH
> 
> On 23/05/2024 09:29, Keith Maton via Dxspider-support wrote:
>> Answering myself because I’m experimenting ;-)
>> 
>> I added the following variable into /spider/scripts/startup
>> 
>> set/var $Spot::spotcachedays = 365
>> 
>> As the node loaded, I saw the following appear in the debug log
>> 
>> ...
>> Spot::init read 10992 spots from 2023|246
>> Spot::init read 14019 spots from 2023|245
>> Spot::init 365 files of spots read into cache in 16493mS
>> 
>> So what I think was happening there was the last 365 days (I don’t quite have a years worth of history yet) of spot data were loaded into cache, probably to speed up searching.  Looking at the memory usage on the host, I was up at over 3Gb of RAM in use, so this makes sense.
>> 
>> However, an sh/dx still doesn’t search all the way back so this isn’t the correct variable to change.
>> 
>> When I removed that line and restarted, the memory usage dropped back to normal and I could see from debug that by default it loads two days of old spots into cache.
>> 
>> So I’m learning by experimentation but I still don’t know how far back sh/dx searches by default and whether it’s possible to change it.
>> 
>> 73 Keith.
>> 
>> 
>>> On 21 May 2024, at 11:59, Keith Maton via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
>>> 
>>> I see there’s a variable in /spider/cmd/show/dx.pl called spotcachedays which I assume is the period that spots are cached.
>>> 
>>> Does anyone know where this variable is defined and can I change it?
>>> 
>>> Thanks,
>>> 
>>> 73 Keith.
>>> 
>>>> On 16 May 2024, at 16:29, Keith Maton via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> A while ago, I increased the amount of time I keep old spot files in /spider/local_data/spots up to one year, I did this by tweaking the cleanup entry I have in my crontab.
>>>> 
>>>> I wanted to check the date an expedition was last spotted and I just used sh/dx but they weren’t found.   It was definitely within the last year.
>>>> 
>>>> How far back does sh/dx actually search by default please and can this be changed?
>>>> 
>>>> Thanks,
>>>> 
>>>> 73 Keith
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Dxspider-support mailing list
>>>> Dxspider-support at tobit.co.uk
>>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>> 
>>> _______________________________________________
>>> Dxspider-support mailing list
>>> Dxspider-support at tobit.co.uk
>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>> 
>> _______________________________________________
>> Dxspider-support mailing list
>> Dxspider-support at tobit.co.uk
>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
> 
> 
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support