[Dxspider-support] Downloads

[Björn Ekelund]

Thanks Dirk, much appreciated.

Björn SM7IUN

On Sun, Mar 12, 2023 at 10:57 PM Dirk Koopman <djk at tobit.co.uk> wrote:

> Create_master_badip_file.pl is the program I use to obtain the date to
> create the badip.torexit and .torrelay files. The site from which this data
> is derived requests that large numbers of people to do not connect to the
> site for bandwidth (and I suspect cost) reasons. They suggest that we do as
> we are and have a few sites that obtain, process and then distribute the
> result to our community.
>
> One of the things I need to sort out are 2-4 backup sites that can that
> program and then redistribute it (independently) from GB7DJK, via a web
> download site. I am getting to an age where I could, after all, drop dead
> tomorrow or  be run over by a bus.
>
> I also have to devise a system to merge badip.local files into one or more
> copies of badip.global, so that that too can be managed by the community
> rather than just me. I feel an 'upload' command coming on :-)
>
> But it appears I have still got some more firefighting to do first...
>
> 73 Dirk G1TLH
>
> On 12/03/2023 16:58, Björn Ekelund wrote:
>
> Dirk,
>
> Please forgive my ignorance but I have noticed a file called
> create_master_badip_files.pl in spider/perl.
>
> The comments in it seem to suggest it is doing what the download commands
> you
> propose for crontab do. It however does not seem to be in use, at least
> not on my installation.
> Is it obsolete?
>
> What are the "conversion programs" you are referring to in your post? Is
> there a special command to be added
> to the local crontab to add the downloaded data to badip.local? Or am I
> just misunderstanding?
>
> Björn SM7IUN
>
>
>
>
>
>
>
> On Sun, Mar 12, 2023 at 12:06 AM Dirk Koopman via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
>> I should add: the 'download' is non-blocking. As you can see in the
>> example below they run in parallel:
>>
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.torexit')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.torexit
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.torexit
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.torexit' by GB7TLH ip: 127.0.0.1
>> 2mS
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.torrelay')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.torrelay
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.torrelay
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.torrelay' by GB7TLH ip: 127.0.0.1
>> 1mS
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.global')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.global
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.global
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.global' by GB7TLH ip: 127.0.0.1
>> 1mS
>> 11Mar2023 at 20:24:00 (progress) download: /spider/local_data/badip.global
>> successfully downloaded
>> 11Mar2023 at 20:24:00 (progress) download: /spider/local_data/badip.torexit
>> successfully downloaded
>> 11Mar2023 at 20:24:00 (progress) download:
>> /spider/local_data/badip.torrelay successfully downloaded
>>
>> Given that sysops are successfully running the mojo branch on Windows
>> (apart from spawning commands), I believe this will work identically there
>> as well.
>>
>> On 11/03/2023 22:54, Dirk Koopman via Dxspider-support wrote:
>>
>> I have just pushed new DXSpider command 'download', it is a bit
>> experimental, but I have been using it successfully in a few places. There
>> may be additional facilities added to it based on demand. Here is what the
>> Changes file says (can't be bothered to type it all out again):
>>
>> 1. I am aware that there are windows nodes out there on mojo (brave,
>> impetuous
>>    that you are) and wget is not an easy option to download badip files.
>> So,
>>    through the power of mojo I give you the new 'download' command. As it
>>    stands at the moment, it will download a file from a webserver and
>> place
>>    it in /spider/local_data. If it downloads successfully, it will
>> overwrite
>>    any file with that name in /spider/local_data. Needless to say this is
>> a
>>    sysop only command.
>>
>> 24 * * * * run_cmd('download
>> http://www.dxspider.net/download/badip.torexit')
>> 24 * * * * run_cmd('download
>> http://www.dxspider.net/download/badip.torrelay')
>> 24 * * * * run_cmd('download
>> http://www.dxspider.net/download/badip.global')
>> 25 * * * * run_cmd('load/badip')
>>
>>    would be an obvious use. But it can also be used to download spot,
>> USDB
>>    and other data as well. If you do use it then you will still need to
>>    do a spawn_cmd in crontab for the conversion programs, but that too is
>>    going to change.
>>
>> You don't need to restart, everything it does is in the command.
>>
>> Enjoy
>>
>> Dirk G1TLH
>>
>> _______________________________________________
>> Dxspider-support mailing listDxspider-support at tobit.co.ukhttps://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>>
>> _______________________________________________
>> Dxspider-support mailing list
>> Dxspider-support at tobit.co.uk
>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20230313/73edc5ea/attachment-0001.htm>