[Dxspider-support] Downloads

[Dirk Koopman]

Create_master_badip_file.pl is the program I use to obtain the date to 
create the badip.torexit and .torrelay files. The site from which this 
data is derived requests that large numbers of people to do not connect 
to the site for bandwidth (and I suspect cost) reasons. They suggest 
that we do as we are and have a few sites that obtain, process and then 
distribute the result to our community.

One of the things I need to sort out are 2-4 backup sites that can that 
program and then redistribute it (independently) from GB7DJK, via a web 
download site. I am getting to an age where I could, after all, drop 
dead tomorrow or  be run over by a bus.

I also have to devise a system to merge badip.local files into one or 
more copies of badip.global, so that that too can be managed by the 
community rather than just me. I feel an 'upload' command coming on :-)

But it appears I have still got some more firefighting to do first...

73 Dirk G1TLH

On 12/03/2023 16:58, Björn Ekelund wrote:
> Dirk,
>
> Please forgive my ignorance but I have noticed a file called 
> create_master_badip_files.pl <http://create_master_badip_files.pl> in 
> spider/perl.
>
> The comments in it seem to suggest it is doing what the download 
> commands you
> propose for crontab do. It however does not seem to be in use, at 
> least not on my installation.
> Is it obsolete?
>
> What are the "conversion programs" you are referring to in your post? 
> Is there a special command to be added
> to the local crontab to add the downloaded data to badip.local? Or am 
> I just misunderstanding?
>
> Björn SM7IUN
>
>
>
>
>
>
>
> On Sun, Mar 12, 2023 at 12:06 AM Dirk Koopman via Dxspider-support 
> <dxspider-support at tobit.co.uk> wrote:
>
>     I should add: the 'download' is non-blocking. As you can see in
>     the example below they run in parallel:
>
>     11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>     'run_cmd('download http://www.dxspider.net/download/badip.torexit')'
>     11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>     http://www.dxspider.net/download/badip.torexit
>     11Mar2023 at 20:24:00 (*) download: url
>     http://www.dxspider.net/download/badip.torexit
>     11Mar2023 at 20:24:00 (progress) CMD: 'download
>     http://www.dxspider.net/download/badip.torexit' by GB7TLH ip:
>     127.0.0.1 2mS
>     11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>     'run_cmd('download http://www.dxspider.net/download/badip.torrelay')'
>     11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>     http://www.dxspider.net/download/badip.torrelay
>     11Mar2023 at 20:24:00 (*) download: url
>     http://www.dxspider.net/download/badip.torrelay
>     11Mar2023 at 20:24:00 (progress) CMD: 'download
>     http://www.dxspider.net/download/badip.torrelay' by GB7TLH ip:
>     127.0.0.1 1mS
>     11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>     'run_cmd('download http://www.dxspider.net/download/badip.global')'
>     11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>     http://www.dxspider.net/download/badip.global
>     11Mar2023 at 20:24:00 (*) download: url
>     http://www.dxspider.net/download/badip.global
>     11Mar2023 at 20:24:00 (progress) CMD: 'download
>     http://www.dxspider.net/download/badip.global' by GB7TLH ip:
>     127.0.0.1 1mS
>     11Mar2023 at 20:24:00 (progress) download:
>     /spider/local_data/badip.global successfully downloaded
>     11Mar2023 at 20:24:00 (progress) download:
>     /spider/local_data/badip.torexit successfully downloaded
>     11Mar2023 at 20:24:00 (progress) download:
>     /spider/local_data/badip.torrelay successfully downloaded
>
>     Given that sysops are successfully running the mojo branch on
>     Windows (apart from spawning commands), I believe this will work
>     identically there as well.
>
>     On 11/03/2023 22:54, Dirk Koopman via Dxspider-support wrote:
>>     I have just pushed new DXSpider command 'download', it is a bit
>>     experimental, but I have been using it successfully in a few
>>     places. There may be additional facilities added to it based on
>>     demand. Here is what the Changes file says (can't be bothered to
>>     type it all out again):
>>
>>     1. I am aware that there are windows nodes out there on mojo
>>     (brave, impetuous
>>        that you are) and wget is not an easy option to download badip
>>     files. So,
>>        through the power of mojo I give you the new 'download'
>>     command. As it
>>        stands at the moment, it will download a file from a webserver
>>     and place
>>        it in /spider/local_data. If it downloads successfully, it
>>     will overwrite
>>        any file with that name in /spider/local_data. Needless to say
>>     this is a
>>        sysop only command.
>>
>>     24 * * * * run_cmd('download
>>     http://www.dxspider.net/download/badip.torexit')
>>     24 * * * * run_cmd('download
>>     http://www.dxspider.net/download/badip.torrelay')
>>     24 * * * * run_cmd('download
>>     http://www.dxspider.net/download/badip.global')
>>     25 * * * * run_cmd('load/badip')
>>
>>        would be an obvious use. But it can also be used to download
>>     spot, USDB
>>        and other data as well. If you do use it then you will still
>>     need to
>>        do a spawn_cmd in crontab for the conversion programs, but
>>     that too is
>>        going to change.
>>
>>     You don't need to restart, everything it does is in the command.
>>
>>     Enjoy
>>
>>     Dirk G1TLH
>>
>>     _______________________________________________
>>     Dxspider-support mailing list
>>     Dxspider-support at tobit.co.uk
>>     https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>     _______________________________________________
>     Dxspider-support mailing list
>     Dxspider-support at tobit.co.uk
>     https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20230312/64ac48d3/attachment.htm>