[Dxspider-support] Downloads
Howard Leadmon
howard at leadmon.net
Thu Mar 16 05:33:24 GMT 2023
Hello Dirk,
I am just now trying to catch up on the emails on the list, catching
Covid for sure knocked me down for a few. That said, if I can help
with replication, or distribution of anything that will help the cluster
network just let me know. My server cluster is all on 10GE links, and
sets on multiple 10GE pathways to the network. I could probably serve
the lists to all the clusters every minute 24/7 and laugh while I did
it. The WB3FFV-2 node is a VM on my cluster, and it would be a piece
of cake to build a VM we could use to distribute anything you desire..
73's de WB3FFV
---
Howard Leadmon -howard at leadmon.net
PBW Communications, LLC
http://www.pbwcomm.com
On 3/12/2023 5:55 PM, Dirk Koopman via Dxspider-support wrote:
> Create_master_badip_file.pl is the program I use to obtain the date to
> create the badip.torexit and .torrelay files. The site from which this
> data is derived requests that large numbers of people to do not
> connect to the site for bandwidth (and I suspect cost) reasons. They
> suggest that we do as we are and have a few sites that obtain, process
> and then distribute the result to our community.
>
> One of the things I need to sort out are 2-4 backup sites that can
> that program and then redistribute it (independently) from GB7DJK, via
> a web download site. I am getting to an age where I could, after all,
> drop dead tomorrow or be run over by a bus.
>
> I also have to devise a system to merge badip.local files into one or
> more copies of badip.global, so that that too can be managed by the
> community rather than just me. I feel an 'upload' command coming on :-)
>
> But it appears I have still got some more firefighting to do first...
>
> 73 Dirk G1TLH
>
> On 12/03/2023 16:58, Björn Ekelund wrote:
>> Dirk,
>>
>> Please forgive my ignorance but I have noticed a file called
>> create_master_badip_files.pl <http://create_master_badip_files.pl> in
>> spider/perl.
>>
>> The comments in it seem to suggest it is doing what the download
>> commands you
>> propose for crontab do. It however does not seem to be in use, at
>> least not on my installation.
>> Is it obsolete?
>>
>> What are the "conversion programs" you are referring to in your post?
>> Is there a special command to be added
>> to the local crontab to add the downloaded data to badip.local? Or am
>> I just misunderstanding?
>>
>> Björn SM7IUN
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Mar 12, 2023 at 12:06 AM Dirk Koopman via Dxspider-support
>> <dxspider-support at tobit.co.uk> wrote:
>>
>> I should add: the 'download' is non-blocking. As you can see in
>> the example below they run in parallel:
>>
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.torexit')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.torexit
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.torexit
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.torexit' by GB7TLH ip:
>> 127.0.0.1 2mS
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.torrelay')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.torrelay
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.torrelay
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.torrelay' by GB7TLH ip:
>> 127.0.0.1 1mS
>> 11Mar2023 at 20:24:00 (cron) cron: UTC 24 20 11 3 6 -> doing
>> 'run_cmd('download http://www.dxspider.net/download/badip.global')'
>> 11Mar2023 at 20:24:00 (cron) DXCron::run_cmd: download
>> http://www.dxspider.net/download/badip.global
>> 11Mar2023 at 20:24:00 (*) download: url
>> http://www.dxspider.net/download/badip.global
>> 11Mar2023 at 20:24:00 (progress) CMD: 'download
>> http://www.dxspider.net/download/badip.global' by GB7TLH ip:
>> 127.0.0.1 1mS
>> 11Mar2023 at 20:24:00 (progress) download:
>> /spider/local_data/badip.global successfully downloaded
>> 11Mar2023 at 20:24:00 (progress) download:
>> /spider/local_data/badip.torexit successfully downloaded
>> 11Mar2023 at 20:24:00 (progress) download:
>> /spider/local_data/badip.torrelay successfully downloaded
>>
>> Given that sysops are successfully running the mojo branch on
>> Windows (apart from spawning commands), I believe this will work
>> identically there as well.
>>
>> On 11/03/2023 22:54, Dirk Koopman via Dxspider-support wrote:
>>> I have just pushed new DXSpider command 'download', it is a bit
>>> experimental, but I have been using it successfully in a few
>>> places. There may be additional facilities added to it based on
>>> demand. Here is what the Changes file says (can't be bothered to
>>> type it all out again):
>>>
>>> 1. I am aware that there are windows nodes out there on mojo
>>> (brave, impetuous
>>> that you are) and wget is not an easy option to download
>>> badip files. So,
>>> through the power of mojo I give you the new 'download'
>>> command. As it
>>> stands at the moment, it will download a file from a
>>> webserver and place
>>> it in /spider/local_data. If it downloads successfully, it
>>> will overwrite
>>> any file with that name in /spider/local_data. Needless to
>>> say this is a
>>> sysop only command.
>>>
>>> 24 * * * * run_cmd('download
>>> http://www.dxspider.net/download/badip.torexit')
>>> 24 * * * * run_cmd('download
>>> http://www.dxspider.net/download/badip.torrelay')
>>> 24 * * * * run_cmd('download
>>> http://www.dxspider.net/download/badip.global')
>>> 25 * * * * run_cmd('load/badip')
>>>
>>> would be an obvious use. But it can also be used to download
>>> spot, USDB
>>> and other data as well. If you do use it then you will still
>>> need to
>>> do a spawn_cmd in crontab for the conversion programs, but
>>> that too is
>>> going to change.
>>>
>>> You don't need to restart, everything it does is in the command.
>>>
>>> Enjoy
>>>
>>> Dirk G1TLH
>>>
>>> _______________________________________________
>>> Dxspider-support mailing list
>>> Dxspider-support at tobit.co.uk
>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>> _______________________________________________
>> Dxspider-support mailing list
>> Dxspider-support at tobit.co.uk
>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20230316/e234a501/attachment-0001.htm>
More information about the Dxspider-support
mailing list