[Dxspider-support] File Moves going to Mojo..
Howard Leadmon
howard at leadmon.net
Sat Dec 4 03:47:22 GMT 2021
Oh do I remember then days! I just filled a dumpster with old stuff
I had held on to for far to long, 8" floppies, modem banks, ISDN banks,
old DEC Alpha processors, and Sun processors, was amazing how ancient
some of the gear was still in storage. No teletypes though, guess I
got rid of all of them.. I was just thinking the other day, pretty
amazing, I went though hell to get a T1/E1 line at my house long ago,
and how I was so ahead of all the modem users at that point. Now I
actually have 10GE lit fiber in my house, and built a 10GE LAN so I
could take advantage of it. Heck the new DSspider server has multiple
10GE ports connecting it to the internet, how times change..
On the space used by spots, I actually have logs back to 2011 from
when I first used your code I believe, but I copied some of it out, as I
didn't think it was used. Is there any point in putting it all back
into the data directory structure, as like you said space is not an
issue. I just assumed it was clutter and copied and then removed it.
I like your idea of a web reg page, that would be nice, and I sure
have no issues loading a LAMP stack or whatever is needed on the
machine, pretty much anything is adjustable with overall ease.\
Oh one last question, in my crontab on the old node I had the following:
#
# Monday 1am local - Get latest FCC data (W1NR updates Sundays)
3 5 * * 1 spawn("cd /tmp && wget -Nq ftp://ftp.w1nr.net/usdbraw.gz &&
/spider/perl/create_usdb.pl usdbraw.gz")
44 5 * * 1 run_cmd("load/usdb")
50 5 * * 1 spawn("rm /tmp/usdbraw.gz")
#
# Friday 1am local - Get latest Keps to be ready for any contest
1 5 * * 5 spawn("cd /tmp && wget -Nq
https://www.amsat.org/amsat/ftp/keps/current/nasabare.txt &&
/spider/perl/convkeps.pl -p nasabare.txt")
4 5 * * 5 run_cmd("load/keps")
5 5 * * 5 spawn("rm /tmp/nasabare.txt")
#
# Friday 1am local - Get latest cty.dat to be ready for any contest
4 5 * * 5 spawn("cd /spider/data && wget -Nq
https://www.country-files.com/cty/cty.dat")
6 5 * * 5 spawn("cd /spider/data && wget -Nq
https://www.country-files.com/cty/wpxloc.raw")
11 5 * * 5 spawn("cd /spider/data && /spider/perl/create_prefix.pl")
13 5 * * 5 run_cmd("load/prefixes")
#
# Connect for RBN Spots
* * * * * start_connect('sk0mmr') unless connected('sk0mmr')
* * * * * start_connect('sk1mmr') unless connected('sk1mmr')
#
Should I still be running stuff like this? If not, I need to clean
it up, as it's been in there for years.. LOL
---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com
On 12/3/2021 9:53 PM, Dirk Koopman wrote:
> When I wrote the first versions of the code (in 1997) I had envisaged
> that people would like to add their own commands to it, hence cmd and
> local_cmd directories. But in those days people were still on mainly
> on ax25, dialup or 64/128Kb ISDN lines (as I was) and it was up to me
> to keep things like prefixes/keps. The concept of usdb and other
> databases were not a thing until later. When the mojo branch happened
> I split the data into stuff I provided and stuff sysops would
> routinely update like prefixes etc. This has caused problems with
> updating in the past, so the concept of local_data happened. If there
> is a "clash" between a file in data and one in local_data, then the
> newest version wins.
>
> So you can safely leave stuff that is in data alone (and this is data
> that I consider I control) and may change from one release to another
> via git. In fact, if I do a release I will generally generate a prefix
> list from the latest CTY data and include that with whatever else is
> in the update. I just works seamlessly and also provides a "base" set
> of data on first startup. Stuff that I consider definitely sysop
> changeable then gets moved to local_data from data.
>
> So don't move anything that is left in data after first startup. What
> is left is git controlled defaults. I've fixed the motd and issue issues.
>
> As for data files generated. The debug directory is the only one that
> is automatically pruned and IIRC keeps 30 days worth by default. The
> rest is up to you. I have a record of every spot received since 1997
> (and wwv/wcy/system logs as well) for GB7DJK. Because they don't
> really take up much space and having them has proven useful on more
> than one occasion for people doing research and suchlike. Curiously
> the sqlite version of the spot data is more than twice as big as the
> ascii based versions. And neither is it as quick to access, especially
> when doing needle in haystack searches like 'sh/dx g1tlh'.
>
> The web interface is slowly creeping up my to do list. Once I have
> dealt with our random contest self spotter to my satisfaction, it
> could even be next. One of the strengths of the mojo branch is that
> the networking stack comes with a build in web server + websocket
> interface (see https://mojolicious.org/) . I am starting to modernise
> the instructions and I could easily "self-serve" them as static pages
> - as a start/proof of concept.
>
> All user data just gets converted into the new users file format
> (json) and that includes all registration data. That's another reason
> for activating the web server as I have numerous requests for an
> "easy" (i.e. low sysop involvement) registration system. I suspect
> that, after the shenanigans in this years CQWWs, registration may well
> become more widespread. Maybe even (more or less) mandatory.
>
> I think you will find that the new version is pretty stable right now...
>
> 73 Dirk G1TLH
>
> On 03/12/2021 23:07, Howard Leadmon wrote:
>> Hello Dirk,
>>
>> Thanks for the reply, and input. On the older server it had two
>> dual-core processors in a physical server, but as I have lots of free
>> resourcxes on one my my VM hosts, so I figured what the heck and just
>> virtualized the configuration.
>>
>> Now I get that most files were moved, and I had to copy across the
>> issue and motd files, but do any of the files I see in the old dir
>> need moved to the new directory?
>>
>> Looking at the old node I see:
>>
>> $ ls
>> CVS bands.pl motd_nor
>> usdbraw.gz.1 user_asc.oooo
>> badw_regex cty.dat prefix_data.pl
>> user_asc user_asc.ooooo
>> badw_regex.es.issue debug qsl.v1
>> user_asc.bak users.v3
>> badw_regex.gb.issue dupefile spots
>> user_asc.o wcy
>> badword log usdb.v1
>> user_asc.oo wpxloc.raw
>> badword.issue motd usdbraw.gz
>> user_asc.ooo wwv
>>
>>
>> Some of that is not over in the new local_data directory, so curious
>> if anything else needs moved.
>>
>> Also is there any recommendation on how long some of the data files
>> should be kept?
>>
>> If I look at the running node, I see under data:
>>
>> 9768372 ./debug/2021
>> 9768373 ./debug
>> 318079 ./spots/2020
>> 314002 ./spots/2021
>> 632081 ./spots
>> 166 ./wwv/2018
>> 173 ./wwv/2020
>> 171 ./wwv/2019
>> 160 ./wwv/2021
>> 671 ./wwv
>> 80079 ./log/2018
>> 74187 ./log/2020
>> 81871 ./log/2019
>> 534903 ./log/2021
>> 771039 ./log
>> 385 ./wcy/2021
>> 426 ./wcy/2019
>> 431 ./wcy/2020
>> 430 ./wcy/2018
>> 1674 ./wcy
>> 5 ./CVS
>>
>>
>> I just moved all this over to the new node intact, but figured maybe
>> there is a recommendation you might have.
>>
>> I did also enable the RBN code you have in Mojo, and it seems to be
>> running fine. Wondering if worth trying to toss a web interface on
>> it, as I for sure have lots of resources available if needed. The
>> host has two 12core modern Xeon's, with a few hundred gig of RAM and
>> quite a few TB of storage, so I just slice out what is needed.
>>
>> Finally, and I know many don't do this, and I am sure some users
>> avoid the node because of it, but I have always required the user to
>> register to post spots. I am guessing this just moved across and
>> works, but if there is anything special I need to be aware of with
>> Mojo in regards to this, by all means please let me know..
>>
>> 73's...
>>
>> ---
>> Howard Leadmon -howard at leadmon.net
>> PBW Communications, LLC
>> http://www.pbwcomm.com
>>
>> On 12/3/2021 4:45 PM, Dirk Koopman via Dxspider-support wrote:
>>> On 03/12/2021 17:35, Howard Leadmon via Dxspider-support wrote:
>>>>
>>>> With Dirk saying that he was now ready to switch to Mojo being
>>>> the main branch, I decided it was time to get off my backside and
>>>> setup a new server for DXspider.
>>> Hurrah! Please encourage all your friends to do the same.
>>>>
>>>> Overall the conversion went fairly easy, and I have the new Mojo
>>>> node running and connected back to the original mode, I did want to
>>>> check on one thing. I know from this list, that local_data needed
>>>> to created, and I know that my motd (and motd_nor) needed to be
>>>> moved over to the this directory.
>>>
>>> I'm glad that it is very easy to do [ed: I may have exaggerated that
>>> a tad]. There are some setup things that happen on first run of the
>>> mojo version. It appears that motd et al are not moved with the
>>> other things. That is changed on the next release. It will move the
>>> issue and motd file across to local_data automagically.
>>>
>>>>
>>>> What I am curious about is, did I need to move any other files
>>>> across to local_data? Do items like the city files need moved,
>>>> and scripts updated vs the old spider version?
>>> See above.
>>>>
>>>> At the moment in local_data I see:
>>>>
>>>> $ ls local_data
>>>> badword debug dxqsl.v1j motd rbn_cache usdb.v1
>>>> users.v3j wwv
>>>> cluster.lck dupefile log motd_nor spots user_json wcy
>>>>
>>>>
>>>> Before I take this public in place of the old version, I for sure
>>>> want to make sure everything is working OK, as I know a bunch of
>>>> people use it while contesting, so I strive for 100% uptime.
>>>>
>>>> /If any see this, and care to test this sucker out, it's always
>>>> appreciated. It can be reached at //*dxtest.wb3ffv.us*//on
>>>> //*port 7300*//over both IPv4 and IPv6./
>>>>
>>>> Dirk, if you see this, and recommendations on CPU and memory for a
>>>> VM running Spider. As I built this in a VM, I have it running
>>>> current Ubuntu, with 4x VCPU's and 8G of memory, and it seems quick
>>>> as heck, but then again there is nothing really connected to it..
>>>
>>> Well WA9PIE-2 runs in a 1CPU 1GB RAM 25GB Digital Ocean droplet. But
>>> with some recent changes to speed up simple sh/dx (sh/dx, sh/dx 100)
>>> in scripts and bring it back "inline" so, as a result of going
>>> through two CQWW contests, I would recommend a tad more RAM (say
>>> 2GB) to handle forking cmds (sh/dx g1tlh). 2CPUs is a nice to have,
>>> but WA9PIE seems to run 1200 users in CQWW SSB at about 15-20% CPU
>>> usage. But then the users don't do much except listen and ask for
>>> headings.
>>>
>>> I would suggest what you have there is (as Rolls Royce famously said
>>> when asked how much power the engine produced) adequate :-)
>>>
>>> I suspect a 2GB RPi4 with a 250TB SSD in a USB3 attached UASP
>>> capable caddy will run WA9PIE's load just fine.
>>>
>>>>
>>>>
>>>> 73's de WB3FFV
>>>>
>>>> ---
>>>> Howard Leadmon
>>>> PBW Communications, LLC
>>>> http://www.pbwcomm.com
>>>>
>>>>
>>>> _______________________________________________
>>>> Dxspider-support mailing list
>>>> Dxspider-support at tobit.co.uk
>>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>>
>>>
>>> _______________________________________________
>>> Dxspider-support mailing list
>>> Dxspider-support at tobit.co.uk
>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20211203/1db50500/attachment-0001.htm>
More information about the Dxspider-support
mailing list