[Dxspider-support] File Moves going to Mojo..
Dirk Koopman
djk at tobit.co.uk
Sat Dec 4 02:53:03 GMT 2021
When I wrote the first versions of the code (in 1997) I had envisaged
that people would like to add their own commands to it, hence cmd and
local_cmd directories. But in those days people were still on mainly on
ax25, dialup or 64/128Kb ISDN lines (as I was) and it was up to me to
keep things like prefixes/keps. The concept of usdb and other databases
were not a thing until later. When the mojo branch happened I split the
data into stuff I provided and stuff sysops would routinely update like
prefixes etc. This has caused problems with updating in the past, so the
concept of local_data happened. If there is a "clash" between a file in
data and one in local_data, then the newest version wins.
So you can safely leave stuff that is in data alone (and this is data
that I consider I control) and may change from one release to another
via git. In fact, if I do a release I will generally generate a prefix
list from the latest CTY data and include that with whatever else is in
the update. I just works seamlessly and also provides a "base" set of
data on first startup. Stuff that I consider definitely sysop changeable
then gets moved to local_data from data.
So don't move anything that is left in data after first startup. What is
left is git controlled defaults. I've fixed the motd and issue issues.
As for data files generated. The debug directory is the only one that is
automatically pruned and IIRC keeps 30 days worth by default. The rest
is up to you. I have a record of every spot received since 1997 (and
wwv/wcy/system logs as well) for GB7DJK. Because they don't really take
up much space and having them has proven useful on more than one
occasion for people doing research and suchlike. Curiously the sqlite
version of the spot data is more than twice as big as the ascii based
versions. And neither is it as quick to access, especially when doing
needle in haystack searches like 'sh/dx g1tlh'.
The web interface is slowly creeping up my to do list. Once I have dealt
with our random contest self spotter to my satisfaction, it could even
be next. One of the strengths of the mojo branch is that the
networking stack comes with a build in web server + websocket interface
(see https://mojolicious.org/) . I am starting to modernise the
instructions and I could easily "self-serve" them as static pages - as a
start/proof of concept.
All user data just gets converted into the new users file format (json)
and that includes all registration data. That's another reason for
activating the web server as I have numerous requests for an "easy"
(i.e. low sysop involvement) registration system. I suspect that, after
the shenanigans in this years CQWWs, registration may well become more
widespread. Maybe even (more or less) mandatory.
I think you will find that the new version is pretty stable right now...
73 Dirk G1TLH
On 03/12/2021 23:07, Howard Leadmon wrote:
> Hello Dirk,
>
> Thanks for the reply, and input. On the older server it had two
> dual-core processors in a physical server, but as I have lots of free
> resourcxes on one my my VM hosts, so I figured what the heck and just
> virtualized the configuration.
>
> Now I get that most files were moved, and I had to copy across the
> issue and motd files, but do any of the files I see in the old dir
> need moved to the new directory?
>
> Looking at the old node I see:
>
> $ ls
> CVS bands.pl motd_nor
> usdbraw.gz.1 user_asc.oooo
> badw_regex cty.dat prefix_data.pl
> user_asc user_asc.ooooo
> badw_regex.es.issue debug qsl.v1
> user_asc.bak users.v3
> badw_regex.gb.issue dupefile spots
> user_asc.o wcy
> badword log usdb.v1
> user_asc.oo wpxloc.raw
> badword.issue motd usdbraw.gz
> user_asc.ooo wwv
>
>
> Some of that is not over in the new local_data directory, so curious
> if anything else needs moved.
>
> Also is there any recommendation on how long some of the data files
> should be kept?
>
> If I look at the running node, I see under data:
>
> 9768372 ./debug/2021
> 9768373 ./debug
> 318079 ./spots/2020
> 314002 ./spots/2021
> 632081 ./spots
> 166 ./wwv/2018
> 173 ./wwv/2020
> 171 ./wwv/2019
> 160 ./wwv/2021
> 671 ./wwv
> 80079 ./log/2018
> 74187 ./log/2020
> 81871 ./log/2019
> 534903 ./log/2021
> 771039 ./log
> 385 ./wcy/2021
> 426 ./wcy/2019
> 431 ./wcy/2020
> 430 ./wcy/2018
> 1674 ./wcy
> 5 ./CVS
>
>
> I just moved all this over to the new node intact, but figured maybe
> there is a recommendation you might have.
>
> I did also enable the RBN code you have in Mojo, and it seems to be
> running fine. Wondering if worth trying to toss a web interface on
> it, as I for sure have lots of resources available if needed. The
> host has two 12core modern Xeon's, with a few hundred gig of RAM and
> quite a few TB of storage, so I just slice out what is needed.
>
> Finally, and I know many don't do this, and I am sure some users
> avoid the node because of it, but I have always required the user to
> register to post spots. I am guessing this just moved across and
> works, but if there is anything special I need to be aware of with
> Mojo in regards to this, by all means please let me know..
>
> 73's...
>
> ---
> Howard Leadmon -howard at leadmon.net
> PBW Communications, LLC
> http://www.pbwcomm.com
>
> On 12/3/2021 4:45 PM, Dirk Koopman via Dxspider-support wrote:
>> On 03/12/2021 17:35, Howard Leadmon via Dxspider-support wrote:
>>>
>>> With Dirk saying that he was now ready to switch to Mojo being
>>> the main branch, I decided it was time to get off my backside and
>>> setup a new server for DXspider.
>> Hurrah! Please encourage all your friends to do the same.
>>>
>>> Overall the conversion went fairly easy, and I have the new Mojo
>>> node running and connected back to the original mode, I did want to
>>> check on one thing. I know from this list, that local_data needed
>>> to created, and I know that my motd (and motd_nor) needed to be
>>> moved over to the this directory.
>>
>> I'm glad that it is very easy to do [ed: I may have exaggerated that
>> a tad]. There are some setup things that happen on first run of the
>> mojo version. It appears that motd et al are not moved with the other
>> things. That is changed on the next release. It will move the issue
>> and motd file across to local_data automagically.
>>
>>>
>>> What I am curious about is, did I need to move any other files
>>> across to local_data? Do items like the city files need moved, and
>>> scripts updated vs the old spider version?
>> See above.
>>>
>>> At the moment in local_data I see:
>>>
>>> $ ls local_data
>>> badword debug dxqsl.v1j motd rbn_cache usdb.v1
>>> users.v3j wwv
>>> cluster.lck dupefile log motd_nor spots user_json wcy
>>>
>>>
>>> Before I take this public in place of the old version, I for sure
>>> want to make sure everything is working OK, as I know a bunch of
>>> people use it while contesting, so I strive for 100% uptime.
>>>
>>> /If any see this, and care to test this sucker out, it's always
>>> appreciated. It can be reached at //*dxtest.wb3ffv.us*//on //*port
>>> 7300*//over both IPv4 and IPv6./
>>>
>>> Dirk, if you see this, and recommendations on CPU and memory for a
>>> VM running Spider. As I built this in a VM, I have it running
>>> current Ubuntu, with 4x VCPU's and 8G of memory, and it seems quick
>>> as heck, but then again there is nothing really connected to it..
>>
>> Well WA9PIE-2 runs in a 1CPU 1GB RAM 25GB Digital Ocean droplet. But
>> with some recent changes to speed up simple sh/dx (sh/dx, sh/dx 100)
>> in scripts and bring it back "inline" so, as a result of going
>> through two CQWW contests, I would recommend a tad more RAM (say 2GB)
>> to handle forking cmds (sh/dx g1tlh). 2CPUs is a nice to have, but
>> WA9PIE seems to run 1200 users in CQWW SSB at about 15-20% CPU usage.
>> But then the users don't do much except listen and ask for headings.
>>
>> I would suggest what you have there is (as Rolls Royce famously said
>> when asked how much power the engine produced) adequate :-)
>>
>> I suspect a 2GB RPi4 with a 250TB SSD in a USB3 attached UASP capable
>> caddy will run WA9PIE's load just fine.
>>
>>>
>>>
>>> 73's de WB3FFV
>>>
>>> ---
>>> Howard Leadmon
>>> PBW Communications, LLC
>>> http://www.pbwcomm.com
>>>
>>>
>>> _______________________________________________
>>> Dxspider-support mailing list
>>> Dxspider-support at tobit.co.uk
>>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>>
>>
>> _______________________________________________
>> Dxspider-support mailing list
>> Dxspider-support at tobit.co.uk
>> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20211204/800d9da8/attachment.htm>
More information about the Dxspider-support
mailing list