<div dir="ltr">Hi Lee,<br><div><br></div><div>The way you approach certain problems implies continuous maintenance on your part, does this not generate a great dependence between the algorithm/data and the programmer? <br></div><div>In one of the world contests, does your implementation require a lot of machine resources?</div><div>How much would the increase in resources be from day to day and a global contest?<br></div><div><br></div><div>Kin EA3CV</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El vie., 10 jul. 2020 a las 18:07, Lee Sawkins via Dxspider-support (<<a href="mailto:dxspider-support@tobit.co.uk">dxspider-support@tobit.co.uk</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Dirk<br>
<br>
There are many things that complicate determining which calls are valid and which are not. One is that you only have a limited time to make a determination. During a recent contest there were 8.5 million valid spots in a 48 hour period. Some of the "running" stations only identify themselves once every several minutes. Everyone is in a hurry. So some just work the stations and log the call that was in their band map. They should not do that, but they do. I am sure that few look at the comments field of the spots for validity flags.<br>
<br>
At one time the RBN tried using both AR Cluster and DXSP cluster programs to distribute their skimmer spots. Neither one of these programs could keep up with the massive number of spots. Both of them would crash during contests. Clusters must process >100 spots per second. The RBN are now using a program that was specifically written to just forward spots and not do any error checking at all.<br>
<br>
CT1BOH proposed a solution for removing bad spots. Both the AR Cluster author and myself used many of his suggestions for our cluster software. My software does not allow users to configure the parameters of the algorithms. All my spots either pass or fail. There are no "maybe" spots. AR Cluster is different.<br>
<a href="http://lists.contesting.com/archives/html/CQ-Contest/2013-07/msg00138.html" rel="noreferrer" target="_blank">http://lists.contesting.com/archives/html/CQ-Contest/2013-07/msg00138.html</a><br>
<br>
USA calls and JA calls have specific rules for call signs formats. A call such as JN78SB or JN87FI are invalid format for a JA call sign and are dropped by my software. USA calls not found in the latest FCC data are all dropped. 1x1 USA calls are not issued by the FCC (ie K2A). KG4xx are issued by the US military and not the FCC and therefore not dropped.<br>
<br>
Some active operators send by hand. Their calls are always busted by skimmers. I have a list and drop all these calls.<br>
<br>
Some calls end in numbers and do not have a "/" before the number. These calls are dropped unless they are in a special call list which I continually update. There are many other rules for valid calls.<br>
<br>
Lee VE7CC<br>
<br>
----- Original Message -----<br>
From: Dirk Koopman via Dxspider-support <<a href="mailto:dxspider-support@tobit.co.uk" target="_blank">dxspider-support@tobit.co.uk</a>><br>
To: Lee Sawkins via Dxspider-support <<a href="mailto:dxspider-support@tobit.co.uk" target="_blank">dxspider-support@tobit.co.uk</a>><br>
Cc: Dirk Koopman <<a href="mailto:djk@tobit.co.uk" target="_blank">djk@tobit.co.uk</a>><br>
Sent: Fri, 10 Jul 2020 05:52:02 -0600 (MDT)<br>
Subject: Re: [Dxspider-support] RBN Busted Skimmer Spots<br>
<br>
On 10/07/2020 08:59, Lee Sawkins via Dxspider-support wrote:<br>
> Not to rain on anyone's parade but ... the RBN calls their spot streams "raw". Their spots contain bad calls and frequencies and little is done to remove them. The RBN depends on the clusters to do this. I have seen no mention of how this is being done in the new software.<br>
<br>
I have not used the word "busted". But there is stuff about checking <br>
callsign and prefix validity in RBN.mojo and the Q: value.<br>
<br>
><br>
> There are about 150+ CW Skimmers. Each CW Skimmer is about 99% or so accurate for calls. These skimmers do not all bust the calls the same way. So what you end up with is a much higher error rate of bad to good calls. When working a contest the longer one stays on a band and works stations, the higher the percentage of busted calls there is in the band map. Stay on a band long enough and the band map has more busted calls than good ones. I currently see EV73NS, OE7NS, V73N, VE3NS, V73NI, V73T, V73NA, V73NN and V73NS spotted on the same frequency. Only one of those calls is right.<br>
<br>
This is an area that is being worked on. But I would also observe that <br>
so many variations of the same call implies many spots arriving for a <br>
callsign on CW or RTTY being on (exactly) the same frequency is unlikely <br>
- I have seen variations of reported frequency varying by up to 600hz <br>
(so far). This is one of the motivations for the Q: factor. If one sees <br>
a Q:2 or higher with a smattering of other variations with Q:1 around it <br>
- take the Q:2 and above spot as the likely one. Or as Kin EA5CV says: <br>
use one's ears (seems a bit radical, but perhaps that's just me :-).<br>
<br>
But you are right, a reverse index on frequency is a good idea that I am <br>
also working on. But, by request, I needed to get this version out in <br>
time for the contest.<br>
<br>
10Jul2020@07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE <br>
on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR<br>
10Jul2020@06:40:23 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on <br>
14025 by JJ2VLY @ 0640Z Q:9 route: SK0MMR<br>
10Jul2020@06:43:16 (progress) SPOT: V73NS on 14025.0 @ 0643Z by <br>
ON7PQ(178.118.27.218)@ON0DXK-5 'cq up ...' route: AE5E<br>
10Jul2020@06:43:29 (progress) RBN: SPOT key: 'V73NI|14025' = V73NI on <br>
14025 by KM3T @ 0643Z Q:1 route: SK0MMR<br>
10Jul2020@06:50:44 (progress) SPOT: V73NS on 14025.0 @ 0650Z by <br>
HL1VAU(222.107.121.212)@VE7CC-1 'CW CQ UP' route: VE7CC-1<br>
10Jul2020@06:57:46 (progress) SPOT: V73NS on 14025.0 @ 0657Z by <br>
JA1NVF@JA4PXC-7 '' route: VE7CC-1<br>
10Jul2020@07:11:43 (progress) RBN: SPOT key: 'V73T|14025' = V73T on <br>
14025 by KM3T @ 0711Z Q:1 route: SK0MMR<br>
10Jul2020@07:14:15 (progress) RBN: SPOT key: 'V73N|14025' = V73N on <br>
14025 by NX5M @ 0714Z Q:1 route: SK0MMR<br>
10Jul2020@07:23:12 (progress) RBN: SPOT key: 'V73NA|14025' = V73NA on <br>
14025.2 by VE6AO @ 0722Z Q:1 route: SK0MMR<br>
10Jul2020@07:27:31 (progress) RBN: SPOT key: 'V73NN|14025' = V73NN on <br>
14025 by BA7KW @ 0727Z Q:1 route: SK0MMR<br>
10Jul2020@07:33:23 (progress) RBN: SPOT key: 'V73NS|14026' = V73NS on <br>
14025.6 by KH0/KC0W @ 0733Z Q:1 route: SK0MMR<br>
10Jul2020@07:37:15 (progress) RBN: SPOT key: 'V73NS|14017' = V73NS on <br>
14016.9 by KH0/KC0W @ 0736Z Q:1 route: SK0MMR<br>
10Jul2020@07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE <br>
on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR<br>
10Jul2020@07:38:15 (progress) RBN: SPOT key: 'V73NS|14033' = V73NS on <br>
14033.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR<br>
10Jul2020@07:40:48 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on <br>
14025 by VE2WU @ 0740Z Q:1 route: SK0MMR<br>
<br>
Look at the variation in reported frequency ("<call> on 140xx.y" above), <br>
for starters. Then there is the question of which version do I use. The <br>
obvious choice is either the Q:9 spot on 06:40:23 or combine with . But <br>
then one asks: is it still valid later, and if so, for how long?<br>
<br>
Then there are the word from CT1BOH himself:<br>
<br>
"I think a clarification is needed regarding the so called CT1BOH error free<br>
RBN algorithm<br>
In order to do that let me first put all relevant "boxes" in this contest<br>
eco-system:<br>
<br>
1. Contest signals on the bands<br>
2. SDR receivers that provide two outputs<br>
1. SDR's recording of the audio of all the bands for later playback<br>
2. Call, frequency, time, SNR,.., data from Skimmers to feed<br>
"cluster" networks<br>
3. Cluster networks (RBN, Human feed networks,...)<br>
4. Contest loggers<br>
5. Contesters<br>
<br>
The purpose of CT1BOH error free RBN algorithm is to provide a "Quality<br>
Tag" next to each spot originating from RBN in order to have "later down<br>
the line" a clean band map<br>
My algorithm provides the following tags:<br>
<br>
- Good spot<br>
- Good call, frequency?<br>
- The majority of these spots come from I/Q image problem from<br>
skimmers that inject bad frequency spots into the RBN<br>
- The first and the second spots when a station QSY's to a new<br>
frequency, until the third spot becomes a "Good Spot"<br>
- ?Spot<br>
- Some of these spots come from people answering CQ calls and skimmer<br>
wrongly think they are calling CQ<br>
- The first and the second spots when a station starts a run, until<br>
the third spot becomes a "Good Spot"<br>
- Busted spots<br>
<br>
I will repeat:<br>
The purpose of my algorithm is just to provide a quality tag next to the<br>
RBN spot."<br>
<br>
<a href="http://lists.contesting.com/_cq-contest/2013-07/msg00187.html" rel="noreferrer" target="_blank">http://lists.contesting.com/_cq-contest/2013-07/msg00187.html</a><br>
<br>
Now I have a rather cruder "Qwalitee" tag and I do say that any spot <br>
with a Q:1 is suspect. But I am a) trying to find CT1BOH's actual <br>
algorithm (I know I've seen it somewhere) or b) roll my own "hamming <br>
distance" mechanism. But the question remains: which version of the <br>
callsign + frequency + source does one use as gospel, and for how long <br>
does it remain valid?<br>
<br>
Choosing the wrongly could obliterate the correct one for a long time.<br>
<br>
><br>
> CW and RTTY are bad for being busted. Not so for FT4/FT8, but it still happens. Sometimes grid squares get mistaken for call signs. I see spots for JN87FI in the last few minutes. Frequently SP5xxx calls get shortened to P5xxx. That usually creates a big pile up.<br>
<br>
How does one determine that a potentially valid call (it passes <br>
is_callsign, is_prefix and is_valid country) is not a reasonable spot if <br>
it has a Q: > 1? Think of all the special calls that pop up at the drop <br>
of a hat?<br>
<br>
I am very open to suggestions and help in this area.<br>
<br>
73 Dirk G1TLH<br>
<br>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a><br>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" rel="noreferrer" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
<br>
<br>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a><br>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" rel="noreferrer" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote></div>