[Dxspider-support] RBN Busted Skimmer Spots

Lee Sawkins ve7cc at shaw.ca
Fri Jul 10 18:06:16 CEST 2020 

Hi Dirk

There are many things that complicate determining which calls are valid and which are not.   One is that you only have a limited time to make a determination.  During a recent contest there were 8.5 million valid spots in a 48 hour period.  Some of the "running" stations only identify themselves once every several minutes.  Everyone is in a hurry.  So some just work the stations and log the call that was in their band map.  They should not do that, but they do.  I am sure that few look at the comments field of the spots for validity flags.

At one time the RBN tried using both AR Cluster and DXSP cluster programs to distribute their skimmer spots.  Neither one of these programs could keep up with the massive number of spots.  Both of them would crash during contests.   Clusters must process >100 spots per second.  The RBN are now using a program that was specifically written to just forward spots and not do any error checking at all.

CT1BOH proposed a solution for removing bad spots.  Both the AR Cluster author and myself used many of his suggestions for our cluster software.  My software does not allow users to configure the parameters of the algorithms.  All my spots either pass or fail.  There are no "maybe" spots.  AR Cluster is different.
http://lists.contesting.com/archives/html/CQ-Contest/2013-07/msg00138.html

USA calls and JA calls have specific rules for call signs formats.  A call such as JN78SB or JN87FI are invalid format for a JA call sign and are dropped by my software.  USA calls not found in the latest FCC data are all dropped.  1x1 USA calls are not issued by the FCC (ie K2A).  KG4xx are issued by the US military and not the FCC and therefore not dropped.

Some active operators send by hand.  Their calls are always busted by skimmers.  I have a list and drop all these calls.

Some calls end in numbers and do not have a "/" before the number.  These calls are dropped unless they are in a special call list which I continually update.  There are many other rules for valid calls.

Lee VE7CC

----- Original Message -----
From: Dirk Koopman via Dxspider-support <dxspider-support at tobit.co.uk>
To: Lee Sawkins via Dxspider-support <dxspider-support at tobit.co.uk>
Cc: Dirk Koopman <djk at tobit.co.uk>
Sent: Fri, 10 Jul 2020 05:52:02 -0600 (MDT)
Subject: Re: [Dxspider-support] RBN Busted Skimmer Spots

On 10/07/2020 08:59, Lee Sawkins via Dxspider-support wrote:
> Not to rain on anyone's parade but ... the RBN calls their spot streams "raw".  Their spots contain bad calls and frequencies and little is done to remove them.  The RBN depends on the clusters to do this.  I have seen no mention of how this is being done in the new software.

I have not used the word "busted". But there is stuff about checking 
callsign and prefix validity in RBN.mojo and the Q: value.

>
> There are about 150+ CW Skimmers.   Each CW Skimmer is about 99% or so accurate for calls.   These skimmers do not all bust the calls the same way.  So what you end up with is a much higher error rate of bad to good calls.  When working a contest the longer one stays on a band and works stations, the higher the percentage of busted calls there is in the band map.  Stay on a band long enough and the band map has more busted calls than good ones.  I currently see EV73NS, OE7NS, V73N, VE3NS, V73NI, V73T, V73NA, V73NN and V73NS spotted on the same frequency.  Only one of those calls is right.

This is an area that is being worked on. But I would also observe that 
so many variations of the same call implies many spots arriving for a 
callsign on CW or RTTY being on (exactly) the same frequency is unlikely 
- I have seen variations of reported frequency varying by up to 600hz 
(so far). This is one of the motivations for the Q: factor. If one sees 
a Q:2 or higher with a smattering of other variations with Q:1 around it 
- take the Q:2 and above spot as the likely one. Or as Kin EA5CV says: 
use one's ears (seems a bit radical, but perhaps that's just me :-).

But you are right, a reverse index on frequency is a good idea that I am 
also working on. But, by request, I needed to get this version out in 
time for the contest.

10Jul2020 at 07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE 
on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
10Jul2020 at 06:40:23 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on 
14025 by JJ2VLY @ 0640Z Q:9 route: SK0MMR
10Jul2020 at 06:43:16 (progress) SPOT: V73NS on 14025.0 @ 0643Z by 
ON7PQ(178.118.27.218)@ON0DXK-5 'cq up ...' route: AE5E
10Jul2020 at 06:43:29 (progress) RBN: SPOT key: 'V73NI|14025' = V73NI on 
14025 by KM3T @ 0643Z Q:1 route: SK0MMR
10Jul2020 at 06:50:44 (progress) SPOT: V73NS on 14025.0 @ 0650Z by 
HL1VAU(222.107.121.212)@VE7CC-1 'CW CQ UP' route: VE7CC-1
10Jul2020 at 06:57:46 (progress) SPOT: V73NS on 14025.0 @ 0657Z by 
JA1NVF at JA4PXC-7 '' route: VE7CC-1
10Jul2020 at 07:11:43 (progress) RBN: SPOT key: 'V73T|14025' = V73T on 
14025 by KM3T @ 0711Z Q:1 route: SK0MMR
10Jul2020 at 07:14:15 (progress) RBN: SPOT key: 'V73N|14025' = V73N on 
14025 by NX5M @ 0714Z Q:1 route: SK0MMR
10Jul2020 at 07:23:12 (progress) RBN: SPOT key: 'V73NA|14025' = V73NA on 
14025.2 by VE6AO @ 0722Z Q:1 route: SK0MMR
10Jul2020 at 07:27:31 (progress) RBN: SPOT key: 'V73NN|14025' = V73NN on 
14025 by BA7KW @ 0727Z Q:1 route: SK0MMR
10Jul2020 at 07:33:23 (progress) RBN: SPOT key: 'V73NS|14026' = V73NS on 
14025.6 by KH0/KC0W @ 0733Z Q:1 route: SK0MMR
10Jul2020 at 07:37:15 (progress) RBN: SPOT key: 'V73NS|14017' = V73NS on 
14016.9 by KH0/KC0W @ 0736Z Q:1 route: SK0MMR
10Jul2020 at 07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE 
on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
10Jul2020 at 07:38:15 (progress) RBN: SPOT key: 'V73NS|14033' = V73NS on 
14033.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
10Jul2020 at 07:40:48 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on 
14025 by VE2WU @ 0740Z Q:1 route: SK0MMR

Look at the variation in reported frequency ("<call> on 140xx.y" above), 
for starters. Then there is the question of which version do I use. The 
obvious choice is either the Q:9 spot on 06:40:23 or combine with . But 
then one asks: is it still valid later, and if so, for how long?

Then there are the word from CT1BOH himself:

"I think a clarification is needed regarding the so called CT1BOH error free
RBN algorithm
In order to do that let me first put all relevant "boxes" in this contest
eco-system:

    1. Contest signals on the bands
    2. SDR receivers that provide two outputs
       1. SDR's recording of the audio of all the bands for later playback
       2. Call, frequency, time, SNR,.., data from Skimmers to feed
       "cluster" networks
    3. Cluster networks (RBN, Human feed networks,...)
    4. Contest loggers
    5. Contesters

The purpose of CT1BOH error free RBN algorithm is to provide a "Quality
Tag" next to each spot originating from RBN in order to have "later down
the line" a clean band map
My algorithm provides the following tags:

    - Good spot
    - Good call, frequency?
       - The majority of these spots come from I/Q image problem from
       skimmers that inject bad frequency spots into the RBN
       - The first and the second spots when a station QSY's to a new
       frequency, until the third spot becomes a "Good Spot"
    - ?Spot
       - Some of these spots come from people answering CQ calls and skimmer
       wrongly think they are calling CQ
       - The first and the second spots when a station starts a run, until
       the third spot becomes a "Good Spot"
    - Busted spots

I will repeat:
The purpose of my algorithm is just to provide a quality tag next to the
RBN spot."

http://lists.contesting.com/_cq-contest/2013-07/msg00187.html

Now I have a rather cruder "Qwalitee" tag and I do say that any spot 
with a Q:1 is suspect. But I am a) trying to find CT1BOH's actual 
algorithm (I know I've seen it somewhere) or b) roll my own "hamming 
distance" mechanism. But the question remains: which version of the 
callsign + frequency + source does one use as gospel, and for how long 
does it remain valid?

Choosing the wrongly could obliterate the correct one for a long time.

>
> CW and RTTY are bad for being busted.  Not so for FT4/FT8, but it still happens.  Sometimes grid squares get mistaken for call signs.  I see spots for JN87FI in the last few minutes.  Frequently SP5xxx calls get shortened to P5xxx.  That usually creates a big pile up.

How does one determine that a potentially valid call (it passes 
is_callsign, is_prefix and is_valid country) is not a reasonable spot if 
it has a Q: > 1? Think of all the special calls that pop up at the drop 
of a hat?

I am very open to suggestions and help in this area.

73 Dirk G1TLH

_______________________________________________
Dxspider-support mailing list
Dxspider-support at tobit.co.uk
https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support

More information about the Dxspider-support mailing list