[Dxspider-support] RBN Busted Skimmer Spots

Fri Jul 10 20:36:33 CEST 2020

Hi Lee,

The way you approach certain problems implies continuous maintenance on
your part, does this not generate a great dependence between the
algorithm/data and the programmer?
In one of the world contests, does your implementation require a lot of
machine resources?
How much would the increase in resources be from day to day and a global
contest?

Kin EA3CV

El vie., 10 jul. 2020 a las 18:07, Lee Sawkins via Dxspider-support (<
dxspider-support at tobit.co.uk>) escribió:

> Hi Dirk
>
> There are many things that complicate determining which calls are valid
> and which are not.   One is that you only have a limited time to make a
> determination.  During a recent contest there were 8.5 million valid spots
> in a 48 hour period.  Some of the "running" stations only identify
> themselves once every several minutes.  Everyone is in a hurry.  So some
> just work the stations and log the call that was in their band map.  They
> should not do that, but they do.  I am sure that few look at the comments
> field of the spots for validity flags.
>
> At one time the RBN tried using both AR Cluster and DXSP cluster programs
> to distribute their skimmer spots.  Neither one of these programs could
> keep up with the massive number of spots.  Both of them would crash during
> contests.   Clusters must process >100 spots per second.  The RBN are now
> using a program that was specifically written to just forward spots and not
> do any error checking at all.
>
> CT1BOH proposed a solution for removing bad spots.  Both the AR Cluster
> author and myself used many of his suggestions for our cluster software.
> My software does not allow users to configure the parameters of the
> algorithms.  All my spots either pass or fail.  There are no "maybe"
> spots.  AR Cluster is different.
> http://lists.contesting.com/archives/html/CQ-Contest/2013-07/msg00138.html
>
> USA calls and JA calls have specific rules for call signs formats.  A call
> such as JN78SB or JN87FI are invalid format for a JA call sign and are
> dropped by my software.  USA calls not found in the latest FCC data are all
> dropped.  1x1 USA calls are not issued by the FCC (ie K2A).  KG4xx are
> issued by the US military and not the FCC and therefore not dropped.
>
> Some active operators send by hand.  Their calls are always busted by
> skimmers.  I have a list and drop all these calls.
>
> Some calls end in numbers and do not have a "/" before the number.  These
> calls are dropped unless they are in a special call list which I
> continually update.  There are many other rules for valid calls.
>
> Lee VE7CC
>
> ----- Original Message -----
> From: Dirk Koopman via Dxspider-support <dxspider-support at tobit.co.uk>
> To: Lee Sawkins via Dxspider-support <dxspider-support at tobit.co.uk>
> Cc: Dirk Koopman <djk at tobit.co.uk>
> Sent: Fri, 10 Jul 2020 05:52:02 -0600 (MDT)
> Subject: Re: [Dxspider-support] RBN Busted Skimmer Spots
>
> On 10/07/2020 08:59, Lee Sawkins via Dxspider-support wrote:
> > Not to rain on anyone's parade but ... the RBN calls their spot streams
> "raw".  Their spots contain bad calls and frequencies and little is done to
> remove them.  The RBN depends on the clusters to do this.  I have seen no
> mention of how this is being done in the new software.
>
> I have not used the word "busted". But there is stuff about checking
> callsign and prefix validity in RBN.mojo and the Q: value.
>
> >
> > There are about 150+ CW Skimmers.   Each CW Skimmer is about 99% or so
> accurate for calls.   These skimmers do not all bust the calls the same
> way.  So what you end up with is a much higher error rate of bad to good
> calls.  When working a contest the longer one stays on a band and works
> stations, the higher the percentage of busted calls there is in the band
> map.  Stay on a band long enough and the band map has more busted calls
> than good ones.  I currently see EV73NS, OE7NS, V73N, VE3NS, V73NI, V73T,
> V73NA, V73NN and V73NS spotted on the same frequency.  Only one of those
> calls is right.
>
> This is an area that is being worked on. But I would also observe that
> so many variations of the same call implies many spots arriving for a
> callsign on CW or RTTY being on (exactly) the same frequency is unlikely
> - I have seen variations of reported frequency varying by up to 600hz
> (so far). This is one of the motivations for the Q: factor. If one sees
> a Q:2 or higher with a smattering of other variations with Q:1 around it
> - take the Q:2 and above spot as the likely one. Or as Kin EA5CV says:
> use one's ears (seems a bit radical, but perhaps that's just me :-).
>
> But you are right, a reverse index on frequency is a good idea that I am
> also working on. But, by request, I needed to get this version out in
> time for the contest.
>
> 10Jul2020 at 07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE
> on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
> 10Jul2020 at 06:40:23 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on
> 14025 by JJ2VLY @ 0640Z Q:9 route: SK0MMR
> 10Jul2020 at 06:43:16 (progress) SPOT: V73NS on 14025.0 @ 0643Z by
> ON7PQ(178.118.27.218)@ON0DXK-5 'cq up ...' route: AE5E
> 10Jul2020 at 06:43:29 (progress) RBN: SPOT key: 'V73NI|14025' = V73NI on
> 14025 by KM3T @ 0643Z Q:1 route: SK0MMR
> 10Jul2020 at 06:50:44 (progress) SPOT: V73NS on 14025.0 @ 0650Z by
> HL1VAU(222.107.121.212)@VE7CC-1 'CW CQ UP' route: VE7CC-1
> 10Jul2020 at 06:57:46 (progress) SPOT: V73NS on 14025.0 @ 0657Z by
> JA1NVF at JA4PXC-7 '' route: VE7CC-1
> 10Jul2020 at 07:11:43 (progress) RBN: SPOT key: 'V73T|14025' = V73T on
> 14025 by KM3T @ 0711Z Q:1 route: SK0MMR
> 10Jul2020 at 07:14:15 (progress) RBN: SPOT key: 'V73N|14025' = V73N on
> 14025 by NX5M @ 0714Z Q:1 route: SK0MMR
> 10Jul2020 at 07:23:12 (progress) RBN: SPOT key: 'V73NA|14025' = V73NA on
> 14025.2 by VE6AO @ 0722Z Q:1 route: SK0MMR
> 10Jul2020 at 07:27:31 (progress) RBN: SPOT key: 'V73NN|14025' = V73NN on
> 14025 by BA7KW @ 0727Z Q:1 route: SK0MMR
> 10Jul2020 at 07:33:23 (progress) RBN: SPOT key: 'V73NS|14026' = V73NS on
> 14025.6 by KH0/KC0W @ 0733Z Q:1 route: SK0MMR
> 10Jul2020 at 07:37:15 (progress) RBN: SPOT key: 'V73NS|14017' = V73NS on
> 14016.9 by KH0/KC0W @ 0736Z Q:1 route: SK0MMR
> 10Jul2020 at 07:38:14 (progress) RBN: SPOT key: 'EV73NSE|14025' = EV73NSE
> on 14025.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
> 10Jul2020 at 07:38:15 (progress) RBN: SPOT key: 'V73NS|14033' = V73NS on
> 14033.3 by KH0/KC0W @ 0738Z Q:1 route: SK0MMR
> 10Jul2020 at 07:40:48 (progress) RBN: SPOT key: 'V73NS|14025' = V73NS on
> 14025 by VE2WU @ 0740Z Q:1 route: SK0MMR
>
> Look at the variation in reported frequency ("<call> on 140xx.y" above),
> for starters. Then there is the question of which version do I use. The
> obvious choice is either the Q:9 spot on 06:40:23 or combine with . But
> then one asks: is it still valid later, and if so, for how long?
>
> Then there are the word from CT1BOH himself:
>
> "I think a clarification is needed regarding the so called CT1BOH error
> free
> RBN algorithm
> In order to do that let me first put all relevant "boxes" in this contest
> eco-system:
>
>     1. Contest signals on the bands
>     2. SDR receivers that provide two outputs
>        1. SDR's recording of the audio of all the bands for later playback
>        2. Call, frequency, time, SNR,.., data from Skimmers to feed
>        "cluster" networks
>     3. Cluster networks (RBN, Human feed networks,...)
>     4. Contest loggers
>     5. Contesters
>
> The purpose of CT1BOH error free RBN algorithm is to provide a "Quality
> Tag" next to each spot originating from RBN in order to have "later down
> the line" a clean band map
> My algorithm provides the following tags:
>
>     - Good spot
>     - Good call, frequency?
>        - The majority of these spots come from I/Q image problem from
>        skimmers that inject bad frequency spots into the RBN
>        - The first and the second spots when a station QSY's to a new
>        frequency, until the third spot becomes a "Good Spot"
>     - ?Spot
>        - Some of these spots come from people answering CQ calls and
> skimmer
>        wrongly think they are calling CQ
>        - The first and the second spots when a station starts a run, until
>        the third spot becomes a "Good Spot"
>     - Busted spots
>
> I will repeat:
> The purpose of my algorithm is just to provide a quality tag next to the
> RBN spot."
>
> http://lists.contesting.com/_cq-contest/2013-07/msg00187.html
>
> Now I have a rather cruder "Qwalitee" tag and I do say that any spot
> with a Q:1 is suspect. But I am a) trying to find CT1BOH's actual
> algorithm (I know I've seen it somewhere) or b) roll my own "hamming
> distance" mechanism. But the question remains: which version of the
> callsign + frequency + source does one use as gospel, and for how long
> does it remain valid?
>
> Choosing the wrongly could obliterate the correct one for a long time.
>
> >
> > CW and RTTY are bad for being busted.  Not so for FT4/FT8, but it still
> happens.  Sometimes grid squares get mistaken for call signs.  I see spots
> for JN87FI in the last few minutes.  Frequently SP5xxx calls get shortened
> to P5xxx.  That usually creates a big pile up.
>
> How does one determine that a potentially valid call (it passes
> is_callsign, is_prefix and is_valid country) is not a reasonable spot if
> it has a Q: > 1? Think of all the special calls that pop up at the drop
> of a hat?
>
> I am very open to suggestions and help in this area.
>
> 73 Dirk G1TLH
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20200710/10f83869/attachment-0001.htm>