[Dxspider-support] Self-spotting and de-duplication

Wed Jan 15 21:26:07 GMT 2025

Thanks Kin, much appreciated.

However, in spite of having a PhD I still struggle to understand your
description.

So timegranularity defines a window within which spots are considered to
have
the same time, Does this then effectively define the minimum respotting
time?

Or is this the function of dupage? If it is, then I do not see the purpose
of quantizing
time into timegranularity steps.

And how can 600 be 4 minutes? Is time counted in steps of 0.4s?
Does this mean that dupage 3600 is 24 minutes?

And what is the function of qrggranularity? My empirical experiments say
that I need
to move 10kHz away for DXSpider to not consider a new spot a dupe. Not
25kHz.

All I'm trying to do is to set up my node so that

1. a new spot of the same station on the same QRG after more than 3 minutes
is never considered a dupe
2. a new spot of a station 1kHz away from the previous spot is never
considered a dupe.
3. self spots are never ever blocked

Björn SM7IUN

On Tue, Jan 14, 2025 at 7:35 PM Kin <ea3cv at cronux.net> wrote:

> Björn,
>
>
>
> Analyzing the code, I think the behavior is this:
>
>
>
> A spot is received:
>
>
>
> (1736855040)
>
> PC61^28470.0^A71A^14-Jan-2025^1144Z^Tnx
> ciao^IT9IVN^EA4RCH-5^5.90.149.187^H1^~
>
>
>
> 1736855040 is rounded according to a $timegranularity = 600 (mod 600),
> which is the same as -4 min.
>
>
>
> 1736855040 -> 1736854800
>
> 11:44:00   -> 11:40:00
>
>
>
> And he is left with a hash like this for comparison:
>
> X|A71A|IT9IVN|28475||1736854800|TNXCIAO
>
>
>
> Duration of the duplicate spots --> $Spot::dupage  = 3600
>
> It is added to the dupefile with a time = $main::systime+$dupage, ergo (
> Current time +3600)
>
>
>
> # QRG is normalised using:
>
> $Spot::qrggranularity = 25
>
> 28475 -> Remainder of the division 0, we have --> 28475
>
> Ergo, default 25 kHz
>
>
>
> # Self spotting
>
> Minimum freq above which self spotting is allowed -->  $Spot:: $
> minselfspotqrg = 1240000
>
>
>
> Kin EA3CV
>
>
>
>
>
> *De:* Dxspider-support <dxspider-support-bounces at tobit.co.uk> *En nombre
> de *Björn Ekelund via Dxspider-support
> *Enviado el:* martes, 14 de enero de 2025 9:27
> *Para:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *CC:* Björn Ekelund <bjorn at ekelund.nu>
> *Asunto:* Re: [Dxspider-support] Self-spotting and de-duplication
>
>
>
> What does these numbers mean? Is your re-spotting timer really one hour?
>
>
>
> Do you know what variable controls the minimum frequency change for a spot
> to
>
> be considered not a dupe? Today it seems to be around 5kHz which is far,
> far too much.
>
>
>
> Björn SM7IUN
>
>
>
> On Tue, Jan 14, 2025 at 8:28 AM Kin via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
> It is advisable not to touch Stop.pm.
>
> It would be enough to do:
>
>
>
> set/var $Spot::dupage = XXXX
>
> set/var $Spot::timegranularity = ZZZZ
>
>
>
> The default values I have are:
>
>
>
> $Spot::dupage = 3600
>
> $Spot::timegranularity = 600
>
>
>
> Regards,
>
>
>
> Kin EA3CV
>
>
>
>
>
> *De:* Dxspider-support <dxspider-support-bounces at tobit.co.uk> *En nombre
> de *Simon Ravnic via Dxspider-support
> *Enviado el:* lunes, 13 de enero de 2025 23:58
> *Para:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *CC:* Simon Ravnič <s53zo at t-2.net>
> *Asunto:* Re: [Dxspider-support] Self-spotting and de-duplication
>
>
>
> There is an easy way to change it locally. In perl/Spot.pm change
>
>
>
> dupage = 120;
>
> timegranularity = 120;
>
>
>
> to allow a dupe spot every 120 seconds.
>
>
>
> The issue that remains is that such post is visible only on the original
> node. It appears that it gets deduped on your partner nodes and is
> therefore not propagated to the network.
>
>
>
> With the new contesting rules where more and more of them allow
> self-spotting I believe the network should accommodate to this fact.
>
>
>
> 73
>
> Simon, S53ZO
>
> S50DXS sysop
>
>
>
> On 13 Jan 2025, at 22:05, IZ2LSC via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
>
>
> Back to the main topic,
>
> Based on tests I did in the past, the logic behind the deduplication is
> not clear, or at least I cannot understand it.
>
> Sometime ago I already wrote about this topic in the list, but no
> one answered.
>
> I have seen many time spots classified as dupe even if a very long time
> has passed from last spot.
>
> I also tried to change some parameters on the cluster configuration, but
> with no luck.
>
> I end up by clearing on my cluster the dupe file every hour, to make the
> deduplication less aggressive.
>
>
>
> 73
>
> Andrea, IZ2LSC
>
> -->
>
>
>
>
>
> Il giorno lun 13 gen 2025 alle ore 19:37 Björn Ekelund via
> Dxspider-support <dxspider-support at tobit.co.uk> ha scritto:
>
> Don't worry, my toes may be large but they can handle a good stepping.
>
>
>
> When I put ham radio aside around 1990 for family and career reasons, DX
> clusters did
>
> not really exist so when I returned in 2016 I had already missed the early
> days of clusters
>
> where all spots were made by humans and the terminal bell went off every
> time a spot arrived.
>
>
>
> I can understand there may be hams wanting to stay with that.
>
> I enjoy CW for pretty much the same nostalgic reasons.
>
>
>
> I'm of course perfectly fine with there being cluster nodes set up and
> optimized
>
> for this type of old school usage. As long the cluster software does not
> enforce it
>
> or prevents more modern ways.
>
>
>
> However, the way I see it, a casual user actually has ever more reason to
> embrace a
>
> more modern cluster usage. The casual user does not spend endless hours by
> the radio
>
> so he should have good reasons to maximize his productivity and/or fun.
>
>
>
> Having a software (like SpotCollector or HRDLog) monitor the cluster and
> present
>
> maps or lists or graphs with needed DXCC band slots, friends, event
> stations, etc. is a great
>
> way to do exactly this.
>
>
>
> And when you have a piece of software to collect your spots, information
> overload is no longer
>
> an issue and you can tap into as many information sources as you like;
> RBN, PSKReporter, IRC, etc.
>
> To get even more productive and have more fun.
>
>
>
> But there are of course different definitions of fun. People do a lot of
> things that are difficult or
>
> uncomfortable for fun.
>
>
>
> So perhaps I should have used a bit more respectful language when
> describing the practices of
>
> the early days of clusters.
>
>
>
> Björn SM7IUN
>
>
>
> On Mon, Jan 13, 2025 at 6:00 PM Rene Olsen via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
> Hi.
>
> To be honest I am not quite sure how to respond to this, without stepping
> on someones toes.
> But here it goes.
>
> You are aware that the DX-Cluster system is also used by casual users, who
> couldn't care
> less about bandmaps, contests and what else?
>
> DX-Clusters have been used just fine for 30+ years, without the use of RBN.
>
> > A cluster node without skimmer spots is of very little value.
>
> That is some statement. So what you are saying, is that all DX-Clusters
> without RBN feed
> might as well close down? I mean since they are of very little value.
>
> RBN is probably a good thing for some. But the user can turn it OFF if the
> user don't want it.
>
> I don't know how much RBN spots are used on various DX-Clusters. I can
> only speak for
> OZ5BBS-7. I have an average of maybe 45-55 users on a daily basis. Not
> many of them have
> enabled the RBN feed.
>
> I doubt that the casual user of a DX-Cluster wants to get a DX spot of
> some W station spotting
> itself 930 times during the CQWW contest.
>
> Its like everything revolves about contesting. Thats not the case.
>
> Many users who use DX-Clusters, couldn't care less about contests.
>
> I am not saying that DX-Clusters shouldn't evolve, and we should be stuck
> at how it was 30
> years ago. But, it should be done with the casual DX-Cluster user in mind
> as well.
>
> I hope I didn't step on too many toes here :-)
>
> Vy 73 de Rene / OZ1LQH
>
>
> On 12 Jan 2025 at 22:51, Björn Ekelund via Dxspider- wrote:
>
> > A cluster node without skimmer spots is of very little value.
> > The reverse beacon network produces about 12 million spots on a
> > big contest weekend. This is consolidated and de-duped into perhaps half
> a
> > million.
> > With this in mind, what makes a few thousand self spots a problem?
> >
> > The days of watching the telnet feed scroll by are long gone.
> > Today computers turn the spot flow into bandmap items and lists of wanted
> > stations.
> > The cluster connection is just a data feed. Not a user interface.
> >
> > Björn SM7IUN
> >
> > On Sun, Jan 12, 2025 at 9:41PM Rene Olsen via Dxspider-support <
> > dxspider-support at tobit.co.uk> wrote:
> >
> > > Hi.
> > >
> > > Does this mean that we can get selfspots every 3 minutes from 5000+
> users
> > > or whatever?
> > >
> > > If thats the case it is just stupid.
> > >
> > > If it is really the case, it makes one wonder if it is even worth
> running
> > > a spider node anymore. It
> > > will totally ruin the entire idea of DX clusters. Which is not to allow
> > > some OZ station or
> > > whatever, to send self spot every 3 minutes 24/7.
> > >
> > > Just my opinion.
> > >
> > > Vy 73 de Rene / OZ1LQH
> > >
> > > On 12 Jan 2025 at 18:54, Björn Ekelund via Dxspider- wrote:
> > >
> > > > Some of the default settings of DXSpider seem optimized for users
> staring
> > > > at the telnet feed.
> > > >
> > > > In my scripts/startup I have added
> > > >
> > > > set/var $RBN::respottime = 180
> > > > set/var $Spot::minselfspotqrg  0
> > > >
> > > > A cluster node should never suppress a self spot. The minimum self
> > > spotting
> > > > periodicity is set
> > > > by the contest's rules and the cluster should never interfere with
> this.
> > > > Typical minimum periods for
> > > > self spotting in contests are between 3 and 10 minutes.
> > > >
> > > > If DXSpider does suppress self spots under certain conditions I would
> > > like
> > > > to know how to disable this.
> > > >
> > > > Björn SM7IUN
> > > >
> > > > On Sun, Jan 12, 2025 at 4:35PM Keith, G6NHU via Dxspider-support <
> > > > dxspider-support at tobit.co.uk> wrote:
> > > >
> > > > > Now that self spotting is allowed for a trial period in all RSGB HF
> > > > > contests, I´ve been asked about self-spotting and submitting
> duplicate
> > > > > spots by a couple of my users.
> > > > >
> > > > > Some time ago I set minselfspotqrg to 0 to allow self spotting but
> > > what is
> > > > > the situation regarding duplicates because I understand some
> logging
> > > > > software can now self spot every few minutes.  I don´t necessarily
> > > agree
> > > > > with this but it´s going to happen more and more.
> > > > >
> > > > > What´s the period during which the cluster will reject a self spot
> as a
> > > > > dupe please and is there a variable that can be tweaked to adjust
> it?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > 73 Keith.
> > > > > _______________________________________________
> > > > > Dxspider-support mailing list
> > > > > Dxspider-support at tobit.co.uk
> > > > > https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Dxspider-support mailing list
> > > Dxspider-support at tobit.co.uk
> > > https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
> > >
> >
>
>
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250115/1131e6a2/attachment-0001.htm>