[Dxspider-support] Self-spotting and de-duplication
Keith, G6NHU
g6nhu at me.com
Sun Jan 19 17:03:37 GMT 2025
I’ve had a couple of users ask me about this because yesterday was the RSGB AFS contest which now allows self spotting for the first time.
All their spots were accepted at my node but sadly other nodes didn’t take them all, I assume due to de-duplication settings.
So until everyone is unified, self spots at five minute intervals won’t propagate anyway.
73 Keith
On 17 Jan 2025 at 13:31 +0000, IZ2LSC via Dxspider-support <dxspider-support at tobit.co.uk>, wrote:
> Thanks Kin,
> I will try it on my backup (isolated) cluster node.
>
> 73
> Andrea
> iz2lsc
>
> -->
>
>
> On Fri, Jan 17, 2025 at 12:55 PM Kin <ea3cv at cronux.net> wrote:
> > Morning Andrea,
> >
> > While waiting for Dirk to confirm the possible bug and its solution, I'm passing you a modified version of DXDupe.pm that fixes the deletion of hashes depending on the $Spot::dupage variable.
> > It also solves the problem of deleting expired spots.
> >
> > It should be copied to: /spider/Local and restart the node.
> >
> > Kin EA3CV
> >
> > De: Dxspider-support <dxspider-support-bounces at tobit.co.uk> En nombre de IZ2LSC via Dxspider-support
> > Enviado el: viernes, 17 de enero de 2025 10:18
> > Para: The DXSpider Support list <dxspider-support at tobit.co.uk>
> > CC: IZ2LSC <iz2lsc.andrea at gmail.com>
> > Asunto: Re: [Dxspider-support] Self-spotting and de-duplication
> >
> > If every time a spot is ignored is also added again to the dupe list, this is a loop. Maybe is this the bug?
> >
> > 1737101764^(pc11) PCPROT: Duplicate Spot IZ2LSC-97: PC61 7006.0|JF4IQS/1|1737101700|JF4IQS|IZ2LSC-97 ignored
> >
> > view_dupes JF4IQS/1
> > |JF4IQS/1|JF4IQS|433060||1737100440|POTAJP1149FMHIK = 17-Jan-2025 0754Z expires 17-Jan-2025 0824Z
> > |JF4IQS/1|JF4IQS|7005||1737100800|POTAJP1149CWHIK = 17-Jan-2025 0801Z expires 17-Jan-2025 0831Z
> > |JF4IQS/1|JF4IQS|7005|[-POTA-] JP-114 = 17-Jan-2025 0801Z expires 17-Jan-2025 0831Z
> > |JF4IQS/1|JI2OJV|7005||1737101580|POTAJP1149CWHIK = 17-Jan-2025 0814Z expires 17-Jan-2025 0844Z
> > |JF4IQS/1|JI2OJV|7005|[-POTA-] JP-114 = 17-Jan-2025 0814Z expires 17-Jan-2025 0844Z
> > |JF4IQS/1|JF4IQS|7005||1737101700|POTAJP1149CWHIK = 17-Jan-2025 0816Z expires 17-Jan-2025 0846Z
> >
> > -->
> >
> >
> > On Thu, Jan 16, 2025 at 11:01 PM IZ2LSC <iz2lsc.andrea at gmail.com> wrote:
> > > Kin,
> > > when the hash is expired, is it removed from the file?
> > > I can see hash already expired and still present in the file.
> > > Not sure if this is expected.
> > >
> > > Andrea
> > >
> > > -->
> > >
> > >
> > > On Thu, Jan 16, 2025 at 8:12 PM Björn Ekelund via Dxspider-support <dxspider-support at tobit.co.uk> wrote:
> > > > Thanks. For human spots it makes perfect sense to include the comment.
> > > >
> > > > Björn SM7IUN
> > > >
> > > > On Thu, Jan 16, 2025 at 7:06 PM Kin <ea3cv at cronux.net> wrote:
> > > > > Björn,
> > > > >
> > > > > All this is for non-RBN spots.
> > > > > I don't remember now the treatment of RBN spots.
> > > > >
> > > > > Kin
> > > > >
> > > > > De: Björn Ekelund <bjorn at ekelund.nu>
> > > > > Enviado el: jueves, 16 de enero de 2025 18:26
> > > > > Para: Kin <ea3cv at cronux.net>
> > > > > CC: The DXSpider Support list <dxspider-support at tobit.co.uk>
> > > > > Asunto: Re: [Dxspider-support] Self-spotting and de-duplication
> > > > >
> > > > > Kin,
> > > > >
> > > > > this was very clear and very helpful.
> > > > > Thank you for spending the time to do it.
> > > > > I am sure others appreciate it too.
> > > > >
> > > > > In my opinion, all comments should be ignored when determining dupes for skimmer spots.
> > > > >
> > > > > Björn SM7IUN
> > > > >
> > > > > On Thu, Jan 16, 2025 at 10:28 AM Kin <ea3cv at cronux.net> wrote:
> > > > > > Morning Björn,
> > > > > >
> > > > > > >So timegranularity defines a window within which spots are considered to have
> > > > > > >the same time, Does this then effectively define the minimum respotting time?
> > > > > >
> > > > > > Yes, timegranularity effectively defines the minimum respotting time. It determines the size of the time window within which spots are considered to have the same timestamp. For example, if timegranularity is set to 600 seconds (10 minutes), any spots reported within the same 10-minute interval will be grouped together, effectively preventing additional spots from being considered separate unless they occur in a different interval.
> > > > > >
> > > > > > >Or is this the function of dupage? If it is, then I do not see the purpose of quantizing
> > > > > > >time into timegranularity steps.
> > > > > >
> > > > > > timegranularity is about standardizing time representation, while dupage enforces the minimum time between duplicate spots.
> > > > > >
> > > > > > >And how can 600 be 4 minutes? Is time counted in steps of 0.4s?
> > > > > > >Does this mean that dupage 3600 is 24 minutes?
> > > > > >
> > > > > > >And what is the function of qrggranularity? My empirical experiments say that I need
> > > > > > >to move 10kHz away for DXSpider to not consider a new spot a dupe. Not 25kHz.
> > > > > >
> > > > > > I hope that the following examples will clarify how it works.
> > > > > >
> > > > > > >All I'm trying to do is to set up my node so that
> > > > > >
> > > > > > >1. a new spot of the same station on the same QRG after more than 3 minutes is never considered a dupe
> > > > > >
> > > > > > $Spot::timegranularity = 60
> > > > > > $Spot::dupage = 180
> > > > > >
> > > > > > You should know that a duplicate is the sum of the fields spotted+spotter+qrg+time+comments, which will generate a hash like this: |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM. For this reason, just using spotted+qrg+time is not sufficient to determine duplication.
> > > > > >
> > > > > > >2. a new spot of a station 1kHz away from the previous spot is never considered a dupe.
> > > > > >
> > > > > > $Spot::qrggranularity = 1
> > > > > >
> > > > > > >3. self spots are never ever blocked
> > > > > >
> > > > > > $Spot::minselfspotqrg = 0
> > > > > >
> > > > > >
> > > > > > Example
> > > > > >
> > > > > > * Variables
> > > > > >
> > > > > > $Spot::timegranularity = 600 --> 10 min
> > > > > > $Spot::qrggranularity = 25 --> 25 kHz
> > > > > > $Spot::dupage = 3600 --> 1 h
> > > > > >
> > > > > > * Spot
> > > > > >
> > > > > > DX de IK2JTS: 7065.0 IZ0ARL DTMBA I1823RM 0731Z (1737012660)
> > > > > >
> > > > > >
> > > > > > Adjusts to the nearest multiple (either downward or upward)
> > > > > >
> > > > > > 1737012660/600 = 2895021.1 --> 2895021 (round to the nearest integer)
> > > > > > 2895021*600 = 1737012600
> > > > > >
> > > > > > 7065.0/25 = 282.6 --> 283 (round to the nearest integer)
> > > > > > 283×25 = 7075
> > > > > >
> > > > > > * dupefile
> > > > > >
> > > > > > |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM = 16-Jan-2025 0731Z expires 16-Jan-2025 0831Z
> > > > > >
> > > > > > This means that time is fixed in 10-minute intervals, and frequency is fixed in 25 kHz steps.
> > > > > >
> > > > > >
> > > > > > * Comparison with different values for the same spot:
> > > > > >
> > > > > > DX de IK2JTS: 7065.0 IZ0ARL DTMBA #I-1823RM 0735Z (1737012900)
> > > > > >
> > > > > > Scenario 1:
> > > > > >
> > > > > > $Spot::timegranularity = 600
> > > > > > $Spot::qrggranularity = 25
> > > > > > $Spot::dupage = 3600
> > > > > >
> > > > > > |IZ0ARL|IT9JZK|7075||1737013200|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0835Z
> > > > > > (0740Z)
> > > > > >
> > > > > > Scenario 2:
> > > > > >
> > > > > > $Spot::timegranularity = 60
> > > > > > $Spot::qrggranularity = 1
> > > > > > $Spot::dupage = 300
> > > > > >
> > > > > > |IZ0ARL|IT9JZK|7065||1737012900|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0740Z
> > > > > > (0735Z)
> > > > > >
> > > > > >
> > > > > > I'm not sure if I’ve managed to explain myself clearly this time. This is my interpretation after analyzing the code, from someone who is not an expert. I think the person who can help us best is Dirk, the creator of this. And sorry because my English is terrible
> > > > > >
> > > > > > Kin EA3CV
> > > > > >
> > > > _______________________________________________
> > > > Dxspider-support mailing list
> > > > Dxspider-support at tobit.co.uk
> > > > https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250119/cd40b167/attachment-0001.htm>
More information about the Dxspider-support
mailing list