[Dxspider-support] Self-spotting and de-duplication

Fri Jan 17 11:55:25 GMT 2025

Morning Andrea,

While waiting for Dirk to confirm the possible bug and its solution, I'm passing you a modified version of DXDupe.pm that fixes the deletion of hashes depending on the $Spot::dupage variable.
It also solves the problem of deleting expired spots.

It should be copied to: /spider/Local and restart the node.

Kin EA3CV

De: Dxspider-support <dxspider-support-bounces at tobit.co.uk> En nombre de IZ2LSC via Dxspider-support
Enviado el: viernes, 17 de enero de 2025 10:18
Para: The DXSpider Support list <dxspider-support at tobit.co.uk>
CC: IZ2LSC <iz2lsc.andrea at gmail.com>
Asunto: Re: [Dxspider-support] Self-spotting and de-duplication

If every time a spot is ignored is also added again to the dupe list, this is a loop. Maybe is this the bug?

1737101764^(pc11) PCPROT: Duplicate Spot  IZ2LSC-97: PC61 7006.0|JF4IQS/1|1737101700|JF4IQS|IZ2LSC-97 ignored

view_dupes JF4IQS/1
|JF4IQS/1|JF4IQS|433060||1737100440|POTAJP1149FMHIK = 17-Jan-2025 0754Z expires 17-Jan-2025 0824Z
|JF4IQS/1|JF4IQS|7005||1737100800|POTAJP1149CWHIK = 17-Jan-2025 0801Z expires 17-Jan-2025 0831Z
|JF4IQS/1|JF4IQS|7005|[-POTA-] JP-114 = 17-Jan-2025 0801Z expires 17-Jan-2025 0831Z
|JF4IQS/1|JI2OJV|7005||1737101580|POTAJP1149CWHIK = 17-Jan-2025 0814Z expires 17-Jan-2025 0844Z
|JF4IQS/1|JI2OJV|7005|[-POTA-] JP-114 = 17-Jan-2025 0814Z expires 17-Jan-2025 0844Z
|JF4IQS/1|JF4IQS|7005||1737101700|POTAJP1149CWHIK = 17-Jan-2025 0816Z expires 17-Jan-2025 0846Z

-->

On Thu, Jan 16, 2025 at 11:01 PM IZ2LSC <iz2lsc.andrea at gmail.com <mailto:iz2lsc.andrea at gmail.com> > wrote:
Kin, 

when the hash is expired, is it removed from the file?
I can see hash already expired and still present in the file.
Not sure if this is expected.

Andrea

-->

On Thu, Jan 16, 2025 at 8:12 PM Björn Ekelund via Dxspider-support <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> > wrote:
Thanks. For human spots it makes perfect sense to include the comment.

Björn SM7IUN

On Thu, Jan 16, 2025 at 7:06 PM Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> > wrote:
Björn,

All this is for non-RBN spots.
I don't remember now the treatment of RBN spots.

Kin

De: Björn Ekelund <bjorn at ekelund.nu <mailto:bjorn at ekelund.nu> > 
Enviado el: jueves, 16 de enero de 2025 18:26
Para: Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> >
CC: The DXSpider Support list <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> >
Asunto: Re: [Dxspider-support] Self-spotting and de-duplication

Kin, 

this was very clear and very helpful. 
Thank you for spending the time to do it.
I am sure others appreciate it too.

In my opinion, all comments should be ignored when determining dupes for skimmer spots. 

Björn SM7IUN

On Thu, Jan 16, 2025 at 10:28 AM Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> > wrote:
Morning Björn,

>So timegranularity defines a window within which spots are considered to have 
>the same time, Does this then effectively define the minimum respotting time?

Yes, timegranularity effectively defines the minimum respotting time. It determines the size of the time window within which spots are considered to have the same timestamp. For example, if timegranularity is set to 600 seconds (10 minutes), any spots reported within the same 10-minute interval will be grouped together, effectively preventing additional spots from being considered separate unless they occur in a different interval.

>Or is this the function of dupage? If it is, then I do not see the purpose of quantizing 
>time into timegranularity steps.

timegranularity is about standardizing time representation, while dupage enforces the minimum time between duplicate spots.

>And how can 600 be 4 minutes? Is time counted in steps of 0.4s?
>Does this mean that dupage 3600 is 24 minutes?

>And what is the function of qrggranularity? My empirical experiments say that I need 
>to move 10kHz away for DXSpider to not consider a new spot a dupe. Not 25kHz.

I hope that the following examples will clarify how it works.

>All I'm trying to do is to set up my node so that 

>1. a new spot of the same station on the same QRG after more than 3 minutes is never considered a dupe

$Spot::timegranularity = 60
$Spot::dupage = 180

You should know that a duplicate is the sum of the fields spotted+spotter+qrg+time+comments, which will generate a hash like this: |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM. For this reason, just using spotted+qrg+time is not sufficient to determine duplication.

>2. a new spot of a station 1kHz away from the previous spot is never considered a dupe.

$Spot::qrggranularity = 1

>3. self spots are never ever blocked

$Spot::minselfspotqrg = 0

Example

* Variables

$Spot::timegranularity = 600    --> 10 min
$Spot::qrggranularity = 25      --> 25 kHz
$Spot::dupage = 3600            --> 1 h

* Spot

DX de IK2JTS:     7065.0  IZ0ARL       DTMBA I1823RM                    0731Z (1737012660)

Adjusts to the nearest multiple (either downward or upward)

1737012660/600 = 2895021.1 --> 2895021 (round to the nearest integer)
2895021*600 = 1737012600

7065.0/25 = 282.6 --> 283 (round to the nearest integer)
283×25 = 7075

* dupefile

|IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM = 16-Jan-2025 0731Z expires 16-Jan-2025 0831Z

This means that time is fixed in 10-minute intervals, and frequency is fixed in 25 kHz steps.

* Comparison with different values for the same spot:

DX de IK2JTS:     7065.0  IZ0ARL       DTMBA #I-1823RM                  0735Z (1737012900)

Scenario 1:

$Spot::timegranularity = 600
$Spot::qrggranularity = 25
$Spot::dupage = 3600

|IZ0ARL|IT9JZK|7075||1737013200|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0835Z
                       (0740Z)

Scenario 2:

$Spot::timegranularity = 60
$Spot::qrggranularity = 1
$Spot::dupage = 300

|IZ0ARL|IT9JZK|7065||1737012900|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0740Z
                      (0735Z)

I'm not sure if I’ve managed to explain myself clearly this time. This is my interpretation after analyzing the code, from someone who is not an expert. I think the person who can help us best is Dirk, the creator of this. And sorry because my English is terrible

Kin EA3CV

_______________________________________________
Dxspider-support mailing list
Dxspider-support at tobit.co.uk <mailto:Dxspider-support at tobit.co.uk> 
https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250117/3c7710d7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DXDupe.pm
Type: application/octet-stream
Size: 2839 bytes
Desc: not available
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250117/3c7710d7/attachment-0001.obj>