[Dxspider-support] Self-spotting and de-duplication

Thu Jan 16 18:04:26 GMT 2025

This is how I see it,

We choose one spot:

DX de DJ4EL:     24930.0  TX7N         up, cqing           1732Z  (1737048720)

view_dupe TX7N

...
|TX7N|WB7CAT|24925||1737048000|FINALLYHEARDMEG = 16-Jan-2025 1724Z expires 16-Jan-2025 1824Z
|TX7N|WB7CAT|24925|FINALLY HEARD M = 16-Jan-2025 1724Z expires 16-Jan-2025 1824Z
|TX7N|M7FTM|21275||1737048600| = 16-Jan-2025 1726Z expires 16-Jan-2025 1826Z
|TX7N|VE7SNC|24925||1737048600|5UP = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z
|TX7N|VE7SNC|24925|5 UP = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z
|TX7N|DJ4EL|24925|UP, CQING = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z
|TX7N|DJ4EL|24925||1737048600|UPCQING = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z
|TX7N|VE7SNC|21275||1737048600|5UP = 16-Jan-2025 1733Z expires 16-Jan-2025 1833Z
|TX7N|VE7SNC|21275|5 UP = 16-Jan-2025 1733Z expires 16-Jan-2025 1833Z

Or:  view_dupe TX7N.DJ4EL

|TX7N|DJ4EL|24925|UP, CQING = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z
|TX7N|DJ4EL|24925||1737048600|UPCQING = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z

The one we are interested in is this one:

|TX7N|DJ4EL|24925||1737048600|UPCQING = 16-Jan-2025 1732Z expires 16-Jan-2025 1832Z

(spot)      24930.0  1737048720  up, cqing

Comment field is processed and changed from "up, cqing" to "UPCQING"

We can compare the qrg of the spot with that of the hash:  24930.0 --> 24925    $Spot::qrggranularity = 25
The same goes for the time of the spot: (1732Z) 1737048720 --> 1737048600       $Spot::timegranularity = 600
16-Jan-2025 1732Z expires 16-Jan-2025 1832Z: 1732Z --> 1832Z                    $Spot::dupage = 3600

The formatting is defined by a function that already exists in the code itself, I simply display it without formatting it.

Kin

De: Björn Ekelund <bjorn at ekelund.nu> 
Enviado el: jueves, 16 de enero de 2025 18:29
Para: Kin <ea3cv at cronux.net>
CC: The DXSpider Support list <dxspider-support at tobit.co.uk>
Asunto: Re: [Dxspider-support] Self-spotting and de-duplication

Thanks Kin, 

I saw the earlier post and have tried it. I'm still not sure I understand how to read the output.

Björn SM7IUN

On Thu, Jan 16, 2025 at 2:20 PM Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> > wrote:
Björn, 

I forgot to tell you that if you want to see the changes introduced by the new values, you can use the following script:
https://github.com/EA3CV/dxspider_info/blob/main/view_dupes.pl

just place it in spider/local_cmd, and run it from console

it's easy to use: view_dupes <string>

and it will display the hashes that match the string.

Kin

De: Dxspider-support <dxspider-support-bounces at tobit.co.uk <mailto:dxspider-support-bounces at tobit.co.uk> > En nombre de Kin via Dxspider-support
Enviado el: jueves, 16 de enero de 2025 10:28
Para: 'Björn Ekelund' <bjorn at ekelund.nu <mailto:bjorn at ekelund.nu> >
CC: Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> >; 'The DXSpider Support list' <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> >
Asunto: Re: [Dxspider-support] Self-spotting and de-duplication

Morning Björn,

>So timegranularity defines a window within which spots are considered to have 
>the same time, Does this then effectively define the minimum respotting time?

Yes, timegranularity effectively defines the minimum respotting time. It determines the size of the time window within which spots are considered to have the same timestamp. For example, if timegranularity is set to 600 seconds (10 minutes), any spots reported within the same 10-minute interval will be grouped together, effectively preventing additional spots from being considered separate unless they occur in a different interval.

>Or is this the function of dupage? If it is, then I do not see the purpose of quantizing 
>time into timegranularity steps.

timegranularity is about standardizing time representation, while dupage enforces the minimum time between duplicate spots.

>And how can 600 be 4 minutes? Is time counted in steps of 0.4s?
>Does this mean that dupage 3600 is 24 minutes?

>And what is the function of qrggranularity? My empirical experiments say that I need 
>to move 10kHz away for DXSpider to not consider a new spot a dupe. Not 25kHz.

I hope that the following examples will clarify how it works.

>All I'm trying to do is to set up my node so that 

>1. a new spot of the same station on the same QRG after more than 3 minutes is never considered a dupe

$Spot::timegranularity = 60
$Spot::dupage = 180

You should know that a duplicate is the sum of the fields spotted+spotter+qrg+time+comments, which will generate a hash like this: |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM. For this reason, just using spotted+qrg+time is not sufficient to determine duplication.

>2. a new spot of a station 1kHz away from the previous spot is never considered a dupe.

$Spot::qrggranularity = 1

>3. self spots are never ever blocked

$Spot::minselfspotqrg = 0

Example

* Variables

$Spot::timegranularity = 600    --> 10 min
$Spot::qrggranularity = 25      --> 25 kHz
$Spot::dupage = 3600            --> 1 h

* Spot

DX de IK2JTS:     7065.0  IZ0ARL       DTMBA I1823RM                    0731Z (1737012660)

Adjusts to the nearest multiple (either downward or upward)

1737012660/600 = 2895021.1 --> 2895021 (round to the nearest integer)
2895021*600 = 1737012600

7065.0/25 = 282.6 --> 283 (round to the nearest integer)
283×25 = 7075

* dupefile

|IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM = 16-Jan-2025 0731Z expires 16-Jan-2025 0831Z

This means that time is fixed in 10-minute intervals, and frequency is fixed in 25 kHz steps.

* Comparison with different values for the same spot:

DX de IK2JTS:     7065.0  IZ0ARL       DTMBA #I-1823RM                  0735Z (1737012900)

Scenario 1:

$Spot::timegranularity = 600
$Spot::qrggranularity = 25
$Spot::dupage = 3600

|IZ0ARL|IT9JZK|7075||1737013200|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0835Z
                       (0740Z)

Scenario 2:

$Spot::timegranularity = 60
$Spot::qrggranularity = 1
$Spot::dupage = 300

|IZ0ARL|IT9JZK|7065||1737012900|DTMBAI1823RM = 16-Jan-2025 0735Z expires 16-Jan-2025 0740Z
                      (0735Z)

I'm not sure if I’ve managed to explain myself clearly this time. This is my interpretation after analyzing the code, from someone who is not an expert. I think the person who can help us best is Dirk, the creator of this. And sorry because my English is terrible

Kin EA3CV

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250116/dbd01361/attachment-0001.htm>