[Dxspider-support] Is dedupe really working?

Tue Feb 4 07:46:50 GMT 2025

Hi Andrea,

I believe I've figured out exactly how it works after trying to decipher Dirk’s code:

The variable $Spot::qrggranularity determines the minimum time that the sending node will allow a repeated spot to be transmitted.
In contrast, the variable $Spot::dupage affects the spots that enter the node.

In order to reproduce this behaviour in a test, the nodes through which the spot passes need to have $Spot::dupage times set equal to or lower than that of the node expected to receive the spot.

After analysing the code, I’ve managed to replicate exactly what I’ve just described.

Andrea, you owe me a cup of proper tea or a refreshing beer! 😂

Kin EA3CV

De: IZ2LSC <iz2lsc.andrea at gmail.com> 
Enviado el: lunes, 3 de febrero de 2025 22:45
Para: Kin <ea3cv at cronux.net>
CC: The DXSpider Support list <dxspider-support at tobit.co.uk>
Asunto: Re: [Dxspider-support] Is dedupe really working?

So, what is the aim of having $dupeage variable?
I'm a bit confused.

Andrea

-->

On Sun, Feb 2, 2025 at 9:30 AM Kin <ea3cv at cronux.net <mailto:ea3cv at cronux.net> > wrote:
Hi Andrea,

Use the value of $Spot::timegranularity to determine the minimum amount of time a spot cannot be repeated.
And choose the size of the duped spot history with $Spot::dupage.

In your case, you should have: $Spot::timegranularity = 1800

This is my understanding.

Kin EA3CV

De: Dxspider-support <dxspider-support-bounces at tobit.co.uk <mailto:dxspider-support-bounces at tobit.co.uk> > En nombre de IZ2LSC via Dxspider-support
Enviado el: sábado, 1 de febrero de 2025 22:48
Para: The DXSpider Support list <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> >
CC: IZ2LSC <iz2lsc.andrea at gmail.com <mailto:iz2lsc.andrea at gmail.com> >
Asunto: Re: [Dxspider-support] Is dedupe really working?

Please wait a second.

if my dupage = 1800 

this should mean that the same spot is not accepted and forwarded for 30 min.
So why Kin you have the same spot every minute:

sh/dx 0TEST
    7070.0 0TEST        1-Feb-2025 1931Z test ignore         <EA3CV>
   7070.0 0TEST        1-Feb-2025 1930Z test ignore         <EA3CV>
   7070.0 0TEST        1-Feb-2025 1928Z test ignore         <EA3CV>

I expect to see only the first spot, and no more dupes for the next 30 min.

Or am I misunderstanding the meaning of $dupeage and $timegranularity ?

How I can accomplish the goal of not accepting dupes if they are sent within 30 min?
And for dupes I mean the same spotter, dx, qrg, comment ?

Thanks

Andrea
iz2lsc

-->

On Sat, Feb 1, 2025 at 9:04 PM Dirk Koopman via Dxspider-support <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> > wrote:
A slightly edited version done on a decent screen and not a mobile phone with auto correct...

Edits underlined

On 01/02/2025 18:15, Dirk Koopman via Dxspider-support wrote:
The problem I have is that I cannot see what you are seeing. And further more,  if you are getting all these dupes you would be seeing multiple spots on your console. 

What I suggest you do is to stop your node, delete your /spider/local_cmd/dupefile then restart your node.

Startup a console and type set/debug dxdupe. In another shell window type ‘watchdbg dxdupe’.

Then wait. In amongst all the dupe add lines, once a minute you will see a line with the count of dupe records there are. If you have $Spot::dupe age set to 10 minutes, after that time has passed deletions should start to occur and the count will settle down to a slowly rising count (caused by 3 hours of announces) and then will stabilise to a number which will be in the lowish hundreds (as it’s a contest weekend). On normal days I see something between 120 - 160 ish.

You will see something like this (one minute's worth):

19:26:38 (dxdupe) DXDupe::add key: XZ32TO|IK4ZIF|7182|1738438200||EUDXC time: 1936Z
19:26:45 (dxdupe) DXDupe::add key: XEA4DEI|KB8OTK|24917|1738438200|| time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200||59INICELANDCQCO time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200|5/9 IN ICELAND time: 1936Z
19:27:05 (dxdupe) DXDupe::add key: XPA4O|RX7T|7015|1738438200|| time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200||MNQPSTE time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200|MNQP STE time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: XPE9GHZ/B|PA2DOL|5760815|1738438200||JO21GXTRJO11WM4 time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: XPE9GHZ/B|PA2DOL|5760815|1738438200|JO21GX(TR)JO11W time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200||MNOLMSTED time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200|MN OLMSTED time: 1937Z
19:27:29 (dxdupe) DXDupe::add key: XF5IN|RX7T|7016|1738438200|| time: 1937Z
19:27:34 (dxdupe) DXDupe::add key: XWA0JGT|AE9LL|14245|1738438200|| time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200||THEYEAROFHOMAGE time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200|THE YEAR OF HOM time: 1937Z
19:27:36 (dxdupe) DXDupe::del key: XDK2NI|UA3BL|3740|1738437600||CONTEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600|EU TEST HA6QU H time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XK3MTO|K5YDD|18128|1738437600||POTA time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XOM6TX|SP5BK|3731|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XTG9AHM|AD4YQ|24985|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600||EUTESTHA6QUHU04 time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XNQ1B|N3KN|21049|1738437600||CW time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600|CQ TEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600||CQTEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XVE3GJP|N5MEF|14250|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XSQ6PLE|DK1HJG|3586|1738437600||RTTY time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XEC1A|KC2GOW|21092|1738437600||RTTY time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XPA3DUU|LZ2HA|7058|1738437600||RTTY time: 1926Z
19:27:36 (dxdupe) DXDupe::per_minute number of records 258

It is normal to see two entries where there is a comment on the spot and the "compressed and normalised" version is different from the raw comment. "Normalised" means: capitalise all letters and remove everything that is not a letter or a digit.

The "number of records" is the current number of records in left in dupefile + the number of records it has just deleted. Just me being lazy, I'll change it to be the correct number at the time of printing. This instance has been running here since about 21h00 last night so should be representative of this level of activity. 

A swift "grepdbg dxdupe per minute", just now, yields:

01Feb2025 at 19:26:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:27:36 (dxdupe) DXDupe::per_minute number of records 258
01Feb2025 at 19:28:36 (dxdupe) DXDupe::per_minute number of records 263
01Feb2025 at 19:29:36 (dxdupe) DXDupe::per_minute number of records 271
01Feb2025 at 19:30:36 (dxdupe) DXDupe::per_minute number of records 262
01Feb2025 at 19:31:36 (dxdupe) DXDupe::per_minute number of records 251
01Feb2025 at 19:32:36 (dxdupe) DXDupe::per_minute number of records 246
01Feb2025 at 19:33:36 (dxdupe) DXDupe::per_minute number of records 244
01Feb2025 at 19:34:36 (dxdupe) DXDupe::per_minute number of records 247
01Feb2025 at 19:35:36 (dxdupe) DXDupe::per_minute number of records 239
01Feb2025 at 19:36:36 (dxdupe) DXDupe::per_minute number of records 234
01Feb2025 at 19:37:36 (dxdupe) DXDupe::per_minute number of records 249
01Feb2025 at 19:38:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:39:36 (dxdupe) DXDupe::per_minute number of records 248
01Feb2025 at 19:40:36 (dxdupe) DXDupe::per_minute number of records 260
01Feb2025 at 19:41:36 (dxdupe) DXDupe::per_minute number of records 259
01Feb2025 at 19:42:36 (dxdupe) DXDupe::per_minute number of records 264

This is a more "normal" level:

30Jan2025 at 11:40:12 (dxdupe) DXDupe::per_minute number of records 92
30Jan2025 at 11:41:12 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:42:26 (dxdupe) DXDupe::per_minute number of records 110
30Jan2025 at 11:44:05 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:44:45 (dxdupe) DXDupe::per_minute number of records 109
30Jan2025 at 11:45:15 (dxdupe) DXDupe::per_minute number of records 112
30Jan2025 at 11:46:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:47:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:48:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:49:12 (dxdupe) DXDupe::per_minute number of records 124
30Jan2025 at 11:50:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:51:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:52:12 (dxdupe) DXDupe::per_minute number of records 113
30Jan2025 at 11:53:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:54:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:55:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:56:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:57:12 (dxdupe) DXDupe::per_minute number of records 111
30Jan2025 at 11:58:12 (dxdupe) DXDupe::per_minute number of records 108
30Jan2025 at 11:59:12 (dxdupe) DXDupe::per_minute number of records 105
30Jan2025 at 12:00:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:01:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:02:12 (dxdupe) DXDupe::per_minute number of records 109
...

Further, just now on my test machine, I have done the same as you:

dx IZ2ABC 7090 test 2
DX de G7BRN:      7090.0  IZ2ABC       test 2           15 1946Z 14
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >

So, as far as I am concerned - just now at least - I believe deduping for spots is working.

73 Dirk G1TLH
_______________________________________________
Dxspider-support mailing list
Dxspider-support at tobit.co.uk <mailto:Dxspider-support at tobit.co.uk> 
https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250204/bf4eb020/attachment-0001.htm>