[Dxspider-support] Is dedupe really working?

Sat Feb 1 19:50:12 GMT 2025

A slightly edited version done on a decent screen and not a mobile phone 
with auto correct...

Edits underlined

On 01/02/2025 18:15, Dirk Koopman via Dxspider-support wrote:
> The problem I have is that I _cannot_ see what you are seeing. And 
> further more, _if you are getting all these dupes you would be seeing 
> multiple spots on your console_.
>
> What I suggest you do is to stop your node, delete your 
> /spider/local_cmd/dupefile then restart your node.
>
> Startup a console and type set/debug _dxdupe_. In another shell window 
> type ‘watchdbg dxdupe’.
>
> Then wait. In amongst all the dupe add lines, once a minute you will 
> see a line with the count of dupe records there are. If you have 
> $Spot::dupe age set to 10 minutes, after that time has passed 
> deletions should start to occur and the count will settle down to a 
> slowly rising count (caused by 3 hours of announces) and then will 
> stabilise to a number which will be in the lowish hundreds (as it’s a 
> contest weekend). On normal days I see something between 120 - 160 ish.

You will see something like this (one minute's worth):

19:26:38 (dxdupe) DXDupe::add key: XZ32TO|IK4ZIF|7182|1738438200||EUDXC 
time: 1936Z
19:26:45 (dxdupe) DXDupe::add key: XEA4DEI|KB8OTK|24917|1738438200|| 
time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: 
XK1LZ|TF1A|14200|1738438200||59INICELANDCQCO time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200|5/9 IN 
ICELAND time: 1936Z
19:27:05 (dxdupe) DXDupe::add key: XPA4O|RX7T|7015|1738438200|| time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200||MNQPSTE 
time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200|MNQP STE 
time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: 
XPE9GHZ/B|PA2DOL|5760815|1738438200||JO21GXTRJO11WM4 time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: 
XPE9GHZ/B|PA2DOL|5760815|1738438200|JO21GX(TR)JO11W time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: 
XAI0M|W2VTV|28420|1738438200||MNOLMSTED time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200|MN 
OLMSTED time: 1937Z
19:27:29 (dxdupe) DXDupe::add key: XF5IN|RX7T|7016|1738438200|| time: 1937Z
19:27:34 (dxdupe) DXDupe::add key: XWA0JGT|AE9LL|14245|1738438200|| 
time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: 
XYQ5A|UB3LBU|3573|1738438200||THEYEAROFHOMAGE time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200|THE YEAR 
OF HOM time: 1937Z
19:27:36 (dxdupe) DXDupe::del key: XDK2NI|UA3BL|3740|1738437600||CONTEST 
time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600|EU TEST 
HA6QU H time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XK3MTO|K5YDD|18128|1738437600||POTA 
time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XOM6TX|SP5BK|3731|1738437600|| time: 
1927Z
19:27:36 (dxdupe) DXDupe::del key: XTG9AHM|AD4YQ|24985|1738437600|| 
time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: 
XHA6QU|HA5KFU|1837|1738437600||EUTESTHA6QUHU04 time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XNQ1B|N3KN|21049|1738437600||CW time: 
1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600|CQ TEST 
time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600||CQTEST 
time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XVE3GJP|N5MEF|14250|1738437600|| 
time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XSQ6PLE|DK1HJG|3586|1738437600||RTTY 
time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XEC1A|KC2GOW|21092|1738437600||RTTY 
time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XPA3DUU|LZ2HA|7058|1738437600||RTTY 
time: 1926Z
19:27:36 (dxdupe) DXDupe::per_minute number of records 258

It is normal to see two entries where there is a comment on the spot and 
the "compressed and normalised" version is different from the raw 
comment. "Normalised" means: capitalise all letters and remove 
everything that is not a letter or a digit.

The "number of records" is the current number of records in left in 
dupefile + the number of records it has just deleted. Just me being 
lazy, I'll change it to be the correct number at the time of printing. 
This instance has been running here since about 21h00 last night so 
should be representative of this level of activity.

A swift "grepdbg dxdupe per minute", just now, yields:

01Feb2025 at 19:26:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:27:36 (dxdupe) DXDupe::per_minute number of records 258
01Feb2025 at 19:28:36 (dxdupe) DXDupe::per_minute number of records 263
01Feb2025 at 19:29:36 (dxdupe) DXDupe::per_minute number of records 271
01Feb2025 at 19:30:36 (dxdupe) DXDupe::per_minute number of records 262
01Feb2025 at 19:31:36 (dxdupe) DXDupe::per_minute number of records 251
01Feb2025 at 19:32:36 (dxdupe) DXDupe::per_minute number of records 246
01Feb2025 at 19:33:36 (dxdupe) DXDupe::per_minute number of records 244
01Feb2025 at 19:34:36 (dxdupe) DXDupe::per_minute number of records 247
01Feb2025 at 19:35:36 (dxdupe) DXDupe::per_minute number of records 239
01Feb2025 at 19:36:36 (dxdupe) DXDupe::per_minute number of records 234
01Feb2025 at 19:37:36 (dxdupe) DXDupe::per_minute number of records 249
01Feb2025 at 19:38:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:39:36 (dxdupe) DXDupe::per_minute number of records 248
01Feb2025 at 19:40:36 (dxdupe) DXDupe::per_minute number of records 260
01Feb2025 at 19:41:36 (dxdupe) DXDupe::per_minute number of records 259
01Feb2025 at 19:42:36 (dxdupe) DXDupe::per_minute number of records 264

This is a more "normal" level:

30Jan2025 at 11:40:12 (dxdupe) DXDupe::per_minute number of records 92
30Jan2025 at 11:41:12 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:42:26 (dxdupe) DXDupe::per_minute number of records 110
30Jan2025 at 11:44:05 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:44:45 (dxdupe) DXDupe::per_minute number of records 109
30Jan2025 at 11:45:15 (dxdupe) DXDupe::per_minute number of records 112
30Jan2025 at 11:46:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:47:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:48:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:49:12 (dxdupe) DXDupe::per_minute number of records 124
30Jan2025 at 11:50:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:51:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:52:12 (dxdupe) DXDupe::per_minute number of records 113
30Jan2025 at 11:53:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:54:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:55:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:56:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:57:12 (dxdupe) DXDupe::per_minute number of records 111
30Jan2025 at 11:58:12 (dxdupe) DXDupe::per_minute number of records 108
30Jan2025 at 11:59:12 (dxdupe) DXDupe::per_minute number of records 105
30Jan2025 at 12:00:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:01:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:02:12 (dxdupe) DXDupe::per_minute number of records 109
...

Further, just now on my test machine, I have done the same as you:

dx IZ2ABC 7090 test 2
DX de G7BRN:      7090.0  IZ2ABC       test 2           15 1946Z 14
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >

So, as far as I am concerned - just now at least - I believe deduping 
for spots is working.

73 Dirk G1TLH

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250201/e9d16f0e/attachment-0001.htm>