[Dxspider-support] Is dedupe really working?

Tue Feb 4 11:18:06 GMT 2025

Hi Kin,
I owe you more than a beer. Maybe also a pizza, let me know when you are
close to Milan.

Honestly speaking the behaviour and how the 2 params interact with each
other is not crystal clear.
I'm talking about timegranularity (not qrggranularity) and dupeage.
In my opinion the dupage is only used to clean up the dupefile and avoid it
reaching a big size.
The check to verify if a spot is a dupe seems to be based on the timestamp
(rounded at timegranularity) that is part of the hash.

Andrea

-->

On Tue, Feb 4, 2025 at 8:47 AM Kin <ea3cv at cronux.net> wrote:

> Hi Andrea,
>
>
>
> I believe I've figured out exactly how it works after trying to decipher
> Dirk’s code:
>
>
>
> The variable $Spot::qrggranularity determines the minimum time that the
> sending node will allow a repeated spot to be transmitted.
>
> In contrast, the variable $Spot::dupage affects the spots that enter the
> node.
>
>
>
> In order to reproduce this behaviour in a test, the nodes through which
> the spot passes need to have $Spot::dupage times set equal to or lower
> than that of the node expected to receive the spot.
>
>
>
> After analysing the code, I’ve managed to replicate exactly what I’ve just
> described.
>
>
>
> Andrea, you owe me a cup of proper tea or a refreshing beer! 😂
>
>
>
> Kin EA3CV
>
>
>
>
>
> *De:* IZ2LSC <iz2lsc.andrea at gmail.com>
> *Enviado el:* lunes, 3 de febrero de 2025 22:45
> *Para:* Kin <ea3cv at cronux.net>
> *CC:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *Asunto:* Re: [Dxspider-support] Is dedupe really working?
>
>
>
> So, what is the aim of having $dupeage variable?
>
> I'm a bit confused.
>
>
>
> Andrea
>
>
>
> -->
>
>
>
>
>
> On Sun, Feb 2, 2025 at 9:30 AM Kin <ea3cv at cronux.net> wrote:
>
> Hi Andrea,
>
>
>
> Use the value of $Spot::timegranularity to determine the minimum amount of
> time a spot cannot be repeated.
>
> And choose the size of the duped spot history with $Spot::dupage.
>
>
>
> In your case, you should have: $Spot::timegranularity = 1800
>
>
>
> This is my understanding.
>
>
>
> Kin EA3CV
>
>
>
>
>
> *De:* Dxspider-support <dxspider-support-bounces at tobit.co.uk> *En nombre
> de *IZ2LSC via Dxspider-support
> *Enviado el:* sábado, 1 de febrero de 2025 22:48
> *Para:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *CC:* IZ2LSC <iz2lsc.andrea at gmail.com>
> *Asunto:* Re: [Dxspider-support] Is dedupe really working?
>
>
>
> Please wait a second.
>
> if my dupage = 1800
>
>
>
> this should mean that the same spot is not accepted and forwarded for 30
> min.
>
> So why Kin you have the same spot every minute:
>
>
>
> sh/dx 0TEST
>
>     7070.0 0TEST        1-Feb-2025 1931Z test ignore         <EA3CV>
>
>    7070.0 0TEST        1-Feb-2025 1930Z test ignore         <EA3CV>
>
>    7070.0 0TEST        1-Feb-2025 1928Z test ignore         <EA3CV>
>
>
>
> I expect to see only the first spot, and no more dupes for the next 30 min.
>
>
>
> Or am I misunderstanding the meaning of $dupeage and $timegranularity ?
>
>
>
> How I can accomplish the goal of not accepting dupes if they are sent
> within 30 min?
>
> And for dupes I mean the same spotter, dx, qrg, comment ?
>
>
>
> Thanks
>
>
>
> Andrea
>
> iz2lsc
>
>
>
>
>
>
>
> -->
>
>
>
>
>
> On Sat, Feb 1, 2025 at 9:04 PM Dirk Koopman via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
> A slightly edited version done on a decent screen and not a mobile phone
> with auto correct...
>
> Edits underlined
>
> On 01/02/2025 18:15, Dirk Koopman via Dxspider-support wrote:
>
> The problem I have is that I *cannot* see what you are seeing. And
> further more, * if you are getting all these dupes you would be seeing
> multiple spots on your console*.
>
>
>
> What I suggest you do is to stop your node, delete your
> /spider/local_cmd/dupefile then restart your node.
>
>
>
> Startup a console and type set/debug *dxdupe*. In another shell window
> type ‘watchdbg dxdupe’.
>
>
>
> Then wait. In amongst all the dupe add lines, once a minute you will see a
> line with the count of dupe records there are. If you have $Spot::dupe age
> set to 10 minutes, after that time has passed deletions should start to
> occur and the count will settle down to a slowly rising count (caused by 3
> hours of announces) and then will stabilise to a number which will be in
> the lowish hundreds (as it’s a contest weekend). On normal days I see
> something between 120 - 160 ish.
>
>
>
> You will see something like this (one minute's worth):
>
> 19:26:38 (dxdupe) DXDupe::add key: XZ32TO|IK4ZIF|7182|1738438200||EUDXC
> time: 1936Z
> 19:26:45 (dxdupe) DXDupe::add key: XEA4DEI|KB8OTK|24917|1738438200|| time:
> 1936Z
> 19:26:50 (dxdupe) DXDupe::add key:
> XK1LZ|TF1A|14200|1738438200||59INICELANDCQCO time: 1936Z
> 19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200|5/9 IN
> ICELAND time: 1936Z
> 19:27:05 (dxdupe) DXDupe::add key: XPA4O|RX7T|7015|1738438200|| time: 1937Z
> 19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200||MNQPSTE
> time: 1937Z
> 19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200|MNQP STE
> time: 1937Z
> 19:27:16 (dxdupe) DXDupe::add key:
> XPE9GHZ/B|PA2DOL|5760815|1738438200||JO21GXTRJO11WM4 time: 1937Z
> 19:27:16 (dxdupe) DXDupe::add key:
> XPE9GHZ/B|PA2DOL|5760815|1738438200|JO21GX(TR)JO11W time: 1937Z
> 19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200||MNOLMSTED
> time: 1937Z
> 19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200|MN OLMSTED
> time: 1937Z
> 19:27:29 (dxdupe) DXDupe::add key: XF5IN|RX7T|7016|1738438200|| time: 1937Z
> 19:27:34 (dxdupe) DXDupe::add key: XWA0JGT|AE9LL|14245|1738438200|| time:
> 1937Z
> 19:27:35 (dxdupe) DXDupe::add key:
> XYQ5A|UB3LBU|3573|1738438200||THEYEAROFHOMAGE time: 1937Z
> 19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200|THE YEAR
> OF HOM time: 1937Z
> 19:27:36 (dxdupe) DXDupe::del key: XDK2NI|UA3BL|3740|1738437600||CONTEST
> time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600|EU TEST
> HA6QU H time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XK3MTO|K5YDD|18128|1738437600||POTA
> time: 1927Z
> 19:27:36 (dxdupe) DXDupe::del key: XOM6TX|SP5BK|3731|1738437600|| time:
> 1927Z
> 19:27:36 (dxdupe) DXDupe::del key: XTG9AHM|AD4YQ|24985|1738437600|| time:
> 1927Z
> 19:27:36 (dxdupe) DXDupe::del key:
> XHA6QU|HA5KFU|1837|1738437600||EUTESTHA6QUHU04 time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XNQ1B|N3KN|21049|1738437600||CW time:
> 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600|CQ TEST
> time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600||CQTEST
> time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XVE3GJP|N5MEF|14250|1738437600|| time:
> 1927Z
> 19:27:36 (dxdupe) DXDupe::del key: XSQ6PLE|DK1HJG|3586|1738437600||RTTY
> time: 1926Z
> 19:27:36 (dxdupe) DXDupe::del key: XEC1A|KC2GOW|21092|1738437600||RTTY
> time: 1927Z
> 19:27:36 (dxdupe) DXDupe::del key: XPA3DUU|LZ2HA|7058|1738437600||RTTY
> time: 1926Z
> 19:27:36 (dxdupe) DXDupe::per_minute number of records 258
>
> It is normal to see two entries where there is a comment on the spot and
> the "compressed and normalised" version is different from the raw comment.
> "Normalised" means: capitalise all letters and remove everything that is
> not a letter or a digit.
>
> The "number of records" is the current number of records in left in
> dupefile + the number of records it has just deleted. Just me being lazy,
> I'll change it to be the correct number at the time of printing. This
> instance has been running here since about 21h00 last night so should be
> representative of this level of activity.
>
> A swift "grepdbg dxdupe per minute", just now, yields:
>
> 01Feb2025 at 19:26:36 (dxdupe) DXDupe::per_minute number of records 256
> 01Feb2025 at 19:27:36 (dxdupe) DXDupe::per_minute number of records 258
> 01Feb2025 at 19:28:36 (dxdupe) DXDupe::per_minute number of records 263
> 01Feb2025 at 19:29:36 (dxdupe) DXDupe::per_minute number of records 271
> 01Feb2025 at 19:30:36 (dxdupe) DXDupe::per_minute number of records 262
> 01Feb2025 at 19:31:36 (dxdupe) DXDupe::per_minute number of records 251
> 01Feb2025 at 19:32:36 (dxdupe) DXDupe::per_minute number of records 246
> 01Feb2025 at 19:33:36 (dxdupe) DXDupe::per_minute number of records 244
> 01Feb2025 at 19:34:36 (dxdupe) DXDupe::per_minute number of records 247
> 01Feb2025 at 19:35:36 (dxdupe) DXDupe::per_minute number of records 239
> 01Feb2025 at 19:36:36 (dxdupe) DXDupe::per_minute number of records 234
> 01Feb2025 at 19:37:36 (dxdupe) DXDupe::per_minute number of records 249
> 01Feb2025 at 19:38:36 (dxdupe) DXDupe::per_minute number of records 256
> 01Feb2025 at 19:39:36 (dxdupe) DXDupe::per_minute number of records 248
> 01Feb2025 at 19:40:36 (dxdupe) DXDupe::per_minute number of records 260
> 01Feb2025 at 19:41:36 (dxdupe) DXDupe::per_minute number of records 259
> 01Feb2025 at 19:42:36 (dxdupe) DXDupe::per_minute number of records 264
>
> This is a more "normal" level:
>
> 30Jan2025 at 11:40:12 (dxdupe) DXDupe::per_minute number of records 92
> 30Jan2025 at 11:41:12 (dxdupe) DXDupe::per_minute number of records 100
> 30Jan2025 at 11:42:26 (dxdupe) DXDupe::per_minute number of records 110
> 30Jan2025 at 11:44:05 (dxdupe) DXDupe::per_minute number of records 100
> 30Jan2025 at 11:44:45 (dxdupe) DXDupe::per_minute number of records 109
> 30Jan2025 at 11:45:15 (dxdupe) DXDupe::per_minute number of records 112
> 30Jan2025 at 11:46:12 (dxdupe) DXDupe::per_minute number of records 119
> 30Jan2025 at 11:47:12 (dxdupe) DXDupe::per_minute number of records 123
> 30Jan2025 at 11:48:12 (dxdupe) DXDupe::per_minute number of records 123
> 30Jan2025 at 11:49:12 (dxdupe) DXDupe::per_minute number of records 124
> 30Jan2025 at 11:50:12 (dxdupe) DXDupe::per_minute number of records 125
> 30Jan2025 at 11:51:12 (dxdupe) DXDupe::per_minute number of records 125
> 30Jan2025 at 11:52:12 (dxdupe) DXDupe::per_minute number of records 113
> 30Jan2025 at 11:53:12 (dxdupe) DXDupe::per_minute number of records 119
> 30Jan2025 at 11:54:12 (dxdupe) DXDupe::per_minute number of records 115
> 30Jan2025 at 11:55:12 (dxdupe) DXDupe::per_minute number of records 115
> 30Jan2025 at 11:56:12 (dxdupe) DXDupe::per_minute number of records 115
> 30Jan2025 at 11:57:12 (dxdupe) DXDupe::per_minute number of records 111
> 30Jan2025 at 11:58:12 (dxdupe) DXDupe::per_minute number of records 108
> 30Jan2025 at 11:59:12 (dxdupe) DXDupe::per_minute number of records 105
> 30Jan2025 at 12:00:12 (dxdupe) DXDupe::per_minute number of records 99
> 30Jan2025 at 12:01:12 (dxdupe) DXDupe::per_minute number of records 99
> 30Jan2025 at 12:02:12 (dxdupe) DXDupe::per_minute number of records 109
> ...
>
> Further, just now on my test machine, I have done the same as you:
>
> dx IZ2ABC 7090 test 2
> DX de G7BRN:      7090.0  IZ2ABC       test 2           15 1946Z 14
> G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
> dx IZ2ABC 7090 test 2
> Sorry, this is a duplicate
> G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
> dx IZ2ABC 7090 test 2
> Sorry, this is a duplicate
> G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
>
> So, as far as I am concerned - just now at least - I believe deduping for
> spots is working.
>
> 73 Dirk G1TLH
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250204/94de6ded/attachment-0001.htm>