[Dxspider-support] Optimized Set and position Is dedupe really working?

Sun Feb 2 11:45:31 GMT 2025

Hi Alex,

For me, these are acceptable values:

$Spot::timegranularity = 180 to 300  (3*60 to 5*60 seconds)
$Spot::qrggranularity = 1            (1 kHz)
$Spot::dupage = 360 to 600           (timegranularity*2 seconds)    

These variables are set by default in the code and there is no place to define them in DXVars.pm.
To make them persistent, what you need to do is to include them in /spider/scripts/startup.

set/var $Spot::timegranularity = 180
set/var $Spot::qrggranularity = 1
set/var $Spot::dupage = 360

And for the new values to take effect without restarting the node, they must be executed on the console:

set/var $Spot::timegranularity = 180
set/var $Spot::qrggranularity = 1
set/var $Spot::dupage = 360

I take this opportunity to point out the need to include these variables (the following notes are Dirk's):

* set/var $DXProt::pc92c_ipaddr_enable = 1

Add a switch to enable the addition of IP addresses to PC92 C records.

   set/var $DXProt::pc92c_ipaddr_enable 1

to enable the addition of IP addresses to outgoing PC92 C records.

* set/var $main::localhost_alias_ipv4 = '190.2.45.251'
* set/var $main::localhost_alias_ipv6 = 'a00:23c4:b91f:5301:c423:19f0:799f:9dff'

Introduce aliasing for localhost in DX Spots and outgoing PC92 A records on login. There are two variables which can be set with the alias to use:
      $main::localhost_alias_ipv4
      $main::localhost_alias_ipv6
These can be set in the /spider/scripts/startup, but this is only necessary if the node has more than one interface, or virtual hosts. If there is ONLY ONE ipv4 and/or ipv6 IP address on the node machine then these variables will be automatically populated on first use. But the SAFE thing to do is to set them in the startup file.

* set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30 172.16.24.2)

Add the variable @main::localhost_names to allow other IP addresses to be treated in the same way as localhost in item 1 on 19Jan23 below.
NOTE you must include ALL the normal localhost names + any other interface names that you might want to include:

set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30)

using the qw() construction is easier than:

set/var @main::localhost_names ('127.0.0.1', '::1', '192.168.1.30')

but either will work. You can define as many IP addresses as you like and they can be IPV4 or 6.
You do NOT need to fiddle with this unless you specifically have more than just the normal definitions of localhost. So for 'normal' nodes with one external interface, you DO NOT NEED TO DO ANY OF THIS.

Conclusion:
In my startup I have:

set/var $Spot::timegranularity = 180
set/var $Spot::qrggranularity = 1
set/var $Spot::dupage = 360
set/var $DXProt::pc92c_ipaddr_enable = 1
set/var $main::localhost_alias_ipv4 = '190.2.45.251'
set/var $main::localhost_alias_ipv6 = 'a00:23c4:b91f:5301:c423:19f0:799f:9dff'
set/var @main::localhost_names qw(127.0.0.1 ::1 192.168.1.30 172.16.24.2)

But understand that this is my opinion, there may be better ones.

Kin EA3CV

De: Dxspider-support <dxspider-support-bounces at tobit.co.uk> En nombre de Alex via Dxspider-support
Enviado el: domingo, 2 de febrero de 2025 11:38
Para: 'The DXSpider Support list' <dxspider-support at tobit.co.uk>
CC: ik5pwj at gmail.com
Asunto: [Dxspider-support] Optimized Set and position Is dedupe really working?

Hi Kin and all,
could you please tell me where to insert the two settings and the sequence?

Is inserting the two commands at the end of the DXVars.pm file ok or does the different position of the two files not change anything?
Which settings in your experience are ok?

$Spot::timegranularity ????
$Spot::dupage ????

Can you indicate them?
Is there some other setting that I missed in the various discussions that can positively affect the double spots?

Thanks
Alex ik5pwj

Da: Dxspider-support <dxspider-support-bounces at tobit.co.uk> Per conto di Kin via Dxspider-support
Inviato: domenica 2 febbraio 2025 09:30
A: iz2lsc.andrea at gmail.com; 'The DXSpider Support list' <dxspider-support at tobit.co.uk>
Cc: Kin <ea3cv at cronux.net>
Oggetto: Re: [Dxspider-support] Is dedupe really working?

Hi Andrea,

Use the value of $Spot::timegranularity to determine the minimum amount of time a spot cannot be repeated.
And choose the size of the duped spot history with $Spot::dupage.

In your case, you should have: $Spot::timegranularity = 1800

This is my understanding.

Kin EA3CV

De: Dxspider-support <dxspider-support-bounces at tobit.co.uk <mailto:dxspider-support-bounces at tobit.co.uk> > En nombre de IZ2LSC via Dxspider-support
Enviado el: sábado, 1 de febrero de 2025 22:48
Para: The DXSpider Support list <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> >
CC: IZ2LSC <iz2lsc.andrea at gmail.com <mailto:iz2lsc.andrea at gmail.com> >
Asunto: Re: [Dxspider-support] Is dedupe really working?

Please wait a second.

if my dupage = 1800 

this should mean that the same spot is not accepted and forwarded for 30 min.
So why Kin you have the same spot every minute:

sh/dx 0TEST
    7070.0 0TEST        1-Feb-2025 1931Z test ignore         <EA3CV>
   7070.0 0TEST        1-Feb-2025 1930Z test ignore         <EA3CV>
   7070.0 0TEST        1-Feb-2025 1928Z test ignore         <EA3CV>

I expect to see only the first spot, and no more dupes for the next 30 min.

Or am I misunderstanding the meaning of $dupeage and $timegranularity ?

How I can accomplish the goal of not accepting dupes if they are sent within 30 min?
And for dupes I mean the same spotter, dx, qrg, comment ?

Thanks

Andrea
iz2lsc

-->

On Sat, Feb 1, 2025 at 9:04 PM Dirk Koopman via Dxspider-support <dxspider-support at tobit.co.uk <mailto:dxspider-support at tobit.co.uk> > wrote:
A slightly edited version done on a decent screen and not a mobile phone with auto correct...

Edits underlined

On 01/02/2025 18:15, Dirk Koopman via Dxspider-support wrote:
The problem I have is that I cannot see what you are seeing. And further more,  if you are getting all these dupes you would be seeing multiple spots on your console. 

What I suggest you do is to stop your node, delete your /spider/local_cmd/dupefile then restart your node.

Startup a console and type set/debug dxdupe. In another shell window type ‘watchdbg dxdupe’.

Then wait. In amongst all the dupe add lines, once a minute you will see a line with the count of dupe records there are. If you have $Spot::dupe age set to 10 minutes, after that time has passed deletions should start to occur and the count will settle down to a slowly rising count (caused by 3 hours of announces) and then will stabilise to a number which will be in the lowish hundreds (as it’s a contest weekend). On normal days I see something between 120 - 160 ish.

You will see something like this (one minute's worth):

19:26:38 (dxdupe) DXDupe::add key: XZ32TO|IK4ZIF|7182|1738438200||EUDXC time: 1936Z
19:26:45 (dxdupe) DXDupe::add key: XEA4DEI|KB8OTK|24917|1738438200|| time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200||59INICELANDCQCO time: 1936Z
19:26:50 (dxdupe) DXDupe::add key: XK1LZ|TF1A|14200|1738438200|5/9 IN ICELAND time: 1936Z
19:27:05 (dxdupe) DXDupe::add key: XPA4O|RX7T|7015|1738438200|| time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200||MNQPSTE time: 1937Z
19:27:06 (dxdupe) DXDupe::add key: XWA0MN|N0UR|28044|1738438200|MNQP STE time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: XPE9GHZ/B|PA2DOL|5760815|1738438200||JO21GXTRJO11WM4 time: 1937Z
19:27:16 (dxdupe) DXDupe::add key: XPE9GHZ/B|PA2DOL|5760815|1738438200|JO21GX(TR)JO11W time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200||MNOLMSTED time: 1937Z
19:27:22 (dxdupe) DXDupe::add key: XAI0M|W2VTV|28420|1738438200|MN OLMSTED time: 1937Z
19:27:29 (dxdupe) DXDupe::add key: XF5IN|RX7T|7016|1738438200|| time: 1937Z
19:27:34 (dxdupe) DXDupe::add key: XWA0JGT|AE9LL|14245|1738438200|| time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200||THEYEAROFHOMAGE time: 1937Z
19:27:35 (dxdupe) DXDupe::add key: XYQ5A|UB3LBU|3573|1738438200|THE YEAR OF HOM time: 1937Z
19:27:36 (dxdupe) DXDupe::del key: XDK2NI|UA3BL|3740|1738437600||CONTEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600|EU TEST HA6QU H time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XK3MTO|K5YDD|18128|1738437600||POTA time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XOM6TX|SP5BK|3731|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XTG9AHM|AD4YQ|24985|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XHA6QU|HA5KFU|1837|1738437600||EUTESTHA6QUHU04 time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XNQ1B|N3KN|21049|1738437600||CW time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600|CQ TEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XPV2K|PY2OF|14248|1738437600||CQTEST time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XVE3GJP|N5MEF|14250|1738437600|| time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XSQ6PLE|DK1HJG|3586|1738437600||RTTY time: 1926Z
19:27:36 (dxdupe) DXDupe::del key: XEC1A|KC2GOW|21092|1738437600||RTTY time: 1927Z
19:27:36 (dxdupe) DXDupe::del key: XPA3DUU|LZ2HA|7058|1738437600||RTTY time: 1926Z
19:27:36 (dxdupe) DXDupe::per_minute number of records 258

It is normal to see two entries where there is a comment on the spot and the "compressed and normalised" version is different from the raw comment. "Normalised" means: capitalise all letters and remove everything that is not a letter or a digit.

The "number of records" is the current number of records in left in dupefile + the number of records it has just deleted. Just me being lazy, I'll change it to be the correct number at the time of printing. This instance has been running here since about 21h00 last night so should be representative of this level of activity. 

A swift "grepdbg dxdupe per minute", just now, yields:

01Feb2025 at 19:26:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:27:36 (dxdupe) DXDupe::per_minute number of records 258
01Feb2025 at 19:28:36 (dxdupe) DXDupe::per_minute number of records 263
01Feb2025 at 19:29:36 (dxdupe) DXDupe::per_minute number of records 271
01Feb2025 at 19:30:36 (dxdupe) DXDupe::per_minute number of records 262
01Feb2025 at 19:31:36 (dxdupe) DXDupe::per_minute number of records 251
01Feb2025 at 19:32:36 (dxdupe) DXDupe::per_minute number of records 246
01Feb2025 at 19:33:36 (dxdupe) DXDupe::per_minute number of records 244
01Feb2025 at 19:34:36 (dxdupe) DXDupe::per_minute number of records 247
01Feb2025 at 19:35:36 (dxdupe) DXDupe::per_minute number of records 239
01Feb2025 at 19:36:36 (dxdupe) DXDupe::per_minute number of records 234
01Feb2025 at 19:37:36 (dxdupe) DXDupe::per_minute number of records 249
01Feb2025 at 19:38:36 (dxdupe) DXDupe::per_minute number of records 256
01Feb2025 at 19:39:36 (dxdupe) DXDupe::per_minute number of records 248
01Feb2025 at 19:40:36 (dxdupe) DXDupe::per_minute number of records 260
01Feb2025 at 19:41:36 (dxdupe) DXDupe::per_minute number of records 259
01Feb2025 at 19:42:36 (dxdupe) DXDupe::per_minute number of records 264

This is a more "normal" level:

30Jan2025 at 11:40:12 (dxdupe) DXDupe::per_minute number of records 92
30Jan2025 at 11:41:12 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:42:26 (dxdupe) DXDupe::per_minute number of records 110
30Jan2025 at 11:44:05 (dxdupe) DXDupe::per_minute number of records 100
30Jan2025 at 11:44:45 (dxdupe) DXDupe::per_minute number of records 109
30Jan2025 at 11:45:15 (dxdupe) DXDupe::per_minute number of records 112
30Jan2025 at 11:46:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:47:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:48:12 (dxdupe) DXDupe::per_minute number of records 123
30Jan2025 at 11:49:12 (dxdupe) DXDupe::per_minute number of records 124
30Jan2025 at 11:50:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:51:12 (dxdupe) DXDupe::per_minute number of records 125
30Jan2025 at 11:52:12 (dxdupe) DXDupe::per_minute number of records 113
30Jan2025 at 11:53:12 (dxdupe) DXDupe::per_minute number of records 119
30Jan2025 at 11:54:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:55:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:56:12 (dxdupe) DXDupe::per_minute number of records 115
30Jan2025 at 11:57:12 (dxdupe) DXDupe::per_minute number of records 111
30Jan2025 at 11:58:12 (dxdupe) DXDupe::per_minute number of records 108
30Jan2025 at 11:59:12 (dxdupe) DXDupe::per_minute number of records 105
30Jan2025 at 12:00:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:01:12 (dxdupe) DXDupe::per_minute number of records 99
30Jan2025 at 12:02:12 (dxdupe) DXDupe::per_minute number of records 109
...

Further, just now on my test machine, I have done the same as you:

dx IZ2ABC 7090 test 2
DX de G7BRN:      7090.0  IZ2ABC       test 2           15 1946Z 14
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >
dx IZ2ABC 7090 test 2
Sorry, this is a duplicate
G7BRN de GB7TLH  1-Feb-2025 1946Z dxspider >

So, as far as I am concerned - just now at least - I believe deduping for spots is working.

73 Dirk G1TLH
_______________________________________________
Dxspider-support mailing list
Dxspider-support at tobit.co.uk <mailto:Dxspider-support at tobit.co.uk> 
https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250202/b0eaa16a/attachment-0001.htm>