[Dxspider-support] Self-spotting and de-duplication

Fri Jan 17 13:31:21 GMT 2025

Thanks Kin,
I will try it on my backup (isolated) cluster node.

73
Andrea
iz2lsc

-->

On Fri, Jan 17, 2025 at 12:55 PM Kin <ea3cv at cronux.net> wrote:

> Morning Andrea,
>
>
>
> While waiting for Dirk to confirm the possible bug and its solution, I'm
> passing you a modified version of DXDupe.pm that fixes the deletion of
> hashes depending on the $Spot::dupage variable.
>
> It also solves the problem of deleting expired spots.
>
>
>
> It should be copied to: /spider/Local and restart the node.
>
>
>
> Kin EA3CV
>
>
>
> *De:* Dxspider-support <dxspider-support-bounces at tobit.co.uk> *En nombre
> de *IZ2LSC via Dxspider-support
> *Enviado el:* viernes, 17 de enero de 2025 10:18
> *Para:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *CC:* IZ2LSC <iz2lsc.andrea at gmail.com>
> *Asunto:* Re: [Dxspider-support] Self-spotting and de-duplication
>
>
>
> If every time a spot is ignored is also added again to the dupe list, this
> is a loop. Maybe is this the bug?
>
>
>
> 1737101764^(pc11) PCPROT: Duplicate Spot  IZ2LSC-97: PC61
> 7006.0|JF4IQS/1|1737101700|JF4IQS|IZ2LSC-97 ignored
>
>
>
> view_dupes JF4IQS/1
> |JF4IQS/1|JF4IQS|433060||1737100440|POTAJP1149FMHIK = 17-Jan-2025 0754Z
> expires 17-Jan-2025 0824Z
> |JF4IQS/1|JF4IQS|7005||1737100800|POTAJP1149CWHIK = 17-Jan-2025 0801Z
> expires 17-Jan-2025 0831Z
> |JF4IQS/1|JF4IQS|7005|[-POTA-] JP-114 = 17-Jan-2025 0801Z expires
> 17-Jan-2025 0831Z
> |JF4IQS/1|JI2OJV|7005||1737101580|POTAJP1149CWHIK = 17-Jan-2025 0814Z
> expires 17-Jan-2025 0844Z
> |JF4IQS/1|JI2OJV|7005|[-POTA-] JP-114 = 17-Jan-2025 0814Z expires
> 17-Jan-2025 0844Z
> *|JF4IQS/1|JF4IQS|7005||1737101700|POTAJP1149CWHIK = 17-Jan-2025 0816Z
> expires 17-Jan-2025 0846Z*
>
>
>
> -->
>
>
>
>
>
> On Thu, Jan 16, 2025 at 11:01 PM IZ2LSC <iz2lsc.andrea at gmail.com> wrote:
>
> Kin,
>
> when the hash is expired, is it removed from the file?
>
> I can see hash already expired and still present in the file.
>
> Not sure if this is expected.
>
>
>
> Andrea
>
>
>
> -->
>
>
>
>
>
> On Thu, Jan 16, 2025 at 8:12 PM Björn Ekelund via Dxspider-support <
> dxspider-support at tobit.co.uk> wrote:
>
> Thanks. For human spots it makes perfect sense to include the comment.
>
>
>
> Björn SM7IUN
>
>
>
> On Thu, Jan 16, 2025 at 7:06 PM Kin <ea3cv at cronux.net> wrote:
>
> Björn,
>
>
>
> All this is for non-RBN spots.
>
> I don't remember now the treatment of RBN spots.
>
>
>
> Kin
>
>
>
> *De:* Björn Ekelund <bjorn at ekelund.nu>
> *Enviado el:* jueves, 16 de enero de 2025 18:26
> *Para:* Kin <ea3cv at cronux.net>
> *CC:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *Asunto:* Re: [Dxspider-support] Self-spotting and de-duplication
>
>
>
> Kin,
>
>
>
> this was very clear and very helpful.
>
> Thank you for spending the time to do it.
>
> I am sure others appreciate it too.
>
>
>
> In my opinion, all comments should be ignored when determining dupes for
> skimmer spots.
>
>
>
> Björn SM7IUN
>
>
>
> On Thu, Jan 16, 2025 at 10:28 AM Kin <ea3cv at cronux.net> wrote:
>
> Morning Björn,
>
>
>
> *>So timegranularity defines a window within which spots are considered to
> have *
>
> *>the same time, Does this then effectively define the minimum respotting
> time?*
>
>
>
> Yes, timegranularity effectively defines the minimum respotting time. It
> determines the size of the time window within which spots are considered to
> have the same timestamp. For example, if timegranularity is set to 600
> seconds (10 minutes), any spots reported within the same 10-minute interval
> will be grouped together, effectively preventing additional spots from
> being considered separate unless they occur in a different interval.
>
>
>
> *>Or is this the function of dupage? If it is, then I do not see the
> purpose of quantizing *
>
> *>time into timegranularity steps.*
>
>
>
> timegranularity is about standardizing time representation, while dupage
> enforces the minimum time between duplicate spots.
>
>
>
> *>And how can 600 be 4 minutes? Is time counted in steps of 0.4s?*
>
> *>Does this mean that dupage 3600 is 24 minutes?*
>
>
>
> *>And what is the function of qrggranularity? My empirical experiments say
> that I need *
>
> *>to move 10kHz away for DXSpider to not consider a new spot a dupe. Not
> 25kHz.*
>
>
>
> I hope that the following examples will clarify how it works.
>
>
>
> *>All I'm trying to do is to set up my node so that *
>
>
>
> *>1. a new spot of the same station on the same QRG after more than 3
> minutes is never considered a dupe*
>
>
>
> $Spot::timegranularity = 60
>
> $Spot::dupage = 180
>
>
>
> You should know that a duplicate is the sum of the fields
> spotted+spotter+qrg+time+comments, which will generate a hash like this:
> |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM. For this reason, just using
> spotted+qrg+time is not sufficient to determine duplication.
>
>
>
> *>2. a new spot of a station 1kHz away from the previous spot is never
> considered a dupe.*
>
>
>
> $Spot::qrggranularity = 1
>
>
>
> *>3. self spots are never ever blocked*
>
>
>
> $Spot::minselfspotqrg = 0
>
>
>
>
>
> Example
>
>
>
> * Variables
>
>
>
> $Spot::timegranularity = 600    --> 10 min
>
> $Spot::qrggranularity = 25      --> 25 kHz
>
> $Spot::dupage = 3600            --> 1 h
>
>
>
> * Spot
>
>
>
> DX de IK2JTS:     7065.0  IZ0ARL       DTMBA I1823RM
> 0731Z (1737012660)
>
>
>
>
>
> Adjusts to the nearest multiple (either downward or upward)
>
>
>
> *1737012660*/600 = 2895021.1 --> 2895021 (round to the nearest integer)
>
> 2895021*600 = *1737012600*
>
>
>
> *7065.0*/25 = 282.6 --> 283 (round to the nearest integer)
>
> 283×25 = *7075*
>
>
>
> * dupefile
>
>
>
> |IZ0ARL|IK2JTS|*7075*||*1737012600*|DTMBAI1823RM = 16-Jan-2025 0731Z
> expires 16-Jan-2025 0831Z
>
>
>
> This means that time is fixed in 10-minute intervals, and frequency is
> fixed in 25 kHz steps.
>
>
>
>
>
> * Comparison with different values for the same spot:
>
>
>
> DX de IK2JTS:     *7065.0*  IZ0ARL       DTMBA #I-1823RM
> *0735Z* (*1737012900*)
>
>
>
> Scenario 1:
>
>
>
> $Spot::timegranularity = 600
>
> $Spot::qrggranularity = 25
>
> $Spot::dupage = 3600
>
>
>
> |IZ0ARL|IT9JZK|*7075*||*1737013200*|DTMBAI1823RM = 16-Jan-2025 *0735Z*
> expires 16-Jan-2025 *0835Z*
>
>                        (*0740Z*)
>
>
>
> Scenario 2:
>
>
>
> $Spot::timegranularity = 60
>
> $Spot::qrggranularity = 1
>
> $Spot::dupage = 300
>
>
>
> |IZ0ARL|IT9JZK|*7065*||*1737012900*|DTMBAI1823RM = 16-Jan-2025 *0735Z*
> expires 16-Jan-2025 *0740Z*
>
>                       (*0735Z*)
>
>
>
>
>
> I'm not sure if I’ve managed to explain myself clearly this time. This is
> my interpretation after analyzing the code, from someone who is not an
> expert. I think the person who can help us best is Dirk, the creator of
> this. And sorry because my English is terrible
>
>
>
> Kin EA3CV
>
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250117/0aaf4fd6/attachment-0001.htm>