[Dxspider-support] Self-spotting and de-duplication
Dirk Koopman
djk at tobit.co.uk
Tue Jan 21 12:22:59 GMT 2025
With apologies to Kin: PLEASE do not do this.
I am working on sorting this out at the moment. There will be a release
soon.
73 Dirk G1TLH
The DXDupe.pm module is used for all sorts of other purposes other than
deduping spots. They all have their own timeouts and strategies(hah!)
for removing stale objects in THEIR dupe
On 17/01/2025 13:31, IZ2LSC via Dxspider-support wrote:
> Thanks Kin,
> I will try it on my backup (isolated) cluster node.
>
> 73
> Andrea
> iz2lsc
>
> -->
>
>
> On Fri, Jan 17, 2025 at 12:55 PM Kin <ea3cv at cronux.net> wrote:
>
> MorningAndrea,
>
> Whilewaiting for Dirk to confirm the possible bug and its
> solution, I'm passing you a modified version of DXDupe.pm that
> fixes the deletion of hashes depending on the $Spot::dupage variable.
>
> Italso solves the problem of deleting expired spots.
>
> Itshould be copied to: /spider/Local and restart the node.
>
> Kin EA3CV
>
> *De:*Dxspider-support <dxspider-support-bounces at tobit.co.uk> *En
> nombre de *IZ2LSC via Dxspider-support
> *Enviado el:* viernes, 17 de enero de 2025 10:18
> *Para:* The DXSpider Support list <dxspider-support at tobit.co.uk>
> *CC:* IZ2LSC <iz2lsc.andrea at gmail.com>
> *Asunto:* Re: [Dxspider-support] Self-spotting and de-duplication
>
> If every time a spot is ignored is also added again to the dupe
> list, this is a loop. Maybe is this the bug?
>
> 1737101764^(pc11) PCPROT: Duplicate Spot IZ2LSC-97: PC61
> 7006.0|JF4IQS/1|1737101700|JF4IQS|IZ2LSC-97 ignored
>
> view_dupes JF4IQS/1
> |JF4IQS/1|JF4IQS|433060||1737100440|POTAJP1149FMHIK = 17-Jan-2025
> 0754Z expires 17-Jan-2025 0824Z
> |JF4IQS/1|JF4IQS|7005||1737100800|POTAJP1149CWHIK = 17-Jan-2025
> 0801Z expires 17-Jan-2025 0831Z
> |JF4IQS/1|JF4IQS|7005|[-POTA-] JP-114 = 17-Jan-2025 0801Z expires
> 17-Jan-2025 0831Z
> |JF4IQS/1|JI2OJV|7005||1737101580|POTAJP1149CWHIK = 17-Jan-2025
> 0814Z expires 17-Jan-2025 0844Z
> |JF4IQS/1|JI2OJV|7005|[-POTA-] JP-114 = 17-Jan-2025 0814Z expires
> 17-Jan-2025 0844Z
> *_|JF4IQS/1|JF4IQS|7005||1737101700|POTAJP1149CWHIK = 17-Jan-2025
> 0816Z expires 17-Jan-2025 0846Z_*
>
> -->
>
> On Thu, Jan 16, 2025 at 11:01 PM IZ2LSC <iz2lsc.andrea at gmail.com>
> wrote:
>
> Kin,
>
> when the hash is expired, is it removed from the file?
>
> I can see hash already expired and still present in the file.
>
> Not sure if this is expected.
>
> Andrea
>
> -->
>
> On Thu, Jan 16, 2025 at 8:12 PM Björn Ekelund via
> Dxspider-support <dxspider-support at tobit.co.uk> wrote:
>
> Thanks. For human spots it makes perfect sense to include
> the comment.
>
> Björn SM7IUN
>
> On Thu, Jan 16, 2025 at 7:06 PM Kin <ea3cv at cronux.net> wrote:
>
> Björn,
>
> All this is for non-RBN spots.
>
> I don't remember now the treatment of RBN spots.
>
> Kin
>
> *De:*Björn Ekelund <bjorn at ekelund.nu>
> *Enviado el:* jueves, 16 de enero de 2025 18:26
> *Para:* Kin <ea3cv at cronux.net>
> *CC:* The DXSpider Support list
> <dxspider-support at tobit.co.uk>
> *Asunto:* Re: [Dxspider-support] Self-spotting and
> de-duplication
>
> Kin,
>
> this was very clear and very helpful.
>
> Thank you for spending the time to do it.
>
> I am sure others appreciate it too.
>
> In my opinion, all comments should be ignored when
> determining dupes for skimmer spots.
>
> Björn SM7IUN
>
> On Thu, Jan 16, 2025 at 10:28 AM Kin
> <ea3cv at cronux.net> wrote:
>
> Morning Björn,
>
> />So timegranularity defines a window within which
> spots are considered to have /
>
> />the same time, Does this then effectively define
> the minimum respotting time?/
>
> Yes, timegranularity effectively defines the
> minimum respotting time. It determines the size of
> the time window within which spots are considered
> to have the same timestamp. For example, if
> timegranularity is set to 600 seconds (10
> minutes), any spots reported within the same
> 10-minute interval will be grouped together,
> effectively preventing additional spots from being
> considered separate unless they occur in a
> different interval.
>
> />Or is this the function of dupage? If it is,
> then I do not see the purpose of quantizing /
>
> />time into timegranularity steps./
>
> timegranularity is about standardizing time
> representation, while dupage enforces the minimum
> time between duplicate spots.
>
> />And how can 600 be 4 minutes? Is time counted in
> steps of 0.4s?/
>
> />Does this mean that dupage 3600 is 24 minutes?/
>
> //
>
> />And what is the function of qrggranularity? My
> empirical experiments say that I need /
>
> />to move 10kHz away for DXSpider to not consider
> a new spot a dupe. Not 25kHz./
>
> I hope that the following examples will clarify
> how it works.
>
> />All I'm trying to do is to set up my node so that /
>
> //
>
> />1. a new spot of the same station on the same
> QRG after more than 3 minutes is never considered
> a dupe/
>
> $Spot::timegranularity = 60
>
> $Spot::dupage = 180
>
> You should know that a duplicate is the sum of the
> fields spotted+spotter+qrg+time+comments, which
> will generate a hash like this:
> |IZ0ARL|IK2JTS|7075||1737012600|DTMBAI1823RM. For
> this reason, just using spotted+qrg+time is not
> sufficient to determine duplication.
>
> />2. a new spot of a station 1kHz away from the
> previous spot is never considered a dupe./
>
> $Spot::qrggranularity = 1
>
> />3. self spots are never ever blocked/
>
> $Spot::minselfspotqrg = 0
>
> Example
>
> * Variables
>
> $Spot::timegranularity = 600 --> 10 min
>
> $Spot::qrggranularity = 25 --> 25 kHz
>
> $Spot::dupage = 3600 --> 1 h
>
> * Spot
>
> DX de IK2JTS: 7065.0 IZ0ARL DTMBA
> I1823RM 0731Z (1737012660)
>
> Adjusts to the nearest multiple (either downward
> or upward)
>
> *1737012660*/600 = 2895021.1 --> 2895021 (round to
> the nearest integer)
>
> 2895021*600 = *1737012600*
>
> *7065.0*/25 = 282.6 --> 283 (round to the nearest
> integer)
>
> 283×25 = *7075*
>
> * dupefile
>
> |IZ0ARL|IK2JTS|*7075*||*1737012600*|DTMBAI1823RM =
> 16-Jan-2025 0731Z expires 16-Jan-2025 0831Z
>
> This means that time is fixed in 10-minute
> intervals, and frequency is fixed in 25 kHz steps.
>
> * Comparison with different values for the same spot:
>
> DX de IK2JTS: *7065.0* IZ0ARL DTMBA
> #I-1823RM *0735Z* (*1737012900*)
>
> Scenario 1:
>
> $Spot::timegranularity = 600
>
> $Spot::qrggranularity = 25
>
> $Spot::dupage = 3600
>
> |IZ0ARL|IT9JZK|*7075*||*1737013200*|DTMBAI1823RM =
> 16-Jan-2025 *0735Z* expires 16-Jan-2025 *0835Z*
>
> (*0740Z*)
>
> Scenario 2:
>
> $Spot::timegranularity = 60
>
> $Spot::qrggranularity = 1
>
> $Spot::dupage = 300
>
> |IZ0ARL|IT9JZK|*7065*||*1737012900*|DTMBAI1823RM =
> 16-Jan-2025 *0735Z* expires 16-Jan-2025 *0740Z*
>
> (*0735Z*)
>
> I'm not sure if I’ve managed to explain myself
> clearly this time. This is my interpretation after
> analyzing the code, from someone who is not an
> expert. I think the person who can help us best is
> Dirk, the creator of this. And sorry because my
> English is terrible
>
> Kin EA3CV
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20250121/307de45f/attachment-0001.htm>
More information about the Dxspider-support
mailing list