<div dir="ltr">If it doesn't fail or reboot, I will do it tonight.<br>At the moment I have updated a copy of the same version and configuration in a local VM, with real traffic and users with different filters.<br><br>The machine of the problems is a Microsoft Azuce VM, and according to them the CPU is: Intel® Haswell 2.4 GHz E5-2673 v3 processors or better. Specifically "Standard B1ms (1 vcpus, 2 GiB memory)"<br><br>I attach a photo of how the system load was yesterday. I got to not respond twice and had to do a hard reset.<br><br>I will report with the results.<br><br>Thanks Dirk.<br><br>Kin<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El dom., 17 may. 2020 a las 12:50, Dirk Koopman via Dxspider-support (<<a href="mailto:dxspider-support@tobit.co.uk">dxspider-support@tobit.co.uk</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>I have back ported a mod to the
Mojo::IOLoop::Subprocess to use the serialisation method that I
was using with the previous (now deprecated) method
(..::ForkCall). I strong suspect that it should fix this problem.
In the meantime I am successfully carrying on <br>
<br>
<br>
Do an update and see whether it fixes your problem. This is a
stepping to stone to a more stable solution to user file errors as
well as errors of this sort, by eliminating Storage as a
serialisation mechanism. <br>
<br>
Please recreate your users file before restarting with this new
version. <br>
<br>
Remind me, what type of cpu are you running this on? I don't see
these errors, but I am running Intel hardware 64bit ubuntu 18.04.<br>
<br>
Dirk<br>
<br>
On 17/05/2020 02:17, Joaquin . via Dxspider-support wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div>Yesterday I had 10 cluster restarts.<br>
I saw that when running a sh/dx/30 sometimes the cluster was
rebooted.<br>
I have reviewed the debug file and observed that in most of
them there was a sh/dx nnn or sh/dx/nnn before the error
message and subsequent restart.<br>
On 3 occasions the machine reached 100% cpu and had to be
restarted for not responding.<br>
I have rebuilt user_asc, but never find error. <br>
</div>
<div><br>
</div>
<div>HW <span>1
cpu, 2 GiB memory</span></div>
<div>load average: 0.03, 0.01, 0.00<br>
Memory 7 %</div>
<div>OS Ubuntu Ubuntu 18.04.4 LTS</div>
<div><br>
I do not know what I can do.<br>
</div>
<div><br>
</div>
<div>Kin EA3CV</div>
<div><br>
</div>
<div>...</div>
<div>1589661271^(chan) <- I EA1FU sh/dx 150<br>
1589661271^(chan) -> D EA1FU EA3CV-2><br>
1589661271^(chan) <- I EA1FU echo #announce#<br>
1589661271^(chan) -> D EA1FU #announce#<br>
1589661271^(chan) -> D EA1FU EA3CV-2><br>
1589661271^(chan) <- I EA1FU echo #wcy#<br>
1589661271^(chan) -> D EA1FU #wcy#<br>
1589661271^(chan) -> D EA1FU EA3CV-2><br>
1589661271^(chan) <- I EA1FU echo #wwv#<br>
1589661271^(chan) -> D EA1FU #wwv#<br>
1589661271^(chan) -> D EA1FU EA3CV-2><br>
1589661271^(chan) <- I EA1FU echo #ready#<br>
1589661271^(chan) -> D EA1FU #ready#<br>
1589661271^(chan) -> D EA1FU EA3CV-2><br>
1589661281^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>
1589661281^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm
line 426.<br>
1589661281^ eval {...} called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>
1589661281^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661281^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661281^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55a0f3a9d578))
called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br>
<a href="http://itter.pm" target="_blank">itter.pm</a>
line 15<br>
1589661281^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55a0f3a9d578),
"close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br>
<a href="http://tream.pm" target="_blank">tream.pm</a>
line 27<br>
...<br>
<br>
1589661549^(chan) <- I EA1FU sh/dx 150<br>
1589661549^(chan) -> D EA1FU EA3CV-2><br>
1589661549^(chan) <- I EA1FU echo #announce#<br>
1589661549^(chan) -> D EA1FU #announce#<br>
1589661549^(chan) -> D EA1FU EA3CV-2><br>
1589661549^(chan) <- I EA1FU echo #wcy#<br>
1589661549^(chan) -> D EA1FU #wcy#<br>
1589661549^(chan) -> D EA1FU EA3CV-2><br>
1589661549^(chan) <- I EA1FU echo #wwv#<br>
1589661549^(chan) -> D EA1FU #wwv#<br>
1589661549^(chan) -> D EA1FU EA3CV-2><br>
1589661549^(chan) <- I EA1FU echo #ready#<br>
1589661549^(chan) -> D EA1FU #ready#<br>
1589661549^(chan) -> D EA1FU EA3CV-2><br>
1589661562^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>
1589661562^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm
line 426.<br>
1589661562^ eval {...} called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>
1589661562^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661562^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661562^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x555d8eb49fa0))
called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br>
<a href="http://itter.pm" target="_blank">itter.pm</a>
line 15<br>
1589661562^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x555d8eb49fa0),
"close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br>
<a href="http://tream.pm" target="_blank">tream.pm</a>
line 27<br>
...<br>
<br>
1589661762^(chan) <- I EA1FU sh/dx 150<br>
1589661762^(chan) -> D EA1FU EA3CV-2><br>
1589661762^(chan) <- I EA1FU echo #announce#<br>
1589661762^(chan) -> D EA1FU #announce#<br>
1589661762^(chan) -> D EA1FU EA3CV-2><br>
1589661762^(chan) <- I EA1FU echo #wcy#<br>
1589661762^(chan) -> D EA1FU #wcy#<br>
1589661762^(chan) -> D EA1FU EA3CV-2><br>
1589661762^(chan) <- I EA1FU echo #wwv#<br>
1589661762^(chan) -> D EA1FU #wwv#<br>
1589661762^(chan) -> D EA1FU EA3CV-2><br>
1589661762^(chan) <- I EA1FU echo #ready#<br>
1589661762^(chan) -> D EA1FU #ready#<br>
1589661762^(chan) -> D EA1FU EA3CV-2><br>
1589661785^(msg) queue msg (0)<br>
1589661785^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>
1589661785^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm
line 426.<br>
1589661785^ eval {...} called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>
1589661785^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661785^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589661785^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55f98142d4f0))
called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br>
<a href="http://itter.pm" target="_blank">itter.pm</a>
line 15<br>
1589661785^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55f98142d4f0),
"close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br>
<a href="http://tream.pm" target="_blank">tream.pm</a>
line 27<br>
...<br>
<br>
1589673726^(chan) <- I EA7A SH/DX/120<br>
1589673726^(chan) -> D EA7A EA3CV-2><br>
1589673728^(chan) <- I AE5E
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H94^<br>
1589673728^(chan) -> D SV5FRI-1
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D WB3FFV-2
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D N6WS-6
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D VE9SC
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D N0OJ-7
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D F1OYP-6
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D IW9EDP-6
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) -> D F4EYQ-1
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) <- I WB3FFV-2
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H92^<br>
1589673728^(chan) <- I SV5FRI-1
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H94^<br>
1589673728^(chan) <- I VE9SC
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H95^<br>
1589673728^(chan) <- I IW9EDP-6
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>
1589673728^(chan) <- I F1OYP-6
PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H90^<br>
1589673741^(msg) queue msg (0)<br>
1589673995^Git: 1.57-226-g3b932bf2<br>
1589673995^Git: V=1.57 S= B=226 g=3b932bf2<br>
1589673995^DXSpider V1.57, build 226 (git: 3b932bf2[r])
started<br>
1589673995^Copyright (c) 1998-2020 Dirk Koopman G1TLH<br>
1589673995^loading prefixes ...<br>
...<br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">El jue., 14 may. 2020 a las
23:08, Dirk Koopman via Dxspider-support (<<a href="mailto:dxspider-support@tobit.co.uk" target="_blank">dxspider-support@tobit.co.uk</a>>)
escribió:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Stop the node. <br>
<br>
Rebuild the user file "cd /spider/local_data; perl
user_asc"<br>
<br>
restart the node<br>
<br>
I am getting really sick of Storable and DB_File. I am
doing some benchmarking at the moment for a viable
replacement. I have a Storable replacement, The DB_File
replace could be an issue.<br>
<br>
Part of the problem is I suspect that something that runs
in a spawned command (like sh/dx, sh/log etc) is doing a
$user->put (a write) which will likely screw up
DB_File. I have had a good look for this, but it only
seems to have started when I did the abortive attempt to
store ip addresses. Ir *may* be that there is lingering
corruption there from that attempt.<br>
<br>
But the first thing anyone should do if this sort of thing
happens is always: rebuild the user file (as above) and
delete the dupefile. <br>
<br>
Dirk<br>
<br>
On 14/05/2020 21:18, Joaquin . via Dxspider-support wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
I don't know what is happening but in the last two
days I have had dxspider restarts since I went to
build 226.<br>
<br>
This is the last thing before restarting the cluster:<br>
<br>
1589403382^Magic number checking on storable string
failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426.<br>
1589403382^ at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426.<br>
1589403382^ eval {...} called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426<br>
1589403382^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589403382^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm line
15<br>
1589403382^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 27<br>
1589403382^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589403382^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589403382^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589403382^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55bfd2c61c18),
"I/O watcher", undef, 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 5<br>
2<br>
1589403382^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55bfd46984c0),
3) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm
line 134<br>
1589403382^ Mojo::IOLoop::start("Mojo::IOLoop")
called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 781<br>
1589403382^ main::start_node() called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
1589403382^Magic number checking on storable string
failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426,<br>
1589403382^Magic number checking on storable string
failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426, at /spider/perl/Log/Agent.pm line 21.<br>
1589403382^ Log::Agent::logcroak("Magic number
checking on storable string failed at /usr/lib/x"...)
called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
427<br>
1589403382^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589403382^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm line
15<br>
1589403382^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 27<br>
1589403382^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589403382^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589403382^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589403382^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55bfd2c61c18),
"I/O watcher", "", 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
52<br>
1589403382^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55bfd46984c0),
3) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm
line 134<br>
1589403382^ Mojo::IOLoop::start("Mojo::IOLoop")
called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 781<br>
1589403382^ main::start_node() called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
1589403382^(chan) DXChannel EC5A destroyed (16)<br>
1589403382^(chan) DXChannel EA4SE destroyed (15)<br>
1589403382^(chan) DXChannel EA3CV-1 destroyed (14)<br>
1589403382^(chan) DXChannel EA1FU destroyed (13)<br>
1589403382^ (in cleanup) (in cleanup) at
/spider/perl/Msg.pm line 566 during global
destruction.<br>
<br>
<br>
<br>
1589484851^Magic number checking on storable string
failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426, <GEN1666> line 110,<br>
1589484851^Magic number checking on storable string
failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
426, <GEN1666> line 110, at
/spider/perl/Log/Agent.pm li<br>
ne 21, <GEN1666> line 110.<br>
1589484851^ Log::Agent::logcroak("Magic number
checking on storable string failed at /usr/lib/x"...)
called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line
427<br>
1589484851^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm
line 68<br>
1589484851^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at
/usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm line
15<br>
1589484851^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 27<br>
1589484851^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589484851^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589484851^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8))
called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line
145<br>
1589484851^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8),
"I/O watcher", 1, 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
52<br>
1589484851^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55d2e930bcf0),
3) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589484851^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm
line 134<br>
1589484851^ Mojo::IOLoop::start("Mojo::IOLoop")
called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 781<br>
1589484851^ main::start_node() called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
<br>
The version is:DXSpider V1.57, build 226 (git:
3b932bf2[r])<br>
Any ideas?<br>
</div>
<div dir="ltr"><br>
</div>
<div>73 de Kin</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Dxspider-support mailing list
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a><br>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" rel="noreferrer" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Dxspider-support mailing list
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a><br>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" rel="noreferrer" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote></div>