<div dir="ltr">Hi,<div>Yesterday I had 10 cluster restarts.<br>I saw that when running a sh/dx/30 sometimes the cluster was rebooted.<br>I have reviewed the debug file and observed that in most of them there was a sh/dx nnn or sh/dx/nnn before the error message and subsequent restart.<br>On 3 occasions the machine reached 100% cpu and had to be restarted for not responding.<br>I have rebuilt user_asc, but never find error. <br></div><div><br></div><div>HW <span style="color:rgb(0,0,0);font-family:az_ea_font,"Segoe UI",az_font,system-ui,-apple-system,BlinkMacSystemFont,Roboto,Oxygen,Ubuntu,Cantarell,"Open Sans","Helvetica Neue",sans-serif;font-size:13px;white-space:nowrap">1 cpu, 2 GiB memory</span></div><div>load average: 0.03, 0.01, 0.00<br>Memory 7 %</div><div>OS Ubuntu Ubuntu 18.04.4 LTS</div><div><br>I do not know what I can do.<br></div><div><br></div><div>Kin EA3CV</div><div><br></div><div>...</div><div>1589661271^(chan) <- I EA1FU sh/dx 150<br>1589661271^(chan) -> D EA1FU EA3CV-2><br>1589661271^(chan) <- I EA1FU echo #announce#<br>1589661271^(chan) -> D EA1FU #announce#<br>1589661271^(chan) -> D EA1FU EA3CV-2><br>1589661271^(chan) <- I EA1FU echo #wcy#<br>1589661271^(chan) -> D EA1FU #wcy#<br>1589661271^(chan) -> D EA1FU EA3CV-2><br>1589661271^(chan) <- I EA1FU echo #wwv#<br>1589661271^(chan) -> D EA1FU #wwv#<br>1589661271^(chan) -> D EA1FU EA3CV-2><br>1589661271^(chan) <- I EA1FU echo #ready#<br>1589661271^(chan) -> D EA1FU #ready#<br>1589661271^(chan) -> D EA1FU EA3CV-2><br>1589661281^Magic number checking on storable string failed at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661281^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661281^ eval {...} called at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>1589661281^ Storable::thaw("") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661281^ eval {...} called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661281^ Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55a0f3a9d578)) called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br><a href="http://itter.pm">itter.pm</a> line 15<br>1589661281^ Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55a0f3a9d578), "close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br><a href="http://tream.pm">tream.pm</a> line 27<br>...<br><br>1589661549^(chan) <- I EA1FU sh/dx 150<br>1589661549^(chan) -> D EA1FU EA3CV-2><br>1589661549^(chan) <- I EA1FU echo #announce#<br>1589661549^(chan) -> D EA1FU #announce#<br>1589661549^(chan) -> D EA1FU EA3CV-2><br>1589661549^(chan) <- I EA1FU echo #wcy#<br>1589661549^(chan) -> D EA1FU #wcy#<br>1589661549^(chan) -> D EA1FU EA3CV-2><br>1589661549^(chan) <- I EA1FU echo #wwv#<br>1589661549^(chan) -> D EA1FU #wwv#<br>1589661549^(chan) -> D EA1FU EA3CV-2><br>1589661549^(chan) <- I EA1FU echo #ready#<br>1589661549^(chan) -> D EA1FU #ready#<br>1589661549^(chan) -> D EA1FU EA3CV-2><br>1589661562^Magic number checking on storable string failed at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661562^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661562^ eval {...} called at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>1589661562^ Storable::thaw("") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661562^ eval {...} called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661562^ Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x555d8eb49fa0)) called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br><a href="http://itter.pm">itter.pm</a> line 15<br>1589661562^ Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x555d8eb49fa0), "close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br><a href="http://tream.pm">tream.pm</a> line 27<br>...<br><br>1589661762^(chan) <- I EA1FU sh/dx 150<br>1589661762^(chan) -> D EA1FU EA3CV-2><br>1589661762^(chan) <- I EA1FU echo #announce#<br>1589661762^(chan) -> D EA1FU #announce#<br>1589661762^(chan) -> D EA1FU EA3CV-2><br>1589661762^(chan) <- I EA1FU echo #wcy#<br>1589661762^(chan) -> D EA1FU #wcy#<br>1589661762^(chan) -> D EA1FU EA3CV-2><br>1589661762^(chan) <- I EA1FU echo #wwv#<br>1589661762^(chan) -> D EA1FU #wwv#<br>1589661762^(chan) -> D EA1FU EA3CV-2><br>1589661762^(chan) <- I EA1FU echo #ready#<br>1589661762^(chan) -> D EA1FU #ready#<br>1589661762^(chan) -> D EA1FU EA3CV-2><br>1589661785^(msg) queue msg (0)<br>1589661785^Magic number checking on storable string failed at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661785^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>1589661785^ eval {...} called at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>1589661785^ Storable::thaw("") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661785^ eval {...} called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>1589661785^ Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55f98142d4f0)) called at /usr/local/share/perl/5.26.1/Mojo/EventEm<br><a href="http://itter.pm">itter.pm</a> line 15<br>1589661785^ Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55f98142d4f0), "close") called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/S<br><a href="http://tream.pm">tream.pm</a> line 27<br>...<br><br>1589673726^(chan) <- I EA7A SH/DX/120<br>1589673726^(chan) -> D EA7A EA3CV-2><br>1589673728^(chan) <- I AE5E PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H94^<br>1589673728^(chan) -> D SV5FRI-1 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D WB3FFV-2 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D N6WS-6 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D VE9SC PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D N0OJ-7 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D F1OYP-6 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D IW9EDP-6 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) -> D F4EYQ-1 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) <- I WB3FFV-2 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H92^<br>1589673728^(chan) <- I SV5FRI-1 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H94^<br>1589673728^(chan) <- I VE9SC PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H95^<br>1589673728^(chan) <- I IW9EDP-6 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H93^<br>1589673728^(chan) <- I F1OYP-6 PC92^F5LEN-7^128^D^^1RA4NHY^1HB9GFJ^1SQ7OTA^H90^<br>1589673741^(msg) queue msg (0)<br>1589673995^Git: 1.57-226-g3b932bf2<br>1589673995^Git: V=1.57 S= B=226 g=3b932bf2<br>1589673995^DXSpider V1.57, build 226 (git: 3b932bf2[r]) started<br>1589673995^Copyright (c) 1998-2020 Dirk Koopman G1TLH<br>1589673995^loading prefixes ...<br>...<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El jue., 14 may. 2020 a las 23:08, Dirk Koopman via Dxspider-support (<<a href="mailto:dxspider-support@tobit.co.uk">dxspider-support@tobit.co.uk</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Stop the node. <br>
<br>
Rebuild the user file "cd /spider/local_data; perl user_asc"<br>
<br>
restart the node<br>
<br>
I am getting really sick of Storable and DB_File. I am doing some
benchmarking at the moment for a viable replacement. I have a
Storable replacement, The DB_File replace could be an issue.<br>
<br>
Part of the problem is I suspect that something that runs in a
spawned command (like sh/dx, sh/log etc) is doing a $user->put
(a write) which will likely screw up DB_File. I have had a good
look for this, but it only seems to have started when I did the
abortive attempt to store ip addresses. Ir *may* be that there is
lingering corruption there from that attempt.<br>
<br>
But the first thing anyone should do if this sort of thing happens
is always: rebuild the user file (as above) and delete the
dupefile. <br>
<br>
Dirk<br>
<br>
On 14/05/2020 21:18, Joaquin . via Dxspider-support wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
I don't know what is happening but in the last two days I have
had dxspider restarts since I went to build 226.<br>
<br>
This is the last thing before restarting the cluster:<br>
<br>
1589403382^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426.<br>
1589403382^ at /usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm
line 426.<br>
1589403382^ eval {...} called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426<br>
1589403382^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589403382^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm
line 15<br>
1589403382^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm line 27<br>
1589403382^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589403382^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589403382^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm
line 145<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line 145<br>
1589403382^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55bfd2c61c18),
"I/O watcher", undef, 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 5<br>
2<br>
1589403382^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55bfd46984c0), 3)
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 30<br>
1589403382^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm line 134<br>
1589403382^ Mojo::IOLoop::start("Mojo::IOLoop") called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a> line 781<br>
1589403382^ main::start_node() called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
1589403382^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426,<br>
1589403382^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426, at
/spider/perl/Log/Agent.pm line 21.<br>
1589403382^ Log::Agent::logcroak("Magic number checking on
storable string failed at /usr/lib/x"...) called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 427<br>
1589403382^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589403382^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm
line 15<br>
1589403382^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm line 27<br>
1589403382^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589403382^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55bfd54011b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589403382^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm
line 145<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line 145<br>
1589403382^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55bfd2c61c18),
"I/O watcher", "", 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 52<br>
1589403382^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55bfd46984c0), 3)
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589403382^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 30<br>
1589403382^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55bfd2c61c18))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm line 134<br>
1589403382^ Mojo::IOLoop::start("Mojo::IOLoop") called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a> line 781<br>
1589403382^ main::start_node() called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
1589403382^(chan) DXChannel EC5A destroyed (16)<br>
1589403382^(chan) DXChannel EA4SE destroyed (15)<br>
1589403382^(chan) DXChannel EA3CV-1 destroyed (14)<br>
1589403382^(chan) DXChannel EA1FU destroyed (13)<br>
1589403382^ (in cleanup) (in cleanup) at
/spider/perl/Msg.pm line 566 during global destruction.<br>
<br>
<br>
<br>
1589484851^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426,
<GEN1666> line 110,<br>
1589484851^Magic number checking on storable string failed at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 426,
<GEN1666> line 110, at /spider/perl/Log/Agent.pm li<br>
ne 21, <GEN1666> line 110.<br>
1589484851^ Log::Agent::logcroak("Magic number checking on
storable string failed at /usr/lib/x"...) called at
/usr/lib/x86_64-linux-gnu/perl/5.26/Storable.pm line 427<br>
1589484851^ Storable::thaw("") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 68<br>
1589484851^
Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at /usr/local/share/perl/5.26.1/Mojo/EventEmitter.pm
line 15<br>
1589484851^
Mojo::EventEmitter::emit(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8),
"close") called at
/usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm line 27<br>
1589484851^
Mojo::IOLoop::Stream::close(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 110<br>
1589484851^
Mojo::IOLoop::Stream::_read(Mojo::IOLoop::Stream=HASH(0x55d2e94c95b8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop/Stream.pm
line 58<br>
1589484851^
Mojo::IOLoop::Stream::__ANON__(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8))
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm
line 145<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/Poll.pm line 145<br>
1589484851^
Mojo::Reactor::Poll::_try(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8),
"I/O watcher", 1, 0) called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 52<br>
1589484851^
Mojo::Reactor::EV::__ANON__(EV::IO=SCALAR(0x55d2e930bcf0), 3)
called at /usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line
30<br>
1589484851^ eval {...} called at
/usr/local/share/perl/5.26.1/Mojo/Reactor/EV.pm line 30<br>
1589484851^
Mojo::Reactor::EV::start(Mojo::Reactor::EV=HASH(0x55d2e6c8d1d8))
called at /usr/local/share/perl/5.26.1/Mojo/IOLoop.pm line 134<br>
1589484851^ Mojo::IOLoop::start("Mojo::IOLoop") called at
/spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a> line 781<br>
1589484851^ main::start_node() called at /spider/perl/<a href="http://cluster.pl" target="_blank">cluster.pl</a>
line 798<br>
<br>
The version is:DXSpider V1.57, build 226 (git: 3b932bf2[r])<br>
Any ideas?<br>
</div>
<div dir="ltr"><br>
</div>
<div>73 de Kin</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Dxspider-support mailing list
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a href="mailto:Dxspider-support@tobit.co.uk" target="_blank">Dxspider-support@tobit.co.uk</a><br>
<a href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support" rel="noreferrer" target="_blank">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote></div>