<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Try <font face="monospace">ulimit -Hn
</font>On my machine (64GB RAM, 64bit Debian 12 bookworm) that
gives: 1048576 (file handles).<br>
<br>
If that is the case (or some similar large number) I am guessing
that you are using a distro that uses systemd, in which case the
official way of dealing with this is to:<br>
<br>
<font face="monospace">sudo vi /etc/security/limits.d/numproc.conf<br>
</font><br>
and in that file, write something like:<br>
<br>
<font face="monospace">#<domain> <type>
<item> <value><br>
#<br>
sysop soft nofile 2048<br>
</font><br>
save that file, then restart the whole machine.<br>
<br>
Then open a shell and check that the ulimit has changed.<br>
<br>
Dirk G1TLH<br>
<br>
On 20/04/2025 22:41, Keith, G6NHU wrote:<br>
</div>
<blockquote type="cite"
cite="mid:74743ac4-18f6-4e81-9de5-2e5ca1554a6d@Spark">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title></title>
<div name="messageBodySection">
<div dir="auto">Second day in a row, a crash and restart as per
below.<br>
<br>
Dirk, I know you suggested increasing ulimit in the shell
which I thought I’d done yesterday (see my email from
yesterday afternoon) but it appears not. <br>
<br>
I assume you’ve got around this with WA9PIE-2 so I’d really
appreciate some help please. I don’t want the node rebooting
regularly like this, especially not at weekends but I don’t
know what to do next. The node runs as a service so how can
I launch it from a shell script? I am still relatively
inexperienced with Linux so I just don’t know what to do
here. <br>
<br>
Thanks,<br>
<br>
73 Keith.<br>
<br>
<br>
<strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(trace) writing /spider/local_data/rbn_cache Too
many open files</span><span style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(trace) Stack (2): RBN::DXDebug::confess in
/spider/perl/RBN.pm line: 926</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(trace) Stack (3): RBN::RBN::write_cache in
/spider/perl/RBN.pm line: 871</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(trace) Stack (4): main::RBN::per_minute in
/spider/perl/cluster.pl line: 902</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) DXSpider Ceasing</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) DXQSL finished</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) RBN:WRITE_CACHE size: 377.687KB time to write:
34 mS</span><span style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) DXDupe finishing</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) DXUser finished</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(cluster) DXSpider v1.57 build 615 (git:
mojo/9f7fb47f[r]) using perl v5.38.2 on linux ended</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^RING:
19:37:13^(*) bye bye everyone - bye bye</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#b42419;font-family:Menlo;font-size: 11px">^###</span><span
style="font-size: 11px"><br>
</span><strong
style="color:#b42419;font-family:Menlo;font-size: 11px">1745177833</strong><span
style="color:#000000;font-family:Menlo;font-size: 11px">^###
RINGBUFFER END 501 debug lines written</span><span
style="font-size: 11px"><br>
</span></div>
</div>
<div name="messageReplySection">On 19 Apr 2025 at 16:27 +0100,
Keith, G6NHU via Dxspider-support
<a class="moz-txt-link-rfc2396E" href="mailto:dxspider-support@tobit.co.uk"><dxspider-support@tobit.co.uk></a>, wrote:<br>
<blockquote type="cite"
style="border-left-color:#1abc9c; margin:5px 5px; padding-left:10px; border-left-width:thin; border-left-style:solid;"
class="spark_indent">
<div name="messageBodySection">
<div dir="auto"><span
style="font-family:Arial;font-size: medium">This has
just happened again to me, this time on the DO droplet
that’s running the node.</span><span
style="font-size: medium"><br>
</span><span style="font-size: medium"><br>
</span><span style="font-family:Arial;font-size: 16px">I’ve
added </span><span
style="font-family:Arial;font-size: 16px">ulimit -S -n
65536 to .bashrc in both sysop and root logins as per
here: </span><a
href="https://askubuntu.com/questions/1492277/on-ubuntu-22-04-editing-limits-conf-to-increase-number-of-file-descriptors-does"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://askubuntu.com/questions/1492277/on-ubuntu-22-04-editing-limits-conf-to-increase-number-of-file-descriptors-does</a><span
style="font-size: 16px"><br>
</span><span style="font-size: 16px"><br>
</span><span style="font-family:Arial;font-size: 16px">Hopefully
that’ll fix it.</span><span style="font-size: 16px"><br>
</span><span style="font-size: 16px"><br>
</span><span style="font-family:Arial;font-size: 16px">73
Keith.</span><span style="font-size: 16px"><br>
</span><span style="font-size: 16px"><br>
</span>1745074921^###<br>
1745074921^### RINGBUFFER END 501 debug lines written<br>
1745074921^###<br>
1745074921^(trace) can't open /spider/local_data/wcy/param
Too many open files<br>
1745074921^(trace) Stack (2): WCY::DXDebug::confess in
/spider/perl/WCY.pm line: 79<br>
1745074921^(trace) Stack (3): WCY::WCY::store in
/spider/perl/WCY.pm line: 123<br>
1745074921^(trace) Stack (4): DXProt::WCY::update in
/spider/perl/DXProtHandle.pm line: 1775<br>
1745074921^(trace) Stack (5): DXProt::DXProt::handle_73 in
/spider/perl/DXProt.pm line: 466<br>
1745074921^(trace) Stack (6): DXChannel::DXProt::normal in
/spider/perl/DXChannel.pm line: 746<br>
1745074921^(trace) Stack (7):
DXChannel::DXChannel::process_one in
/spider/perl/DXChannel.pm line: 239<br>
1745074921^(trace) Stack (8): main::DXChannel::rec in
/spider/perl/<a href="http://cluster.pl" target="_blank"
moz-do-not-send="true">cluster.pl</a> line: 424<br>
1745074921^(trace) Stack (9): ExtMsg::main::__ANON__ in
/spider/perl/ExtMsg.pm line: 120<br>
1745074921^(trace) Stack (10): Msg::ExtMsg::dequeue in
/spider/perl/Msg.pm line: 500<br>
1745074921^(trace) Stack (11): ExtMsg::Msg::_rcv in
/spider/perl/ExtMsg.pm line: 83<br>
1745074921^(trace) Stack (12): Msg::ExtMsg::_rcv in
/spider/perl/Msg.pm line: 511<br>
1745074921^(*) DXSpider Ceasing<br>
1745074921^(*) DXQSL finished<br>
1745074921^(*) RBN:WRITE_CACHE size: 357.289KB time to
write: 31 mS<br>
1745074921^(*) DXDupe finishing<br>
1745074921^(*) DXUser finished<br>
1745074921^(cluster) DXSpider v1.57 build 615 (git:
mojo/9f7fb47f[r]) using perl v5.38.2 on linux ended<br>
1745074921^(*) bye bye everyone - bye bye</div>
</div>
<div name="messageReplySection">On 23 Mar 2025 at 23:08 +0000,
djk via Dxspider-support
<a class="moz-txt-link-rfc2396E" href="mailto:dxspider-support@tobit.co.uk"><dxspider-support@tobit.co.uk></a>, wrote:<br>
<blockquote type="cite"
style="border-left-color:#e67e22; margin:5px 5px; padding-left:10px; border-left-width:thin; border-left-style:solid;"
class="spark_indent">
<p>There is standard limit of 1024 files open at once per
process. You can change this in a shell with 'ulimit -n
2048' (for example). There is also way of changing it
system wide in systemd (<spit>) but you'll have to
research that yourself or start the node in a shell
script like:</p>
#!/bin/sh<br>
ulimit -n 2048<br>
/spider/perl/cluster.pl
<p>Personally, 900+ users on a 4GB RPi x is going some,
especially considering power required to run some
windows cluster software (and then still not keeping).</p>
<p>What does your 'top' say when you are running it at
this sort of usage?</p>
<p>Dirk G1TLH<br>
</p>
<div class="moz-cite-prefix">On 23/03/2025 18:54, Keith,
G6NHU via Dxspider-support wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2133a6f9-b6b6-459c-b65f-8964ac3fc45a@Spark"
style="border-left-color:#3498db; margin:5px 5px; padding-left:10px; border-left-width:thin; border-left-style:solid;"
class="spark_indent">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div name="messageBodySection">
<div dir="auto">I suppose this really is for Dirk.<br>
<br>
This has never happened before - I came into the
shack with a freshly poured shackbeer and noticed my
ssh session had closed so I logged back in and saw
my uptime was just 21 minutes.<br>
<br>
Checking the debug log (attached as a .zip), this is
what happened in the same timestamp with the actual
error that caused the crash being at the end.<br>
<br>
My cluster is running on a Pi5 with 4Gb RAM and an
external Samsung SSD. I don’t know the exact
number of connected users but when I logged back on,
there were 938 so I’d imagine the number prior to
the crash was around the same. The node had been
up for about a month.<br>
<br>
“Too many open files” ?<br>
<br>
73 Keith<br>
<br>
1742753087^###<br>
1742753087^### RINGBUFFER END 501 debug lines
written<br>
1742753087^###<br>
1742753087^(trace) writing
/spider/local_data/rbn_cache Too many open files<br>
1742753087^(trace) Stack (2): RBN::DXDebug::confess
in /spider/perl/RBN.pm line: 926<br>
1742753087^(trace) Stack (3): RBN::RBN::write_cache
in /spider/perl/RBN.pm line: 871<br>
1742753087^(trace) Stack (4): main::RBN::per_minute
in /spider/perl/<a href="http://cluster.pl"
target="_blank" moz-do-not-send="true">cluster.pl</a> line:
892<br>
1742753087^(*) DXSpider Ceasing<br>
1742753087^(*) DXQSL finished<br>
1742753087^(*) RBN:WRITE_CACHE size: 423.804KB time
to write: 26 mS<br>
1742753087^(*) DXDupe finishing<br>
1742753087^(*) DXUser finished<br>
1742753087^(cluster) DXSpider v1.57 build 568 (git:
mojo/0920a333[r]) using perl v5.36.0 on linux ended<br>
1742753087^(*) bye bye everyone - bye bye<br>
1742753087^###<br>
1742753087^### RINGBUFFER START at line 0 (zero
base)<br>
1742753087^###<br>
<br>
Then it repeats<br>
<br>
1742753087^RING: 18:04:47^(trace) writing
/spider/local_data/rbn_cache Too many open files<br>
1742753087^RING: 18:04:47^(trace) Stack (2):
RBN::DXDebug::confess in /spider/perl/RBN.pm line:
926<br>
1742753087^RING: 18:04:47^(trace) Stack (3):
RBN::RBN::write_cache in /spider/perl/RBN.pm line:
871<br>
1742753087^RING: 18:04:47^(trace) Stack (4):
main::RBN::per_minute in /spider/perl/<a
href="http://cluster.pl" target="_blank"
moz-do-not-send="true">cluster.pl</a> line: 892<br>
1742753087^RING: 18:04:47^(*) DXSpider Ceasing<br>
1742753087^RING: 18:04:47^(*) DXQSL finished<br>
1742753087^RING: 18:04:47^(*) RBN:WRITE_CACHE size:
423.804KB time to write: 26 mS<br>
1742753087^RING: 18:04:47^(*) DXDupe finishing<br>
1742753087^RING: 18:04:47^(*) DXUser finished<br>
1742753087^RING: 18:04:47^(cluster) DXSpider v1.57
build 568 (git: mojo/0920a333[r]) using perl v5.36.0
on linux ended<br>
1742753087^RING: 18:04:47^(*) bye bye everyone - bye
bye<br>
1742753087^###<br>
1742753087^### RINGBUFFER END 501 debug lines
written<br>
1742753087^###</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
Dxspider-support mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:Dxspider-support@tobit.co.uk"
moz-do-not-send="true">Dxspider-support@tobit.co.uk</a>
<a class="moz-txt-link-freetext"
href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support"
moz-do-not-send="true">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a>
</pre>
</blockquote>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Dxspider-support@tobit.co.uk">Dxspider-support@tobit.co.uk</a><br>
<a class="moz-txt-link-freetext" href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote>
</div>
_______________________________________________<br>
Dxspider-support mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Dxspider-support@tobit.co.uk">Dxspider-support@tobit.co.uk</a><br>
<a class="moz-txt-link-freetext" href="https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support">https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support</a><br>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>