[Dxspider-support] Disk useage problem

[Rene Olsen]

On 21 Oct 2007 at 23:16, Dirk Koopman wrote:

Hi Frank.

> Frank Johnson wrote:
> > HI All,
> > GB7NHT has been running reliably for several years now on a minimalistic 
> > Linux box with 2G hard disk with 73% stable disk usage.
> > Since upgrading to the latest CVS version, Spider has ground to a halt 
> > with no free disk space several times.
> > Each time, I had had a clear out, finally having to delete all the 
> > kernel source. Previously, I had deleted previous years spots and logs.
> > This time its 100% again and I cannot delete any more.
> > 
> > I have two very large user_asc and user_asc.o files and users.v2/3 files
> > 
> > -rw-rw-r--    1 sysop    spider   557596672 Oct 21 17:18 user_asc
> > -rw-rw-r--    1 sysop    spider   320348160 Oct 10 10:16 user_asc.o
> > -rw-rw-r--    1 sysop    spider    1806336 Feb  5  2004 users.v2
> > -rw-rw-r--    1 sysop    spider   17170432 Oct 21 17:23 users.v3
> > 
> 
> These are your problem. Sadly the solution which I put in place to solve 
> the problem of corrupt Berkeley DB files can sometimes also, themselves, 
> cause problems.
> 
> What you will find if you look at one of the user_asc files is that 
> large parts of it are rubbish (use "less user_asc" and stop it 
> calculating line numbers when it says), page down until you get to the 
> rubbish. Hopefully that will be a long way down.
> 
> What I suggest is that you get rid of of user_asc. Then stop the node 
> and see whether you can do:
> 
> rm users.v3
> perl user_asc.o
> 
> It will go so far and then spew loads of errors (note: there may be more 
> valid data after the rubbish) and it will then stop. Hopefully at the 
> end of it, you will have a working users.v3 file (you can get rid of the 
> users.v2 file).
> 
> Just for some verisimilitude, I would:
> 
> cd /spider/perl
> ./update_sysop.pl
> 
> and then restart the node and see what happens.
> 
> If you can't get most of the user file back then we will need to resort 
> to some splitting and editing, but hopefully it won't come to that.

If space is an issue I would guess that the debug files is you problem. With newer versions 
of spider they have increased in size like 10-20 MB a day I would say. On my node the 
debug files for the 10 days that are kept is about 600 MB. Am pretty sure it used to be 
around 400 MB or so.

If this is actually the problem, there is some setting which tells spider how many days back to 
keep these debug files. Don't remember exactly where that setting is though.

Vy 73 de Rene / OZ1LQH