<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I looked at the excellent suggestion for updating bad <div class=""><br class=""></div><div class="">namely having this as a crontab </div><div class=""><br class=""></div><div class=""><div class="">30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.torexit'" class="">http://www.dxspider.net/download/badip.torexit'</a>)</div><div class="">30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.torrelay'" class="">http://www.dxspider.net/download/badip.torrelay'</a>)</div><div class="">30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.global'" class="">http://www.dxspider.net/download/badip.global'</a>)</div><div class="">31 * * * * run_cmd('load/badip')</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">However, the source files contain main duplicates - which should be removed.</div><div class=""><br class=""></div><div class="">cd /tmp</div><div class="">wget -qN <a href="http://www.dxspider.net/download/badip.torexit" class="">http://www.dxspider.net/download/badip.torexit</a></div><div class=""><br class=""></div><div class="">The number of lines in this file is calculated using "wc -l <a href="http://www.dxspider.net/download/badip.torexit" class="">badip.torexit</a>", and outputs 1658 </div><div class="">Running through a basic de-dupe "sort badip.torexit | uniq | wc -l”, outputs 1173 </div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">It would be more optimal if this data filtering is done on <a href="http://www.dxspider.net" class="">www.dxspider.net</a> (he asked nicely)</div><div class=""><br class=""></div><div class=""><div class="">sort badip.torrelay | wc -l</div><div class="">9450</div><div class="">sort badip.torrelay | uniq | wc -l</div><div class="">8115</div></div><div class=""><br class=""></div><div class=""> badip.global is already without duplicates having very few record in it.</div><div class=""><br class=""></div><div class="">Not sure who can process this suggestion ….</div><div class=""><br class=""></div><div class=""> regards</div><div class=""><br class=""></div><div class=""> Tim, DU3TW</div></body></html>