<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EstiloCorreo18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ES link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>Hi Tim,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>I think Dirk will be able to solve the problem.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>But for those who want to remove duplicates for the time being, just do it:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>sort badip.torrelay | uniq > badip.torrelay<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>Or modify the line in crontab by putting the following:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>30 * * * * spawn('cd /spider/local_data; wget -qN http://www.dxspider.net/download/badip.torrelay<b>;sort badip.torrelay | uniq > badip.torrelay'</b>)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>Thanks for the info.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas;mso-fareast-language:EN-US'>Kin EA3CV<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b>De:</b> Dxspider-support <dxspider-support-bounces@tobit.co.uk> <b>En nombre de </b>du3tw via Dxspider-support<br><b>Enviado el:</b> viernes, 24 de febrero de 2023 6:46<br><b>Para:</b> dxspider-support@tobit.co.uk<br><b>CC:</b> charges.larder0p@icloud.com<br><b>Asunto:</b> [Dxspider-support] BadIP files - duplicate data.<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I looked at the excellent suggestion for updating bad <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>namely having this as a crontab <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.torexit'">http://www.dxspider.net/download/badip.torexit'</a>)<o:p></o:p></p></div><div><p class=MsoNormal>30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.torrelay'">http://www.dxspider.net/download/badip.torrelay'</a>)<o:p></o:p></p></div><div><p class=MsoNormal>30 * * * * spawn('cd /spider/local_data; wget -qN <a href="http://www.dxspider.net/download/badip.global'">http://www.dxspider.net/download/badip.global'</a>)<o:p></o:p></p></div><div><p class=MsoNormal>31 * * * * run_cmd('load/badip')<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>However, the source files contain main duplicates - which should be removed.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>cd /tmp<o:p></o:p></p></div><div><p class=MsoNormal>wget -qN <a href="http://www.dxspider.net/download/badip.torexit">http://www.dxspider.net/download/badip.torexit</a><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The number of lines in this file is calculated using "wc -l <a href="http://www.dxspider.net/download/badip.torexit">badip.torexit</a>", and outputs 1658 <o:p></o:p></p></div><div><p class=MsoNormal>Running through a basic de-dupe "sort badip.torexit | uniq | wc -l”, outputs 1173 <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>It would be more optimal if this data filtering is done on <a href="http://www.dxspider.net">www.dxspider.net</a> (he asked nicely)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>sort badip.torrelay | wc -l<o:p></o:p></p></div><div><p class=MsoNormal>9450<o:p></o:p></p></div><div><p class=MsoNormal>sort badip.torrelay | uniq | wc -l<o:p></o:p></p></div><div><p class=MsoNormal>8115<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> badip.global is already without duplicates having very few record in it.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Not sure who can process this suggestion ….<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> regards<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> Tim, DU3TW<o:p></o:p></p></div></div></body></html>