[Dxspider-support] The RBN interface is here...

Mon Jul 13 01:11:00 CEST 2020

Erm...

Not certain what you are counting (i.e. how much TCP protocol you are 
adding in). A normal spot length (one that hasn't got a long frequency 
or skimmer call, is 75 bytes, then add \r\n. A TCP header per spot is 
another 40 bytes but, at full chat there will buffering so, on average, 
TCP overhead will be less than that. Here are some numbers:

Hour

RBN In

Spots Out

Percentage

00:00

3378

269

  8.0%

01:00

7714

580

  7.5%

02:00

8611

661

  7.7%

03:00

7859

554

  7.0%

04:00

14192

796

  5.6%

05:00

6896

564

  8.2%

06:00

8198

669

  8.2%

07:00

6146

572

  9.3%

08:00

7536

742

  9.8%

09:00

8644

732

  8.5%

10:00

8783

733

  8.3%

11:00

19083

1335

  7.0%

12:00

120252

5389

  4.5%

13:00

116299

4994

  4.3%

14:00

113989

5072

  4.4%

15:00

113946

4975

  4.4%

16:00

116948

4891

  4.2%

17:00

122389

5145

  4.2%

18:00

129974

5395

  4.2%

19:00

138272

5566

  4.0%

20:00

150322

5524

  3.7%

21:00

142756

5137

  3.6%

22:00

145415

5109

  3.5%

23:00

146128

4961

  3.4%

00:00

131657

4781

  3.6%

01:00

124652

4630

  3.7%

02:00

126464

4703

  3.7%

03:00

121697

4386

  3.6%

04:00

121391

4183

  3.4%

05:00

113955

4188

  3.7%

06:00

100910

3936

  3.9%

07:00

91241

3845

  4.2%

08:00

96423

3903

  4.0%

09:00

108047

4264

  3.9%

10:00

122572

4709

  3.8%

11:00

129074

5040

  3.9%

12:00

9583

709

  7.4%

13:00

8209

765

  9.3%

14:00

10739

1019

  9.5%

15:00

9875

936

  9.5%

16:00

9557

914

  9.6%

17:00

11926

1071

  9.0%

18:00

12970

1096

  8.5%

19:00

15086

1222

  8.1%

20:00

14340

1043

  7.3%

This is from some stats for the CW Skimmer that  I collected in DXSpider 
over this weekend (so far).  The "spots out" figure is the number of 
spots that I have left after I have done my de-duping thing. A user will 
get only that number if they have asked for all spots and have no 
filters. So, in principle, each user could get the full "spots out" 
number. Most people will likely have some kind of by_zone filter on. But 
I can't give any likely numbers, this time, because the number DXSpider 
users receiving were not large enough to get a meaningful feel of how 
well it is likely to cope, but there does not seem to be any obvious 
difference in CPU between no RBN and RBN+2/3 users, I am reasonably 
confident that it will cope.

But I was curious as to how many I could stuff through - unoptimised.  
So I did some benchmarking.

So analysing today's debug file and extracting just the CW Skimmer's RBN 
spots with 100 simulated users being sent the curated spots gives:

***
*** Totals: raw = 1882260, candidates = 539920, delivered = 76926, 
percentage of raw delivered =   4.087%
***
*** Total  elapsed time = 90.18 secs for RBN activity of any kind, each 
delivery to 100 users = 0.001172 secs or 853 / sec
***

It's a bit optimistic but not by too much. It should cope with 42 / sec 
at the peak. But I *still* think that it is an egregious waste of 
bandwidth just to get 4% "useful" (contest) data. I am also musing to 
myself: what happens if most of the DXSpider nodes start sucking data 
from the RBN? I am guessing that they will be running out of NIC 
bandwidth with another 150 or so DXSpider nodes (out of ~250 odd). At my 
estimate of 77 bytes (not including TCP overhead) that's 4+GB (that's B, 
not b) altogether with 200 other users. That's 5 10Gb NICs bonded 
together into a router, in turn connected to quite a fast pipe to the 
Internet, even by today's standards :-) I'd love to see the AWS bill.

I enclose the csv for your Excel manipulating pleasure (or some other 
competent spreadsheet program [amongst which LibreOffice does NOT figure]).

Dirk

On 12/07/2020 22:17, Lee Sawkins via Dxspider-support wrote:
> Oops sorry.  Multiplied by 4 instead of 2.  Each spot is 150 bytes.  Twice the number of characters.  4,000,000 x 150
>   
> Lee VE7CC
>
> ----- Original Message -----
> From: Lee Sawkins via Dxspider-support <dxspider-support at tobit.co.uk>
> To: rene chr olsen <rene.chr.olsen at gmail.com>
> Cc: Lee Sawkins <ve7cc at shaw.ca>, dxspider-support at tobit.co.uk
> Sent: Sun, 12 Jul 2020 15:13:41 -0600 (MDT)
> Subject: Re: [Dxspider-support] The RBN interface is here...
>
> Each spot from the RBN is 300 bytes.  Busy times can see over 4 million incoming spots per day.  4,000,000 x 300 bytes in.  Then add the outgoing data.
>
> Lee VE7CC
>
> ----- Original Message -----
> From: Rene Olsen via Dxspider-support <dxspider-support at tobit.co.uk>
> To: dxspider-support at tobit.co.uk, Dirk Koopman <djk at tobit.co.uk>
> Cc: Rene Olsen <rene.chr.olsen at gmail.com>
> Sent: Sun, 12 Jul 2020 14:16:12 -0600 (MDT)
> Subject: Re: [Dxspider-support] The RBN interface is here...
>
> Hi Dirk.
>
> Yes, I think that I will upgrade to mojo branch no matter what.
>
> I am still having doubts about the RBN thing though. As I wrote earlier I am on a limited plan,
> which means I pay a very low price per month. I am only allowed 1 TB of data a month.
>
> I was logged on the the RBN network port 7000, and it said that the average at that time was
> 17000 spots an hour. And that is probably at the low end when contests are there. This is a
> normal sunday evening without any contests (that I am aware of).
>
> 17000 spots an hour, I count a spot as 80 bytes, means that just with that I will be using
> around 1 TB of data in 30 days. The next plan at my provider is a lot more money, so guess I
> am bowing out on the RBN stuff. Unless my calculations are way off.
>
> I guess in busy contest weekends the number can increase to 50000 spots an hour or more.
>
> Vy 73 de René / OZ1LQH
>
> On 10 Jul 2020 at 21:22, Dirk Koopman wrote:
>
>> On 10/07/2020 20:52, Rene Olsen wrote:
>>> Thanks Dirk.
>>>
>>> Thanks for the info. I will not touch it now since there is the contest this weekend, and I will
>>> not be home to monitor what happens.
>>>
>>> Will this new RBN data increase the bw used by the node significantly? If we are talking 100
>>> spots a second, like someone mentioned, I am not sure it is for me, since I have limited bw
>>> useage.
>>>
>>> Vy 73 de René / OZ1LQH
>> That is the $64,000 question. If you stick to CW spots, so far, I have
>> seen about ~10/sec during the last big contest. We will see what this
>> weekend brings for a more definite answer. I would be quite surprised to
>> see 100 spots/sec or 8200 bytes/sec. But remember, the most you are
>> likely see coming out is ~820 bytes / sec / RBN Spot user and. even on
>> the last big CW contest the actual "curated" output was 1 or 2 / sec,
>> but not consistently. 30000 incoming spots / hour would be whittled down
>> to about 3000/hour or about 8 every 10 seconds.
>>
>> I am waiting with bated breath how WA9PIE-2 copes with 1000+ users.
>> Human spots is no problem at all, it trundles along at about 2-7% CPU on
>> a big contest.  But it is unlikely that (anything like) that many of
>> them will using the RBN feed - at least this weekend. But we will see.
>> 1000 * 820 is how many bytes/sec again...?
>>
>> But I would strongly recommend that you upgrade, just for the improved
>> speed and reduced power consumption. Whether you connect to the RBN is a
>> separate issue. The 'mojo' branch is where it is all at. The 'master'
>> branch is now just in care and maintenance. All new features will go
>> into the 'mojo' branch (there is a clue as to what it is in UPGRADE.mojo),
>>
>> 73 Dirk G1TLH
>>
>
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support
>
>
> _______________________________________________
> Dxspider-support mailing list
> Dxspider-support at tobit.co.uk
> https://mailman.tobit.co.uk/mailman/listinfo/dxspider-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20200713/057fb56d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: a.csv
Type: text/csv
Size: 1061 bytes
Desc: not available
URL: <https://mailman.tobit.co.uk/pipermail/dxspider-support/attachments/20200713/057fb56d/attachment-0001.csv>