[Cluster-tech] new 'super' network + protocol

[Dirk Koopman]

It is now getting toward the time where we need to think about where to
take the dxcluster network. It is becoming too big, cross-connected (and
at the same time fragmented) for normal sysops to connect to or manage
safely.

DXSpider is now accepting a form of deliberate looping for PC16/17/19/21
routing protocol as well as providing filtering for that. This has
merely highlighted some more of the problems.

The problem is that, although this works reasonably well (in so far as
it goes), it doesn't actually succeed in solving some of the core issues
(such as transparent talk to anyone visible on the network). Another
issue it improves, but doesn't completely solve is providing full
redundancy together with full backwards compatibility (with ak1a for
instance).

Another issue that has arisen out of the ability to loop more than
previously is that setting the software up has become the equivalent of
mananging firewall IP filtering rules - with all the pitfalls that that
entails. Good self training, in an arcane sort of way, but probably only
interesting to head-bangers like me.

Having studied the dynamics of the WWW cluster for a couple of years and
watching what happens, I propose the following:-

1) I will write a 'protocol router' program which will be placed in
strategic parts of the world. This router network is designed to provide
a uniform, standard, FLAT, fully described ak1a interface to the
outside. FLAT means that the router looks like one super large network
node. Experience has shown me that this is the only view that is
reliably compatible with all the DX Cluster programs that are out there.
2) The only connections that will be allowed are AK1A PC protocol
(unless the AR-Cluster people want to join in and send me the
specification of the AR sentences that they use). Authentication with
passwords will be available.

3) these routers will be cross connected via the Internet (or other
RELIABLE IP based networks) so that failure of any one router doesn't
affect any of the others. Obviously full deduplication and other
features will be built in.

4) 'Networks of Nodes'(NoN) will connect to their nearest 'router' (more
than one connection will be possible). 

5) The NoNs should be be 'leaf' networks, that is they should not cross
connect to other 'leaf' networks directly but use the router network via
full, active (for clx sysops) connections. The nodes in a NoN should
have 'active' connections in a tree like manner to their parents. 

6) It is crucial that NoN's are configured so that full protocol can be
maintained from 'their' router(s) outwards. Using 'passive' or 'spot
sucker' links in the path to 'their router' is completely pointless and
defeats the whole point of this network. The routers provide the
redundancy and reliability, the NoN's responsibility is only to make the
connection to 'their' routers reliable. 

7) The router will have full route filtering capability and can provide
'partial' views of the WWW network, but participants are strongly
encouraged to take as 'full' a view (as far 'down' each NoN tree) as
possible.
 
8) Even with partial visibility it will be possible to use the router as
a 'gateway' for things like 'talk', msgs etc. It is likely that
heuristics involving various pattern matches will be applied to
announces to convert (most of) them into talks. Hopefully also the much
greater visibility of users will encourage greater use of 'talk'.

9) The routers will have extensive conferencing facilities which will
use PC12 announce messages in 'closed user group' mode to allow multiple
conferences to be active simultaneously.

10) one of the advantages of such a network is that those nodes that
merely wish to suck spots can do so AND get it in nice regular PC11
format and not have to cope with differing user presentations.  

In case anybody thinks this is another two year job, it is simply the
current version of DXSpider cut about (probably quite a bit) and recast
in a different form. I estimate about 3 man-weeks of work to get a
'working' prototype out the door.

It will continue to be written in perl and be as platform independant as
it now is (ie runs on various flavours of unix and M$ Windows).

All protocols will be published. The software will continue to be in the
public domain and available from sourceforge.

Discuss.

Dirk G1TLH