Some years ago, a group of four schools needed to upgrade their internet connections, but due to their locations the options available were relatively expensive leased lines.  They realised that costs could be reduced by installing a single connection to one of the schools and building a wide area network between the sites.  This would also allow the inter-school network traffic to flow across the new, high bandwidth wide area network (WAN), rather than being routed between the sites over the internet connections.  A further advantage was that the boarding houses could be provided with connectivity to both the internet and the schools' servers by connecting them into the new WAN backbone.

The WAN had been installed and was operational for a few years but had seen several problems regarding its resilience which needed to be addressed.  We were brought in to redesign the infrastructure from the ground up, reusing old equipment where appropriate and installing new equipment where necessary.  The schools were also planning on changing to a new internet service provider, and this gave us more scope for unifying the network under a single IP addressing scheme, thereby reducing the complexity of the routing and eliminating the need for complex network address translation (NAT) to allow the sites to interoperate.

The primary interconnects between the sites were laser links backed up with lower bandwidth microwave links, with the automatic failover handled by spanning tree protocol.  The use of spanning tree protocol to handle the link failover had proved to be difficult to fine tune and debug.  Since the original installation, cheaper internet connections were becoming available, and there was a possibility that the individual sites may later opt to install their own internet connections in the future, whilst maintaining the WAN for the inter-school traffic.

Due to these considerations, we decided that the best course of action was to replace the existing WAN infrastructure with a dynamically routed IP network.  The existing laser and microwave hardware was reused, but rather than each end being terminated at an ethernet switch to perform the spanning tree based failover they were connected to Linux based routers.  The routers were configured to handle routing between the sites using BGP - the boarder gateway protocol used for dynamic routing across the internet.  BGP was chosen as the routing protocol because in the event that the individual sites installed their own internet connections in the future, the inter-school routing could be extended out via the internet connections to improve the network's resilience.

The router at each site was configured to handle that site's firewalling requirements, with a simple web interface to allow the customer to adjust their own firewall rules.  Performing the firewalling at the border of each individual site, rather than at a centralised location, allowed the firewalling requirements to be largely divorced from the routing - it doesn't matter how the traffic arrives at the firewall, whether it be from a local internet connection or any of the WAN interconnects, it will always be handled by the firewall in the same way.  This greatly simplified the firewalling configuration, prevented one school from being able to accidentally break one of the other schools' networks and ensured the sites were properly segregated from eachother so as to keep them secure.  The decision was also made to keep network address translation to an absolute minimum, and never on the inter-school traffic, so the router at the border with the internet was configured to perform this function.

The network has performed well since being installed, and problems with the interconnects have been much easier to debug than with the previous system.  The end-to-end connectivity provided by the wholey routed IP network with minimal NAT has also made it trivial for inter-school services, such as voice-over-IP telephones, to be installed.  The design of the WAN backbone has proved to be very extensible, allowing more of the remote sites, such as boarding houses, to be linked back to it.  A remote site can be connected in to any part of the backbone, and can communicate with its associated school, rather than needing a direct connection; this has allowed connectivity to be extended to remote sites using line-of-sight technologies, even when the site has no direct line of sight with its parent school.  The costs of laying cables between the sites would be prohibitively high, so the benefit of line-of-sight technologies, such as laser interconnects, are clear.