Load balanced, redundant network configuration for Linux using ECMP, Quagga, BGP and OSPF

Consider this diagram:

r1 and r2 are routers (possibly running FreeBSD & Quagga, as described here, or perhaps Cisco 7204/7206)
Each router is connected to both ISPs. Hopefully, each ISP also has each connection handled by a unique switch and router.
The two switches each service a unique network segment, possibly using private IP addresses (192.168.1.0/24 and 192.168.2.0/24)
The organisation has a block of real IP addresses which it want's external hosts to interact with. Each internal host is assigned one of the real addresses on it's loopback interface with a /32 netmask, so that no NAT is necessary.
The servers are each running Linux. The Linux kernel is compiled with Equal Cost Multi-Path routing enabled (configuration option CONFIG_IP_ROUTE_MULTIPATH=y). This means that the kernel will permit multiple default gateways in the routing table, and will load balance outgoing traffic across them.
Routing entries, including the default gateway addresses, are not specified manually on any of the servers or routers. Instead, Quagga is running on every server. On the internal servers, Quagga uses OSPF to discover the router addresses, and then creates the default routes.
If either route becomes unavailable, because of router failure, switch failure or network card/cable failure, or during maintenance, the OSPF protocol will detect the failure within 5 seconds. When failure is detected, the route is automatically deleted from the routing table, so the kernel won't continue sending packets via that route.
Using OSPF to detect failure is ideal, because OSPF relies on packets sent at layer 3 - the IP layer. If, for instance, the cable between the 2950 and router r2 was removed, all other hosts connected to the 2950 would still feel that there was an electrical connection through the switch, and would continue sending some of their outbound packets through that switch. However, OSPF will detect this situation correctly.

Configuration details

We assume each host is running Debian Linux (except the routers). The real IP of the host is to be substituted where you see A.B.C.D in the examples.

Install Quagga on each host with the commands:

      apt-get update
      apt-get install quagga iproute

Put the following in /etc/network/interfaces:

auto lo
iface lo inet loopback
  up ip addr add dev lo A.B.C.D/32 scope global

# notice that we use `manual' rather than `static', so that we can
# over-ride the scope parameter
auto eth0
iface eth0 inet manual
  up ip link set dev eth0 up
  up ip addr add dev eth0 192.168.1.10/24 scope link

auto eth1
iface eth1 inet manual
  up ip link set dev eth1 up
  up ip addr add dev eth1 192.168.2.10/24 scope link

Now put the following in /etc/quagga/zebra.conf:

      hostname www1
      password changeme
      enable password changeme

      interface lo
        ip address 127.0.0.1/8
        ip address A.B.C.D/32       (this is your server's real IP)

      interface eth0
        ip address 192.168.1.10/24
        multicast

      interface eth1
        ip address 192.168.2.10/24
        multicast

     !log file /var/log/quagga/zebra.log

This is /etc/quagga/ospfd.conf

hostname www1
password changeme
enable password changeme

interface eth0
 no ip ospf authentication-key
 ip ospf hello-interval 2
 ip ospf dead-interval 5

interface eth1
 no ip ospf authentication-key
 ip ospf hello-interval 2
 ip ospf dead-interval 5

router ospf
  ospf router-id A.B.C.D
  network 192.168.1.0/24 area 0
  network 192.168.2.0/24 area 0

!log file /var/log/quagga/ospfd.log

Modify /etc/quagga/daemons.conf: set zebra=yes and ospfd=yes

Testing

Once configured, reboot your host.

Type ip route and you should see a list of routes showing multiple default gateways.

Try unplugging one of the routers - then check the routing table on one of the servers. After 5 seconds, the references to the unplugged router should be gone from the routing tables.

Troubleshooting

Make sure Equal Cost Multi-path is enabled in each Linux kernel
Make sure Multicast is enabled in the kernel. Check that the network cards and drivers support multicast - some don't
Make sure that you don't have iptables blocking the OSPF packets. OSPF sends packets using IP protocol 89 (not TCP or UDP). This allows OSPF:
```
iptables --insert INPUT -s 192.168.0.0/16 --protocol ospf -j ACCEPT
```