Many businesses are now relying on the Internet for continuous, uninterrupted communication with their suppliers and customers. Applications like email, instant messaging and even full Internet telephony systems demand constant network availability.
Many businesses are questioning whether the Internet is mature enough for this immense responsibility. The answer is a definite yes - but only when the right protocols and technologies are used. Selection of appropriate partners/suppliers is also important.
BGP (or BGPv4) is the protocol used for connecting a business network to multiple peers. These peers may be Internet Service Providers (ISPs), suppliers, customers or just other businesses in the same building or geographic region.
Many devices exist which claim to offer connectivity to multiple ISPs. Unless these devices implement the BGP protocol, they should be avoided. Some of these devices, despite hefty price tags, are severely limited in performance and flexibility. BGP is the only official standard used extensively within the Internet industry.
The mechanism we describe here is intended for businesses who want a genuine BGP solution that they can implement in-house, without buying high-end Cisco routers (such as the 7204 VXR NPE-300) that typically cost in excess of £10,000 each.
We use a regular PC/server as the BGP router, and a VLAN capable switch (Cisco 2950) to join many networks together.
The router runs the FreeBSD operating system and the `quagga' routing software. This software is mature and reliable. Some people ask me `isn't the Cisco solution better?'. My answer is `yes, but the FreeBSD solution is better than no BGP at all'. Using FreeBSD is about 90% cheaper than a complete Cisco solution.
The first important step is to decide which Internet providers will be used. Selecting two independent providers is a good idea, although you may choose just one provider and use BGP to link to the business next door, with a reciprocal arrangement that you can use their ISP if yours fails.
Technically, there is no reason why you can't do BGP over ISDN or ADSL connections. However, some ISPs refuse to provide BGP service over these connections. Despite the myths spread by their often uneducated sales staff, there are plenty of providers who will provide BGP over ADSL or S.HDSL.
There is also no technical reason why an ISP should charge a higher fee to customers using BGP. Although some argue that BGP customers require more advanced support, the reality is that customers using BGP are not the ones ringing up the help desk because they've plugged their modem cable into the ethernet port.
If DSL is one means of connectivity, try and get a second means of connectivity using another technology, such as WiFi, leased line or point-to-point laser.
Although your ISP will offer to do this for you (or may even do it without telling you), it is best for you to populate the WHOIS database yourself.
If you are in Europe, you will be using RIPE. Go to www.ripe.net. Click `Whois Database' on the left hand side. Now click `Online update facility (webupdates)'.
Some fields on the webupdates forms are quite strict. If a field you want is not on the screen, you can add it using the `Add a field' button at the bottom of the form. You will often have to do this to add the password field to a form.
Under `Create a new object' select person from the menu. Click `Add object'.
Fill out the fields using the template below:
person: Your full name
address: Your address
phone: Your phone (not mobile)
nic-hdl: AUTO-1
changed: your@email.com 20050101 (put today's date, of course)
source: RIPE
If you want to put a multi-line address, click the `+' next to `address' to add more lines.
Now click `Submit update'. On the next screen, you will be given the results. If successful, you will see the line:
Create SUCCEEDED: [person] NIC Your Name
The code displayed where it says NIC is your NIC handle. You will need this often - make a note of it.
A maintainer object requires an MD5 hashed password. Create this first, using these steps:
Go to the `Webupdates' page. Under `Create a new object', select `mntner'. The maintainer object is used to authorise changes to other objects you create, so it is very important.
Fill out the fields like so:
mntner: XYZ-MNT Instead of XYZ, choose a unique code or acronym
that relates to you or your business
descr: business name
admin-c: your NIC handle, that you created in the previous step
upd-to: your email address
auth: MD5-PW your password hash, that you created a moment ago
mnt-by: the same value as the mntner: field
referral-by: the same value as the mntner: field
changed: your@email.com 20050101 (use today's date)
source: RIPE
Now click `Submit update' to insert the maintainer object.
The person object you created earlier can be changed by anyone. This is not a good idea. Therefore, use the `Modify an existing object' feature of Webupdates to load your person object onto the screen again.
Beside `Add New Field', select `mnt-by'. Click `Add a field'
In the `mnt-by' field, put your maintainer handle.
Beside `Add new field', select `password'. Click `Add a field'
In the `password' field, put your password, in clear text.
Now click `Submit update'. No one else will be able to change this object now, unless they know the password for the maintainer object.
Using the Webupdates tool, create an object of type `organisation'.
organisation: AUTO-1
org-name: name of your business
org-type: NON-REGISTRY
address: mail address
e-mail: your@email.com
mnt-ref: your maintainer NIC
mnt-by: your maintainer NIC
password: your password
changed: your@email.com 20050101 (use today's date)
source: RIPE
When you submit the details, a NIC handle for your organisation will be generated automatically. Make a note of this - you will need it later.
Your IP addresses and AS number are also allocated through the WHOIS system. However, they have to be applied for by a RIPE member (typically your ISP will be a RIPE member).
Provider Independent (PI) addresses can be used with multiple providers concurrently. You can even leave the ISP that first allocated them to you and keep using them. The minimum quantity you need is 256 addresses (a /24 network).
An AS number is a unique identifier for each autonymous network that participates in the Internet. You need one.
When you apply for these details, you will need to tell the ISP these things:
Most ISPs will return the IP addresses and AS number to you in 3 - 5 days. Some will charge a fee for doing so. When the ISP gives you your AS number, ask them for theirs too - you will need it shortly.
Once you have been allocated the IP addresses and AS number, view the details in the WHOIS database. Make sure the ISP has linked them to your maintainer object and not theirs - otherwise, you won't be able to maintain the details yourself.
You now need to create an AS set and a route object.
The AS set is a convenient way of listing all AS numbers that you provide transit for. In many cases, this is only for yourself. However, if you provide IP transit to other businesses who you connect to (even if it's just as a failover service), you need to include their AS number in your AS set.
The NIC handle for the AS set will have a form like `AS-identity', where identity is a brief name or acronym, unique in the WHOIS database, that represents your business.
Use the Webupdates facility on RIPE to create the AS set object, like so:
as-set: The NIC handle you chose
descr: business name
members: Your AS number, prefixed with the letters AS, eg AS12345
members: include another `members' field listing the AS number
or AS set NIC handle for each network you provide transit to
tech-c: your personal NIC handle
admin-c: your personal NIC handle
mnt-by: your maintainer NIC handle
changed: your@email.com 20050101
password: your maintainer password
source: RIPE
When complete, submit the AS set object details.
Add the following record to the entry in your AS (aut-num) object:
export: to (provider1's AS) announce (The handle of our AS set)
export: to (provider2's AS) announce (The handle of our AS set)
An example is below:
export: to AS12345 announce AS-MYSETInform you providers that you have created an AS set. Ask them to include import records referencing your AS set in their AS aut-num entries.
Check your providers entries at RIPE, they should look like the example below. In the example, AS12345 is the provider and AS888 is you.
aut-num: AS12345
...
...
import: from AS888 accept AS-MYSET
export: to AS888 announce ANY
...
...
The route object describes routes to your networks.
Normally, you will not want to create route objects for networks smaller than 8 bits (/24). Although you might internally divide your IP address range into 2 or more subnets, you normally advertise your IP addresses to other networks as a single subnet.
Use Webupdates to create the route object like so:
route: A.B.C.D/24
descr: your business name
origin: your AS, eg AS12345
mnt-by: your maintainer NIC handle
changed: your@email.com 20050101
password: your maintainer password
notify: your@email.com
source: RIPE
Many ISPs have filtering systems that protect the Internet from packets containing forged IP addresses. These filters rely on the information contained in the WHOIS database. After you update the WHOIS database, it takes up to 24 hours before all ISPs have updated their filters to include specifications for your network. In the meantime, your will find that your connectivity is rather limited.
For the rest of this illustration, we will assume that you are using AS23456 and the provider independent IP range 213.228.220.0/24. Of course, these numbers already belong to someone else on the Internet so you should not use them. Wherever you see these numbers in the examples, please use your own numbers.
We will assume that that the server you will use as your core router has only 1 or 2 network interfaces. However, you want to use it to control 2 internal networks and manage connections to 4 external ISPs. To achieve this, we will connect each internal or external network to a port on a Cisco 2950 switch. We will then partition the ports into independent VLANs. Finally, we trunk all of the VLANs back to the router through a VLAN trunk port.
Beware of old switches: some older switches use proprietary trunking protocols for joining switches together. For instance, the Cisco 1924 series switches use Cisco's ISL protocol. This is not the same as the 802.1Q VLAN standard. To interoperate with FreeBSD or Linux routers, please only use switches that do 802.1Q VLAN trunking, including the Cisco 2950.
Here is a list of the VLANs and switch ports we will use for this example:
Port(s) | VLAN ID | Purpose | IP subnet/mask | Our Router IP |
---|---|---|---|---|
1 | 101 | ISP 1 | 20.40.60.96/30 | 20.40.60.98 |
2 | 102 | ISP 2 | 30.40.50.64/30 | 30.40.50.66/30 |
3 | 103 | ISP 3 | 40.50.60.128/30 | 40.50.60.130 |
4 | 104 | ISP 4 | 50.60.70.16/32 | 50.60.70.18 |
9 - 16 | 201 | Our servers and PCs | 213.228.220.0/26 | 213.228.220.1 |
22 | 301 | Hosted servers (belong to our clients) | 213.228.220.64/26 | 213.228.220.65 |
23 | 802.1Q Trunk port | Router | X | X |
Here is a sample configuration for a Cisco 2950. Please load this configuration through the serial port:
Current configuration : 2865 bytes ! version 12.1 no service pad service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname sw1 ! enable password my_pass ! username admin password my_pass ip subnet-zero vtp mode transparent ! spanning-tree mode pvst no spanning-tree optimize bpdu transmission spanning-tree extend system-id ! ! vlan 101,102,103,104,201,301 ! interface FastEthernet0/1 switchport access vlan 101 switchport mode access no ip address ! interface FastEthernet0/2 switchport access vlan 102 switchport mode access no ip address ! interface FastEthernet0/3 switchport access vlan 103 switchport mode access no ip address ! interface FastEthernet0/4 switchport access vlan 104 switchport mode access no ip address ! interface FastEthernet0/5 no ip address ! interface FastEthernet0/6 no ip address ! interface FastEthernet0/7 no ip address ! interface FastEthernet0/8 no ip address ! interface FastEthernet0/9 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/10 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/11 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/12 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/13 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/14 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/15 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/16 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/17 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/18 switchport access vlan 201 switchport mode access no ip address ! interface FastEthernet0/19 no ip address ! interface FastEthernet0/20 no ip address ! interface FastEthernet0/21 no ip address ! interface FastEthernet0/22 switchport access vlan 301 switchport mode access no ip address ! interface FastEthernet0/23 switchport mode trunk switchport nonegotiate no ip address ! interface FastEthernet0/24 no ip address ! interface Vlan1 no ip address ip broadcast-address 10.0.0.255 no ip route-cache shutdown ! interface Vlan201 ip address 213.228.220.2 255.255.255.0 ip broadcast-address 213.228.220.255 no ip route-cache ! ip default-gateway 213.228.220.1 no ip http server ! access-list 10 permit 213.228.220.0 0.0.0.63 access-list 10 deny any snmp-server community public RO 10 snmp-server community private RW 10 snmp-server location "Telehouse, London" ! line con 0 line vty 0 4 password my_admin login line vty 5 password my_admin login line vty 6 15 password my_admin login ! end
Select a machine meeting the following requirements:
Download FreeBSD 5.4 CDROM ISO image. Burn the image to a CD and proceed with the installation in a normal way. When prompted for packages, choose `binaries, kernel source only, and no X windows'.
We have chosen to install the following software packages onto FreeBSD:
Download those packages from the FreeBSD ports collection. Install them with the command `pkg_add'.
cd /usr/ports/net-mgmt/net-snmp
make build && make install
Kernel options needed:
otions DEVICE_POLLING
options HZ=1000
For /etc/sysctl.conf:
kern.polling.enable=1
sysinstall
Select > Configure > Distributions > Src > Select Base, Etc, Sys and all other required source code
cd /usr/src/sys/i386/conf
cp GENERIC GENERICPOLL
vi GENERICPOLL
cd /usr/src
make buildkernel KERNCONF=GENERICPOLL
make installkernel KERNCONF=GENERICPOLL
This is the content of /etc/rc.conf:
# Enable network daemons for user convenience. # Use ISP 1 as our default defaultrouter="20.40.60.97" gateway_enable="YES" hostname="r1.mydomain.com" # we don't set an IP address on fxp0, we use the VLANs instead ifconfig_fxp0="up media 100baseTX mediaopt full-duplex" # create VLANs cloned_interfaces="vlan101 vlan102 vlan103 vlan104 vlan201 vlan301" ifconfig_vlan101="inet 20.40.60.98 netmask 255.255.255.252 vlan 101 vlandev fxp0" ifconfig_vlan102="inet 30.40.50.66 netmask 255.255.255.252 vlan 102 vlandev fxp0" ifconfig_vlan102="inet 40.50.60.130 netmask 255.255.255.252 vlan 103 vlandev fxp0" ifconfig_vlan102="inet 50.60.70.18 netmask 255.255.255.252 vlan 104 vlandev fxp0" ifconfig_vlan201="inet 213.228.220.1 netmask 255.255.255.192 vlan 201 vlandev fxp0" ifconfig_vlan301="inet 213.228.220.65 netmask 255.255.255.192 vlan 301 vlandev fxp0" linux_enable="YES" sshd_enable="YES" quagga_enable="YES" quagga_flags="-d" quagga_daemons="zebra bgpd" snmpd_enable="YES" snmpd_flags="-a -p /var/run/snmpd.pid" snmptrapd_enable="YES" snmptrapd_flags="-a -p /var/run/snmptrapd.pid"
This is the content of /usr/local/etc/snmp/snmpd.conf:
ro community public 213.228.220.0/26 load 12.0 12.0 12.0 syslocation "Telehouse, London" syscontact "+44 207 1234 5678"
This is the content of /usr/local/etc/quagga/zebra.conf:
! ! Zebra configuration file ! hostname r1 password my_pass enable password my_pass ! log stdout ! !
This is the content of /usr/local/etc/bgpd.conf:
! ! hostname r1 password my_pass enable password my_pass ! router bgp 23456 bgp router-id 213.228.220.1 network 213.228.220.0/24 aggregate-address 213.228.220.0/24 neighbor 20.40.60.97 remote-as 120001 neighbor 20.40.60.97 next-hop-self neighbor 20.40.60.97 weight 1 neighbor 20.40.60.97 prefix-list a_self out neighbor 30.40.50.65 remote-as 120002 neighbor 30.40.50.65 next-hop-self neighbor 30.40.50.65 weight 1 neighbor 30.40.50.65 prefix-list a_self out neighbor 40.50.60.129 remote-as 120001 neighbor 40.50.60.129 next-hop-self neighbor 40.50.60.129 weight 1 neighbor 40.50.60.129 prefix-list a_self out neighbor 50.60.70.17 remote-as 120001 neighbor 50.60.70.17 next-hop-self neighbor 50.60.70.17 weight 1 neighbor 50.60.70.17 prefix-list a_self out ip prefix-list a_self permit 213.228.220.0/24 ip prefix-list a_self deny any ! log stdout
We modify this line in /etc/ttys so that we can connect over the serial port, just like a Cisco router:
ttyd0 "/usr/libexec/getty 3wire.9600" vt100 on secure
This is /boot/loader.conf, it causes boot messages to go over the serial port too:
console="comconsole"
After configuring all the files above, the easiest way to proceed is to reboot the machine. Upon booting, all the interfaces will be configured and quagga will attempt to get BGP routing tables from your ISPs.
Perform traceroutes to and from your router.
Use looking glass web sites to see how your routing information has been propogated.
Use MRTG, running on another host, to chart the network traffic through each port on the switch and the router.