Process for upgrading to BGP

Introduction

Many businesses are now relying on the Internet for continuous, uninterrupted communication with their suppliers and customers. Applications like email, instant messaging and even full Internet telephony systems demand constant network availability.

Many businesses are questioning whether the Internet is mature enough for this immense responsibility. The answer is a definite yes - but only when the right protocols and technologies are used. Selection of appropriate partners/suppliers is also important.

BGP (or BGPv4) is the protocol used for connecting a business network to multiple peers. These peers may be Internet Service Providers (ISPs), suppliers, customers or just other businesses in the same building or geographic region.

Many devices exist which claim to offer connectivity to multiple ISPs. Unless these devices implement the BGP protocol, they should be avoided. Some of these devices, despite hefty price tags, are severely limited in performance and flexibility. BGP is the only official standard used extensively within the Internet industry.

Our implementation of BGP

The mechanism we describe here is intended for businesses who want a genuine BGP solution that they can implement in-house, without buying high-end Cisco routers (such as the 7204 VXR NPE-300) that typically cost in excess of £10,000 each.

We use a regular PC/server as the BGP router, and a VLAN capable switch (Cisco 2950) to join many networks together.

The router runs the FreeBSD operating system and the `quagga' routing software. This software is mature and reliable. Some people ask me `isn't the Cisco solution better?'. My answer is `yes, but the FreeBSD solution is better than no BGP at all'. Using FreeBSD is about 90% cheaper than a complete Cisco solution.

Step 1: Choose your partners and means of connectivity

The first important step is to decide which Internet providers will be used. Selecting two independent providers is a good idea, although you may choose just one provider and use BGP to link to the business next door, with a reciprocal arrangement that you can use their ISP if yours fails.

Technically, there is no reason why you can't do BGP over ISDN or ADSL connections. However, some ISPs refuse to provide BGP service over these connections. Despite the myths spread by their often uneducated sales staff, there are plenty of providers who will provide BGP over ADSL or S.HDSL.

There is also no technical reason why an ISP should charge a higher fee to customers using BGP. Although some argue that BGP customers require more advanced support, the reality is that customers using BGP are not the ones ringing up the help desk because they've plugged their modem cable into the ethernet port.

If DSL is one means of connectivity, try and get a second means of connectivity using another technology, such as WiFi, leased line or point-to-point laser.

Step 2: Populate WHOIS database

Although your ISP will offer to do this for you (or may even do it without telling you), it is best for you to populate the WHOIS database yourself.

If you are in Europe, you will be using RIPE. Go to www.ripe.net. Click `Whois Database' on the left hand side. Now click `Online update facility (webupdates)'.

Some fields on the webupdates forms are quite strict. If a field you want is not on the screen, you can add it using the `Add a field' button at the bottom of the form. You will often have to do this to add the password field to a form.

Create a `person' object

Under `Create a new object' select person from the menu. Click `Add object'.

Fill out the fields using the template below:

person: Your full name
address: Your address
phone: Your phone (not mobile)
nic-hdl: AUTO-1
changed: your@email.com 20050101 (put today's date, of course)
source: RIPE

If you want to put a multi-line address, click the `+' next to `address' to add more lines.

Now click `Submit update'. On the next screen, you will be given the results. If successful, you will see the line:

Create SUCCEEDED: [person] NIC Your Name

The code displayed where it says NIC is your NIC handle. You will need this often - make a note of it.

Create a maintainer object

A maintainer object requires an MD5 hashed password. Create this first, using these steps:

  1. Choose a password, for example, `my#s4f3#p4ssw0rd'
  2. Go to this page
  3. Make sure `MD5-PW' is selected
  4. Type your password into the `password' field
  5. Click `submit'
  6. You will see a result like `$1$4Aa5Sy9s$gFbjywfdjrZhPsZYWYGay/'. Cut and paste that into the `auth' field referred to in the instructions below.

Go to the `Webupdates' page. Under `Create a new object', select `mntner'. The maintainer object is used to authorise changes to other objects you create, so it is very important.

Fill out the fields like so:

mntner: XYZ-MNT Instead of XYZ, choose a unique code or acronym that relates to you or your business
descr: business name
admin-c: your NIC handle, that you created in the previous step
upd-to: your email address
auth: MD5-PW your password hash, that you created a moment ago
mnt-by: the same value as the mntner: field
referral-by: the same value as the mntner: field
changed: your@email.com 20050101 (use today's date)
source: RIPE

Now click `Submit update' to insert the maintainer object.

Link your person object to the maintainer object

The person object you created earlier can be changed by anyone. This is not a good idea. Therefore, use the `Modify an existing object' feature of Webupdates to load your person object onto the screen again.

Beside `Add New Field', select `mnt-by'. Click `Add a field'

In the `mnt-by' field, put your maintainer handle.

Beside `Add new field', select `password'. Click `Add a field'

In the `password' field, put your password, in clear text.

Now click `Submit update'. No one else will be able to change this object now, unless they know the password for the maintainer object.

Create an organisation object

Using the Webupdates tool, create an object of type `organisation'.

organisation: AUTO-1
org-name: name of your business
org-type: NON-REGISTRY
address: mail address
e-mail: your@email.com
mnt-ref: your maintainer NIC
mnt-by: your maintainer NIC
password: your password
changed: your@email.com 20050101 (use today's date)
source: RIPE

When you submit the details, a NIC handle for your organisation will be generated automatically. Make a note of this - you will need it later.

Step 3: Ask your ISP for `Provider Independent' (PI) IP addresses and an AS number

Your IP addresses and AS number are also allocated through the WHOIS system. However, they have to be applied for by a RIPE member (typically your ISP will be a RIPE member).

Provider Independent (PI) addresses can be used with multiple providers concurrently. You can even leave the ISP that first allocated them to you and keep using them. The minimum quantity you need is 256 addresses (a /24 network).

An AS number is a unique identifier for each autonymous network that participates in the Internet. You need one.

When you apply for these details, you will need to tell the ISP these things:

Most ISPs will return the IP addresses and AS number to you in 3 - 5 days. Some will charge a fee for doing so. When the ISP gives you your AS number, ask them for theirs too - you will need it shortly.

Once you have been allocated the IP addresses and AS number, view the details in the WHOIS database. Make sure the ISP has linked them to your maintainer object and not theirs - otherwise, you won't be able to maintain the details yourself.

Step 4: Create more WHOIS entries

You now need to create an AS set and a route object.

Create the AS set

The AS set is a convenient way of listing all AS numbers that you provide transit for. In many cases, this is only for yourself. However, if you provide IP transit to other businesses who you connect to (even if it's just as a failover service), you need to include their AS number in your AS set.

The NIC handle for the AS set will have a form like `AS-identity', where identity is a brief name or acronym, unique in the WHOIS database, that represents your business.

Use the Webupdates facility on RIPE to create the AS set object, like so:

as-set: The NIC handle you chose
descr: business name
members: Your AS number, prefixed with the letters AS, eg AS12345
members: include another `members' field listing the AS number or AS set NIC handle for each network you provide transit to
tech-c: your personal NIC handle
admin-c: your personal NIC handle
mnt-by: your maintainer NIC handle
changed: your@email.com 20050101
password: your maintainer password
source: RIPE

When complete, submit the AS set object details.

Reference your AS set from your AS

Add the following record to the entry in your AS (aut-num) object:

export: to (provider1's AS) announce (The handle of our AS set)
export: to (provider2's AS) announce (The handle of our AS set)

An example is below:

export: to AS12345 announce AS-MYSET
export: to AS54321 announce AS-MYSET

Inform you providers that you have created an AS set. Ask them to include import records referencing your AS set in their AS aut-num entries.

Check your providers entries at RIPE, they should look like the example below. In the example, AS12345 is the provider and AS888 is you.

aut-num: AS12345
...
...
import: from AS888 accept AS-MYSET
export: to AS888 announce ANY
...
...

Create a route object

The route object describes routes to your networks.

Normally, you will not want to create route objects for networks smaller than 8 bits (/24). Although you might internally divide your IP address range into 2 or more subnets, you normally advertise your IP addresses to other networks as a single subnet.

Use Webupdates to create the route object like so:

route: A.B.C.D/24
descr: your business name
origin: your AS, eg AS12345
mnt-by: your maintainer NIC handle
changed: your@email.com 20050101
password: your maintainer password
notify: your@email.com
source: RIPE

Many ISPs have filtering systems that protect the Internet from packets containing forged IP addresses. These filters rely on the information contained in the WHOIS database. After you update the WHOIS database, it takes up to 24 hours before all ISPs have updated their filters to include specifications for your network. In the meantime, your will find that your connectivity is rather limited.

Step 5: Sample data

For the rest of this illustration, we will assume that you are using AS23456 and the provider independent IP range 213.228.220.0/24. Of course, these numbers already belong to someone else on the Internet so you should not use them. Wherever you see these numbers in the examples, please use your own numbers.

Step 6: Prepare your switch

We will assume that that the server you will use as your core router has only 1 or 2 network interfaces. However, you want to use it to control 2 internal networks and manage connections to 4 external ISPs. To achieve this, we will connect each internal or external network to a port on a Cisco 2950 switch. We will then partition the ports into independent VLANs. Finally, we trunk all of the VLANs back to the router through a VLAN trunk port.

Beware of old switches: some older switches use proprietary trunking protocols for joining switches together. For instance, the Cisco 1924 series switches use Cisco's ISL protocol. This is not the same as the 802.1Q VLAN standard. To interoperate with FreeBSD or Linux routers, please only use switches that do 802.1Q VLAN trunking, including the Cisco 2950.

Here is a list of the VLANs and switch ports we will use for this example:

Port(s) VLAN ID Purpose IP subnet/mask Our Router IP
1 101 ISP 1 20.40.60.96/30 20.40.60.98
2 102 ISP 2 30.40.50.64/30 30.40.50.66/30
3 103 ISP 3 40.50.60.128/30 40.50.60.130
4 104 ISP 4 50.60.70.16/32 50.60.70.18
9 - 16 201 Our servers and PCs 213.228.220.0/26 213.228.220.1
22 301 Hosted servers (belong to our clients) 213.228.220.64/26 213.228.220.65
23 802.1Q Trunk port Router X X

Here is a sample configuration for a Cisco 2950. Please load this configuration through the serial port:

Current configuration : 2865 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname sw1
!
enable password my_pass
!
username admin password my_pass
ip subnet-zero
vtp mode transparent
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
!
vlan 101,102,103,104,201,301
!
interface FastEthernet0/1
 switchport access vlan 101
 switchport mode access
 no ip address
!
interface FastEthernet0/2
 switchport access vlan 102
 switchport mode access
 no ip address
!
interface FastEthernet0/3
 switchport access vlan 103
 switchport mode access
 no ip address
!
interface FastEthernet0/4
 switchport access vlan 104
 switchport mode access
 no ip address
!
interface FastEthernet0/5
 no ip address
!
interface FastEthernet0/6
 no ip address
!
interface FastEthernet0/7
 no ip address
!
interface FastEthernet0/8
 no ip address
!
interface FastEthernet0/9
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/10
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/11
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/12
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/13
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/14
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/15
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/16
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/17
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/18
 switchport access vlan 201
 switchport mode access
 no ip address
!
interface FastEthernet0/19
 no ip address
!
interface FastEthernet0/20
 no ip address
!
interface FastEthernet0/21
 no ip address
!
interface FastEthernet0/22
 switchport access vlan 301
 switchport mode access
 no ip address
!
interface FastEthernet0/23
 switchport mode trunk
 switchport nonegotiate
 no ip address
!
interface FastEthernet0/24
 no ip address
!
interface Vlan1
 no ip address
 ip broadcast-address 10.0.0.255
 no ip route-cache
 shutdown
!
interface Vlan201
 ip address 213.228.220.2 255.255.255.0
 ip broadcast-address 213.228.220.255
 no ip route-cache
!
ip default-gateway 213.228.220.1
no ip http server
!
access-list 10 permit 213.228.220.0 0.0.0.63
access-list 10 deny   any
snmp-server community public RO 10
snmp-server community private RW 10
snmp-server location "Telehouse, London"
!
line con 0
line vty 0 4
 password my_admin
 login
line vty 5
 password my_admin
 login
line vty 6 15
 password my_admin
 login
!
end

Step 7: Select router hardware

Select a machine meeting the following requirements:

Step 8: Install FreeBSD

Download FreeBSD 5.4 CDROM ISO image. Burn the image to a CD and proceed with the installation in a normal way. When prompted for packages, choose `binaries, kernel source only, and no X windows'.

Step 9: Install packages onto FreeBSD

We have chosen to install the following software packages onto FreeBSD:

Download those packages from the FreeBSD ports collection. Install them with the command `pkg_add'.

cd /usr/ports/net-mgmt/net-snmp

make build && make install

Step 9.2: Customise the kernel

Kernel options needed:

otions DEVICE_POLLING

options HZ=1000

For /etc/sysctl.conf:

kern.polling.enable=1

sysinstall

Select > Configure > Distributions > Src > Select Base, Etc, Sys and all other required source code

cd /usr/src/sys/i386/conf

cp GENERIC GENERICPOLL

vi GENERICPOLL

cd /usr/src

make buildkernel KERNCONF=GENERICPOLL

make installkernel KERNCONF=GENERICPOLL

Step 10: Customise FreeBSD

This is the content of /etc/rc.conf:

# Enable network daemons for user convenience.
# Use ISP 1 as our default
defaultrouter="20.40.60.97"
gateway_enable="YES"
hostname="r1.mydomain.com"
# we don't set an IP address on fxp0, we use the VLANs instead
ifconfig_fxp0="up media 100baseTX mediaopt full-duplex"
# create VLANs
cloned_interfaces="vlan101 vlan102 vlan103 vlan104 vlan201 vlan301"
ifconfig_vlan101="inet 20.40.60.98 netmask 255.255.255.252 vlan 101 vlandev fxp0"
ifconfig_vlan102="inet 30.40.50.66 netmask 255.255.255.252 vlan 102 vlandev fxp0"
ifconfig_vlan102="inet 40.50.60.130 netmask 255.255.255.252 vlan 103 vlandev fxp0"
ifconfig_vlan102="inet 50.60.70.18 netmask 255.255.255.252 vlan 104 vlandev fxp0"
ifconfig_vlan201="inet 213.228.220.1 netmask 255.255.255.192 vlan 201 vlandev fxp0"
ifconfig_vlan301="inet 213.228.220.65 netmask 255.255.255.192 vlan 301 vlandev fxp0"
linux_enable="YES"
sshd_enable="YES"

quagga_enable="YES"
quagga_flags="-d"
quagga_daemons="zebra bgpd"

snmpd_enable="YES"
snmpd_flags="-a -p /var/run/snmpd.pid"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

This is the content of /usr/local/etc/snmp/snmpd.conf:

ro community  public 213.228.220.0/26
load  12.0 12.0 12.0
syslocation "Telehouse, London"
syscontact "+44 207 1234 5678"

This is the content of /usr/local/etc/quagga/zebra.conf:

!
! Zebra configuration file
!
hostname r1
password my_pass
enable password my_pass
!
log stdout
!
!

This is the content of /usr/local/etc/bgpd.conf:

!
!
hostname r1
password my_pass
enable password my_pass
!
router bgp 23456
  bgp router-id 213.228.220.1
  network 213.228.220.0/24
  aggregate-address 213.228.220.0/24
  neighbor 20.40.60.97 remote-as 120001
  neighbor 20.40.60.97 next-hop-self
  neighbor 20.40.60.97 weight 1
  neighbor 20.40.60.97 prefix-list a_self out
  neighbor 30.40.50.65 remote-as 120002
  neighbor 30.40.50.65 next-hop-self
  neighbor 30.40.50.65 weight 1
  neighbor 30.40.50.65 prefix-list a_self out
  neighbor 40.50.60.129 remote-as 120001
  neighbor 40.50.60.129 next-hop-self
  neighbor 40.50.60.129 weight 1
  neighbor 40.50.60.129 prefix-list a_self out
  neighbor 50.60.70.17 remote-as 120001
  neighbor 50.60.70.17 next-hop-self
  neighbor 50.60.70.17 weight 1
  neighbor 50.60.70.17 prefix-list a_self out
  ip prefix-list a_self permit 213.228.220.0/24
  ip prefix-list a_self deny any
!
log stdout

We modify this line in /etc/ttys so that we can connect over the serial port, just like a Cisco router:

ttyd0   "/usr/libexec/getty 3wire.9600" vt100 on secure

This is /boot/loader.conf, it causes boot messages to go over the serial port too:


console="comconsole"

Reboot

After configuring all the files above, the easiest way to proceed is to reboot the machine. Upon booting, all the interfaces will be configured and quagga will attempt to get BGP routing tables from your ISPs.

Step 11: Testing

Traceroute

Perform traceroutes to and from your router.

Looking glasses

Use looking glass web sites to see how your routing information has been propogated.

Step 12: Monitoring

MRTG

Use MRTG, running on another host, to chart the network traffic through each port on the switch and the router.

References