The first idea I have when building the 3 nodes solution was to to peer each node with eBGP sessions then, each node would be able to run calico to advertize /32 routes. That way, the Docker workload could move from one site to another with automagical reconfiguration.

In this post, I’ll build 3 peerings:

  • cobra (OpenBSD) to db-sc1 (Debian, BIRD)
  • cobra (OpenBSD) to db-xc1 (Debian, BIRD)
  • db-sc1 (Debian, BIRD) to db-xc1 (Debian, BIRD)

The AS are the following:

  • cobra: AS 65001
  • db-sc1: AS 65003
  • db-xc1: AS 65004

All nodes are communicating well within the 3 GRE tunnels I’ve setup

OpenBSD bgpd

  • bgpd is already installed, therefore only the configuration needs to be done. Here is my copy of bgpd.conf (Yeah I know, one day I should add md5 auth =))
# see bgpd.conf(5)

#macros
peer1="172.16.255.2"
peer2="172.16.255.6"

# global configuration
AS 65001
router-id 172.16.1.1
# holdtime 180
# holdtime min 3
# listen on 127.0.0.1
# listen on ::1
# fib-update no
# route-collector no
# log updates

network 172.16.1.0/24
network 172.16.2.0/24

# restricted socket for bgplg(8)
# socket "/var/www/logs/bgpd.rsock" restricted

neighbor $peer1 {
        remote-as       65003
        descr           db-sc1
        local-address   172.16.255.1
}

neighbor $peer2 {
        remote-as       65004
        descr           db-xc1
        local-address   172.16.255.5
}
# filter out prefixes longer than 24 or shorter than 8 bits for IPv4
# and longer than 48 or shorter than 16 bits for IPv6.
deny from any
allow from any inet prefix 172.16.0.0/12 prefixlen >= 24
allow from any inet prefix 192.168.0.0/16 prefixlen >= 24

# accept a default route (since the previous rule blocks this)
#allow from any prefix 0.0.0.0/0

# filter bogus networks according to RFC5735
deny from any prefix 0.0.0.0/8 prefixlen >= 8
deny from any prefix 10.0.0.0/8 prefixlen >= 8
deny from any prefix 169.254.0.0/16 prefixlen >= 16
deny from any prefix 192.0.2.0/24 prefixlen >= 24
deny from any prefix 198.18.0.0/15 prefixlen >= 15
deny from any prefix 198.51.100.0/24 prefixlen >= 24
deny from any prefix 203.0.113.0/24 prefixlen >= 24
deny from any prefix 224.0.0.0/4 prefixlen >= 4
deny from any prefix 240.0.0.0/4 prefixlen >= 4

# filter bogus IPv6 networks according to IANA
deny from any prefix ::/8 prefixlen >= 8
deny from any prefix 2001:2::/48 prefixlen >= 48        # BMWG [RFC5180]
deny from any prefix 2001:10::/28 prefixlen >= 28       # ORCHID [RFC4843]
deny from any prefix 2001:db8::/32 prefixlen >= 32      # docu range [RFC3849]
deny from any prefix 3ffe::/16 prefixlen >= 16          # old 6bone
deny from any prefix fc00::/7 prefixlen >= 7            # unique local unicast
deny from any prefix fe80::/10 prefixlen >= 10          # link local unicast
deny from any prefix fec0::/10 prefixlen >= 10          # old site local unicast
deny from any prefix ff00::/8 prefixlen >= 8            # multicast
  • Now, enable bgpd at boot time
root@cobra # rcctl enable bgpd

Bird on Debian hosts

  • Install bird on Debian. I choose to go with the repository offered by the bird mainteners, so I created a file /etc/apt/sources.list.d/bird_network_cz_debian.list
deb http://bird.network.cz/debian/ jessie main
  • After adding the repo, add the gpg key, update your package lists and install bird
wget -O - http://bird.network.cz/debian/apt.key | apt-key add -
apt-get update
apt-get install bird
  • Update the configuration to match the desired state. This is the configuration for db-sc1. the other bird node has similar configuration (update router id and AS number)
router id 172.16.3.1;
define myas = 65003;

protocol kernel {
        persist;
        scan time 20;
        learn;
        import all;
        export all;   # Actually insert routes into the kernel routing table
}
protocol device {
        scan time 60;
}
protocol direct {
        interface "virbr1"; # it should match the bridge name of the host
}
function net_infra() {
  return net ~ [ 172.16.0.0/12+, 192.168.0.0/16+ ];
}

function net_martian() {
  return net ~ [ 169.254.0.0/16+, 10.0.0.0/8+, 127.0.0.0/8+, 224.0.0.0/4+,
        240.0.0.0/4+, 0.0.0.0/32-, 0.0.0.0/0{25,32}, 0.0.0.0/0{0,7} ];
}
filter bgp_out {
        if net_martian() then reject;
        else accept;
}

filter bgp_in {
        if net_infra() then accept;
        else reject;
}

# Small peer
protocol bgp db_xc1 {
        local as myas;
        description "to db-xc1";
        neighbor 172.16.255.10 as 65004;
        import filter bgp_in;
        import limit 10;
        export filter bgp_out;
}

protocol bgp cobra {
        local as myas;
        description "to home";
        neighbor 172.16.255.1 as 65001;
        import filter bgp_in;
        import limit 10;
        export filter bgp_out;
}
  • Restart service
service bird restart

post install checks

After checking that all routes are OK, I removed the static routes entries from my gre* definition files on Debian and OpenBSD.