I wanted to try project Calico for some time already. In fact, it was part of my plan to have a redundant docker infrastructure network wise. As my test lab has already enougth encapsulation layers (ipsec + gre) I didn’t want to add another one with flannel. The whole objective of setting up bgp between all 3 routers/firewalls/hosts was to announce docker services throught BGP.

Project Calico

Calico is a solution for interconnecting docker containers to the world by routing direct IP and not managing any NAT tables. In order to advertize nodes Containers’ IP to the DC, Calico uses BGP to exchange routes. It supports full mesh iBGP as well as route reflectors. I first planned full mesh inside one AS, as 1 AS represents 1 physical node with several VMs hosting containers. My physical node not being that powerfull, it doesn’t represent a lot of containers.

Prerequisite

Calico requests a etcd server (or better, cluster) to synchronize all nodes. I already had one cluster ready with my CoreOS tests, but I deleted 1 node and the cluster entered in a bad state. I don’t want to restore the server at home, neither I wan’t to rebuild the cluster. So the solution was to erase the data and start a fresh 1 node cluster from CoreOS. Changing the coreos1 cloud-init file to start etcd in stand alone mode:

  • Erase /var/lib/etcd/member
  • Edit user_dat in /var/lib/libvirt/images/coreos/coreos1/openstack/latest and change the etcd2 options:
   etcd2:
    name: coreos1
    advertise-client-urls: http://172.16.4.10:2379,http://172.16.4.10:4001
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://172.16.4.10:2380
    initial-cluster-token: etcd-cluster-1
    initial-cluster: coreos1=http://172.16.4.10:2380
    initial-advertise-peer-urls: http://172.16.4.10:2380
    initial-cluster-state: new

Installing Calico

  • Export ETCD endpoints
root@ubuntu-docker:~# echo 'export ETCD_ENDPOINTS=http://coreos1:2379' >> ~/.bashrc
  • Download Calico: Calico is distributed as a statically linked go binary, store it in /usr/local/bin
root@ubuntu-docker:~# wget -0 /usr/local/bin/calicoctl https://github.com/projectcalico/calico-containers/releases/download/v1.0.0/calicoctl
root@ubuntu-docker:~# chmod +x /usr/local/bin/calicoctl
  • modify docker to store infos into etcd by changing cluster-store key in /etc/docker/daemon.json:
{
    "cluster-store": "etcd://coreos1.claer.local:2379"
}
  • Restart docker daemon
root@ubuntu-docker:~# service docker restart
  • Pull calico/node out of docker:
root@ubuntu-docker:~# docker pull calico/node:v1.0.0
  • Start calico node
root@ubuntu-docker:~# calicoctl node run

Using Calico

To use Calico, we need to define one network to it and to create a network on our docker host using calico driver. Here are the steps to do so:

  • Create network in Calico. The whole subnet is not used as is by calico, instead, calico is attribuing a /26 per node hosting containers.
root@ubuntu-docker:~# cat << EOF | calicoctl create -f -
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 172.16.5.0/24
  spec:
    ipip:
      enabled: false
    nat-outgoing: false
EOF
  • Create docker network
root@ubuntu-docker:~# docker network create --driver calico --ipam-driver calico-ipam --subnet=172.16.5.0/24 cal_net
root@ubuntu-docker:~# docker network list
NETWORK ID          NAME                DRIVER              SCOPE
89bf55a26e80        bridge              bridge              local
93110c63134f        cal_net             calico              global
9612646b4636        host                host                local
34d0173db2be        none                null                local
  • Create BGP peer on the node by editing /etc/bird/bird.conf
protocol bgp docker {
        local as myas;
        description "to docker";
        neighbor 172.16.4.105 as myas;
        # suppression du next hop nécessaire pour que ca marche
        next hop self;
        import filter bgp_in;
        import limit 10;
        export filter {
                if net_infra() then accept;
                if net_martian() then reject;
                else accept;
        };
}
  • Restart bird daemon
[root@db-xc1 405 ~]# service bird restart
  • Check route propagation
[root@db-xc1 406 ~]# birdc
BIRD 1.6.2 ready.
bird> sh rou
0.0.0.0/0          unreachable [default_originate 22:08:04] ! (200)
                   via 163.172.35.1 on eth0 [kernel1 22:08:04] (10)
172.16.4.0/24      dev br10 [direct1 22:08:04] * (240)
172.16.5.128/26    via 172.16.4.105 on br10 [docker 22:08:08] * (100/0) [i]
172.16.1.0/24      via 172.16.255.5 on gre1 [cobra 22:08:08] * (100) [AS65001i]
172.16.2.0/24      via 172.16.255.5 on gre1 [cobra 22:08:08] * (100) [AS65001i]
root@cobra ~ # bgpctl s rib
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination          gateway          lpref   med aspath origin
AI*>  172.16.1.0/24        0.0.0.0            100     0 i
AI*>  172.16.2.0/24        0.0.0.0            100     0 i
*>    172.16.3.0/24        172.16.255.2       100     0 65003 i
*>    172.16.4.0/24        172.16.255.6       100     0 65004 i
*>    172.16.5.128/26      172.16.255.6       100     0 65004 i

Route is diffused as wished :)

Autostart Calico

I decided to follow the guides on calico web site and to create an env file loaded by the system unit that starts calico node.

  • Create env file in /etc/calico/calico.env
ETCD_ENDPOINTS=http://coreos1.claer.local:2379
ETCD_CA_FILE=""
ETCD_CERT_FILE=""
ETCD_KEY_FILE=""
CALICO_HOSTNAME="ubuntu-docker"
CALICO_NO_DEFAULT_POOLS=""
CALICO_IP=""
CALICO_IP6=""
CALICO_AS=""
CALICO_LIBNETWORK_ENABLED=true
CALICO_NETWORKING_BACKEND=bird
  • Create Calico systemd unit
[Unit]
Description=calico-node
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/etc/calico/calico.env
ExecStartPre=-/usr/bin/docker rm -f calico-node
ExecStart=/usr/bin/docker run --net=host --privileged \
 --name=calico-node \
 -e HOSTNAME=${HOSTNAME} \
 -e IP=${CALICO_IP} \
 -e IP6=${CALICO_IP6} \
 -e CALICO_NETWORKING_BACKEND=${CALICO_NETWORKING_BACKEND} \
 -e AS=${CALICO_AS} \
 -e NO_DEFAULT_POOLS=${CALICO_NO_DEFAULT_POOLS} \
 -e CALICO_LIBNETWORK_ENABLED=${CALICO_LIBNETWORK_ENABLED} \
 -e ETCD_ENDPOINTS=${ETCD_ENDPOINTS} \
 -e ETCD_CA_CERT_FILE=${ETCD_CA_CERT_FILE} \
 -e ETCD_CERT_FILE=${ETCD_CERT_FILE} \
 -e ETCD_KEY_FILE=${ETCD_KEY_FILE} \
 -v /var/log/calico:/var/log/calico \
 -v /run/docker/plugins:/run/docker/plugins \
 -v /lib/modules:/lib/modules \
 -v /var/run/calico:/var/run/calico \
 calico/node:v1.0.0-beta

ExecStop=-/usr/bin/docker stop calico-node

[Install]
WantedBy=multi-user.target
  • Reload systemd units
root@ubuntu-docker:/etc/systemd/system# systemctl daemon-reload
root@ubuntu-docker:/etc/systemd/system# systemctl enable calico-node.service

As I created a new network, I have to open the flows on iptables. Therefore, the following rules were added:

-A FORWARD -d 172.16.5.0/24 -o br10 -j ACCEPT
-A FORWARD -s 172.16.5.0/24 -i br10 -j ACCEPT

Conclusion

As I learnt the hard way, it’s not possible to annouce /32 route per container. Instead of that, Calico summarizes all containers on a node inside a /26. It means that I have to have 1 AS for all physical nodes that hosts containers in 1 “cluster”. So my idea is not possible, I have to use only 1 AS for all my physical nodes. The next step will be to change my BGP topology from 1 AS per server to 1 AS for home and 1 AS for both hosted servers. Then, I’ll have to change Calico topology from full mesh iBGP to 1 route reflector per host.