I wanted to try project Calico for some time already. In fact, it was part of my plan to have a redundant docker infrastructure network wise. As my test lab has already enougth encapsulation layers (ipsec + gre) I didn’t want to add another one with flannel. The whole objective of setting up bgp between all 3 routers/firewalls/hosts was to announce docker services throught BGP.
Project Calico
Calico is a solution for interconnecting docker containers to the world by routing direct IP and not managing any NAT tables.
In order to advertize nodes Containers’ IP to the DC, Calico uses BGP to exchange routes. It supports full mesh iBGP as well as route reflectors.
I first planned full mesh inside one AS, as 1 AS represents 1 physical node with several VMs hosting containers. My physical node not being that powerfull, it doesn’t represent a lot of containers.
Prerequisite
Calico requests a etcd server (or better, cluster) to synchronize all nodes. I already had one cluster ready with my CoreOS tests, but I deleted 1 node and the cluster entered in a bad state. I don’t want to restore the server at home, neither I wan’t to rebuild the cluster. So the solution was to erase the data and start a fresh 1 node cluster from CoreOS.
Changing the coreos1 cloud-init file to start etcd in stand alone mode:
Erase /var/lib/etcd/member
Edit user_dat in /var/lib/libvirt/images/coreos/coreos1/openstack/latest and change the etcd2 options:
Installing Calico
Export ETCD endpoints
Download Calico: Calico is distributed as a statically linked go binary, store it in /usr/local/bin
modify docker to store infos into etcd by changing cluster-store key in /etc/docker/daemon.json:
Restart docker daemon
Pull calico/node out of docker:
Start calico node
Using Calico
To use Calico, we need to define one network to it and to create a network on our docker host using calico driver. Here are the steps to do so:
Create network in Calico. The whole subnet is not used as is by calico, instead, calico is attribuing a /26 per node hosting containers.
Create docker network
Create BGP peer on the node by editing /etc/bird/bird.conf
Restart bird daemon
Check route propagation
Route is diffused as wished :)
Autostart Calico
I decided to follow the guides on calico web site and to create an env file loaded by the system unit that starts calico node.
Create env file in /etc/calico/calico.env
Create Calico systemd unit
Reload systemd units
Related modifications
As I created a new network, I have to open the flows on iptables. Therefore, the following rules were added:
Conclusion
As I learnt the hard way, it’s not possible to annouce /32 route per container.
Instead of that, Calico summarizes all containers on a node inside a /26. It means that I have to have 1 AS for all physical nodes that hosts containers in 1 “cluster”. So my idea is not possible, I have to use only 1 AS for all my physical nodes.
The next step will be to change my BGP topology from 1 AS per server to 1 AS for home and 1 AS for both hosted servers. Then, I’ll have to change Calico topology from full mesh iBGP to 1 route reflector per host.