Rancher High-Availability
This blog post is the last part of a series of three tutorials dedicated to Rancher High-Availability.
We previously installed Rancher single-node using a external databases:
This time we will go full HA by installing Rancher on three servers.
Pre-requisites¶
You need an existing Rancher server (single-node) that uses an external database. You also need from 3 to 5 servers with at least 1GB RAM (8GB recommended) and a load balancer:
- Rancher Server One available at IP address 1.1.1.1,
- Rancher Server Two available at IP address 2.2.2.2,
- Rancher Server Three available at IP address 3.3.3.3,
- Load Balancer available at IP address 4.4.4.4.
All servers must have Docker installed.
Firewall configuration¶
You need a global access for each Rancher server on TCP Ports 22, 80, 443 and 18080.
You need access between Rancher servers :
- to UDP Ports 500, 4500
- to TCP Ports: 2181, 2376, 2888, 3888, 6379
Heap Size configuration¶
For larger deployments you may also need to increase the Rancher Heap Size.
That can be done by passing the environment variable -e JAVA_OPTS="-Xms2048m -Xmx2048m" when running the rancher/server container.
Installing the Rancher Load Balancer¶
We need a load balancer in front of our 3 Rancher servers. The HA documentation gives us configuration samples for Nginx, Apache and HAProxy. Let’s give a try to HAProxy.
Copy the configuration file from Rancher documentation and edit it using you 3 nodes IP addresses:
global
maxconn 4096
ssl-server-verify none
defaults
mode tcp
balance roundrobin
option redispatch
option forwardfor
timeout connect 5s
timeout queue 5s
timeout client 36000s
timeout server 36000s
frontend http-in
mode tcp
bind *:443 ssl crt /etc/haproxy/certificate.pem
default_backend rancher_servers
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend rancher_servers if is_websocket
backend rancher_servers
server websrv1 1.1.1.1:443 weight 1 maxconn 1024 ssl
server websrv2 2.2.2.2:443 weight 1 maxconn 1024 ssl
server websrv3 3.3.3.3:443 weight 1 maxconn 1024 ssl
Connect to the load balancer server using SSH and paste the HAProxy configuration in a file named haproxy.cfg.
You may also need a SSL certificate. Generate it using the command:
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /root/rancher.key -out /root/rancher.crt
And create the .pem file using cat rancher.crt rancher.key > rancher.pem
Then run a Docker container for HAProxy:
docker run -d --name haproxy -v /root/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -v /root/rancher.pem:/etc/haproxy/certificate.pem:ro haproxy:1.5
You can display the logs using the command docker logs -f haproxy.
Exporting the Rancher High-Availability script¶
Go to the Rancher High-Availability page:
- Copy/paste the IP address of your load balancer into the Host Registration URL field.
- Copy/paste the content of the rancher.key file into the Private Key field and the content of rancher.crt into Certificate.
- Click on Generate Config Script and on Download Config Script.
- Stop the Rancher container on Rancher Server One (
docker stop <containerID>). - Copy the downloaded script on each node, and execute it:
sudo bash ./rancher-ha.sh rancher/server:v1.1.4
You can display the logs on each node using docker logs -f rancher-ha.
Also you can access each Rancher UI at http://1.1.1.1:1800/ and view the Audit Logs (Admin > Audit Logs) and the HA Environment stack.
Wait for all the servers to start (this may take a few minutes). Finally the Rancher cluster UI should be available at the IP address of your load balancer https//4.4.4.4/ and should display the High-Availability status:
Conclusion¶
We managed to have multiple Rancher severs working together but the load balancer is still a Single Point of Failure. If it goes does the whole infrastructure would be unavailable.