Overview

Patroni and ETCD provide a great combination for a cluster solution for a PostgreSQL. In a nutshell: Patroni managed the cluster while the ETCD allows configuration data to be stored. The diagram of how they interact with PostgreSQL you can see below:

So let's get started with the configuration of ETCD first. To configure ETCD, we need to:

Install the package

Configure the cluster

Both tasks are fairly simple:

Setup ETCD

Let's assume we have three nodes (as the picture above)

node00 (10.100.10.200)

node01 (10.100.10.201)

node02 (10.100.10.202)

I like starting from “0” I know To install the package, simply use the package provider: yum, apt-get, etc:

Install RPM

Install ETCD package

You need to do that all nodes

Configure the cluster

To configure the cluster, use the following script:

Configure ETCD

Again, you need to run that on all 3 nodes, after that we can start the ETCD

Start the ETCD

Start ETCD

Again, on all nodes. We can after that check the status of the cluster:

Check the status

Check status

If you uses API v3, you have to use the new command:

Check status

You can also “TEST” the server as publishing messages (key-values)

Test ETCD

So we have our key-value propagated among the cluster, nice….

Setup Patroni

The Patroni is really the brain behind the cluster management and the coordination with the distributed key store (in our case ETCD, although other solutions are possible as well: Consul, K8S, etc) Patroni is a fork of Governor and its job is to manage the cluster. With patroni you can also reinitialize a node, as we will see below. To sum it up, the configuration of Patroni, you can use the following script:

Setup Patroni

Execute that command on all servers in the cluster. You don't have to substitute anything. Everything is self generated (I like twerking with bash). On the first server, you just need to create the replication user:

Create replication User

After that, you should have all 3 nodes working together:

Check up

Now, we can see that the node: “node00” is the leader and the rest are Replicas.

Test failover

Let's test a failover. We will halt the first node:

Halt a node

Don't forget, that Patroni is checking if there is a Leader node every 5 seconds or so. So it should see that we are Leaderless and promote a replica:

Auto-Promoting Replica

We see that we have stopped server, so let's see again:

Auto-Promoting Replica P2

Yeyyy, after some time, we have a new Leader. Off we go.

Management

Of course, with every software, we have to manage it. So let's what issues I had. Well, I had many issues, but one of the most important ones was that Node02 was ahead of the Node00 for some reason, so I had to reinitialize Node02:

Reinitialize a node

Re-initializing is quite simple. It will pretty much do a pg_basebackup and restore it on the node:

Reinitialize a node

That was necessary because, even though, node02 was “running” it wasn't in sync with the leader. After the re-initialization, all is fixed.

Appendix

The whole code to run you that cluster using vagrant, you can see below.

That code will create 3 node cluster with everything.