Overview
Barman is a backup & Recovery configuration which uses pg_basebackup and pg_receivexlog underneath in order to provide backup and restore functionalities. Furthermore, barman is able to operate on many servers, thus providing central location for backups for the entire infrastructure. At the same time, barman doesn't have a client application, the server installation is the only installation which has to be done. Even though there isn't client configuration, barman still has to be configured on both:
- Target Machine
- Barman Machine
Let's start with the Barman machine.
Barman Machine Configuration
The machine isn't mandatory to be with high specs, but at the same time, shouldn't be also a leftover :). The basic configuration of the barman is already provided through RPM as follows:
Library installation
Let's install firstly the libraries and thus the app:
[root@lpara orcldb]# yum install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm -y Loaded plugins: fastestmirror Setting up Install Process pgdg-redhat96-9.6-3.noarch.rpm | 4.7 kB 00:00 Examining /var/tmp/yum-root-cM4ST4/pgdg-redhat96-9.6-3.noarch.rpm: pgdg-redhat96-9.6-3.noarch Marking /var/tmp/yum-root-cM4ST4/pgdg-redhat96-9.6-3.noarch.rpm to be installed Loading mirror speeds from cached hostfile * base: ftp.hosteurope.de * extras: ftp.hosteurope.de * updates: ftp.hosteurope.de Resolving Dependencies --> Running transaction check ---> Package pgdg-redhat96.noarch 0:9.6-3 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================= Installing: pgdg-redhat96 noarch 9.6-3 /pgdg-redhat96-9.6-3.noarch 2.7 k Transaction Summary ============================================================================================================================================= Install 1 Package(s) Total size: 2.7 k Installed size: 2.7 k Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : pgdg-redhat96-9.6-3.noarch 1/1 Verifying : pgdg-redhat96-9.6-3.noarch 1/1 Installed: pgdg-redhat96.noarch 0:9.6-3 Complete! [root@lpara orcldb]# ls -lart total 128 drwxr-x---. 3 oracle oinstall 4096 Mar 16 13:46 .. -rw-r-----. 1 oracle oinstall 1344 Mar 16 13:47 **rmanRestoreDatafiles.sql** <- Installed the repository -rw-r-----. 1 oracle oinstall 330 Mar 16 13:52 CloneRmanRestore.log -rw-r-----. 1 oracle oinstall 1291 Mar 16 13:57 cloneDBCreation.log -rw-r-----. 1 oracle oinstall 8 Mar 16 13:58 postScripts.log -rw-r-----. 1 oracle oinstall 6 Mar 16 13:58 lockAccount.log drwxr-x---. 2 oracle oinstall 4096 Mar 16 13:58 . -rw-r-----. 1 oracle oinstall 921 Mar 16 14:02 postDBCreation.log -rw-r-----. 1 oracle oinstall 95838 Mar 16 14:02 trace.log [root@lpara orcldb]# cd /etc/yum.repos.d/
After we have downloaded the main library, we can simply install barman using yum with little interference :).
[root@lpara yum.repos.d]# yum install barman
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: ftp.hosteurope.de
* extras: ftp.hosteurope.de
* updates: ftp.hosteurope.de
pgdg96 | 4.1 kB 00:00
pgdg96/primary_db | 182 kB 00:00
Resolving Dependencies
--> Running transaction check
---> Package barman.noarch 0:2.3-1.rhel6 will be installed
--> Processing Dependency: rsync >= 3.0.4 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-psycopg2 >= 2.4.2 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-argh >= 0.21.2 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-dateutil for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-argcomplete for package: barman-2.3-1.rhel6.noarch
--> Running transaction check
---> Package barman.noarch 0:2.3-1.rhel6 will be installed
--> Processing Dependency: python-argh >= 0.21.2 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-argcomplete for package: barman-2.3-1.rhel6.noarch
---> Package python-dateutil.noarch 0:1.4.1-7.el6 will be installed
--> Processing Dependency: python-six for package: python-dateutil-1.4.1-7.el6.noarch
---> Package python-psycopg2.x86_64 0:2.7.4-1.rhel6 will be installed
--> Processing Dependency: postgresql96-libs for package: python-psycopg2-2.7.4-1.rhel6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: python-psycopg2-2.7.4-1.rhel6.x86_64
---> Package rsync.x86_64 0:3.0.6-12.el6 will be installed
--> Running transaction check
---> Package barman.noarch 0:2.3-1.rhel6 will be installed
--> Processing Dependency: python-argh >= 0.21.2 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-argcomplete for package: barman-2.3-1.rhel6.noarch
---> Package postgresql96-libs.x86_64 0:9.6.8-1PGDG.rhel6 will be installed
---> Package python-six.noarch 0:1.9.0-2.el6 will be installed
--> Finished Dependency Resolution
Error: Package: barman-2.3-1.rhel6.noarch (pgdg96)
Requires: python-argcomplete
Error: Package: barman-2.3-1.rhel6.noarch (pgdg96)
Requires: python-argh >= 0.21.2
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
[root@lpara yum.repos.d]# yum install python-argcomplete
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: ftp.hosteurope.de
* extras: ftp.hosteurope.de
* updates: ftp.hosteurope.de
No package python-argcomplete available.
Error: Nothing to do
[root@lpara yum.repos.d]# wget http://rpmfind.net/linux/epel/6/x86_64/Packages/p/python-argcomplete-0.8.8-1.el6.noarch.rpm
--2018-03-16 15:18:21-- http://rpmfind.net/linux/epel/6/x86_64/Packages/p/python-argcomplete-0.8.8-1.el6.noarch.rpm
Resolving rpmfind.net... 195.220.108.108
Connecting to rpmfind.net|195.220.108.108|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47512 (46K) [application/x-rpm]
Saving to: “python-argcomplete-0.8.8-1.el6.noarch.rpm”
100%[==============================================================================================================================>] 47,512 293K/s in 0.2s
2018-03-16 15:18:22 (293 KB/s) - “python-argcomplete-0.8.8-1.el6.noarch.rpm” saved [47512/47512]
[root@lpara yum.repos.d]# rpm -Uvh python-argcomplete-0.8.8-1.el6.noarch.rpm
warning: python-argcomplete-0.8.8-1.el6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing... ########################################### [100%]
1:python-argcomplete ########################################### [100%]
[root@lpara yum.repos.d]# wget http://rpmfind.net/linux/epel/6/x86_64/Packages/p/python-argh-0.23.2-1.el6.noarch.rpm
--2018-03-16 15:18:45-- http://rpmfind.net/linux/epel/6/x86_64/Packages/p/python-argh-0.23.2-1.el6.noarch.rpm
Resolving rpmfind.net... 195.220.108.108
Connecting to rpmfind.net|195.220.108.108|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37816 (37K) [application/x-rpm]
Saving to: “python-argh-0.23.2-1.el6.noarch.rpm”
100%[==============================================================================================================================>] 37,816 --.-K/s in 0.08s
2018-03-16 15:18:45 (475 KB/s) - “python-argh-0.23.2-1.el6.noarch.rpm” saved [37816/37816]
[root@lpara yum.repos.d]# rpm -Uvh python-argh-0.23.2-1.el6.noarch.rpm
warning: python-argh-0.23.2-1.el6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing... ########################################### [100%]
1:python-argh ########################################### [100%]
[root@lpara yum.repos.d]# yum install barman
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: ftp.hosteurope.de
* extras: ftp.hosteurope.de
* updates: ftp.hosteurope.de
Resolving Dependencies
--> Running transaction check
---> Package barman.noarch 0:2.3-1.rhel6 will be installed
--> Processing Dependency: rsync >= 3.0.4 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-psycopg2 >= 2.4.2 for package: barman-2.3-1.rhel6.noarch
--> Processing Dependency: python-dateutil for package: barman-2.3-1.rhel6.noarch
--> Running transaction check
---> Package python-dateutil.noarch 0:1.4.1-7.el6 will be installed
--> Processing Dependency: python-six for package: python-dateutil-1.4.1-7.el6.noarch
---> Package python-psycopg2.x86_64 0:2.7.4-1.rhel6 will be installed
--> Processing Dependency: postgresql96-libs for package: python-psycopg2-2.7.4-1.rhel6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: python-psycopg2-2.7.4-1.rhel6.x86_64
---> Package rsync.x86_64 0:3.0.6-12.el6 will be installed
--> Running transaction check
---> Package postgresql96-libs.x86_64 0:9.6.8-1PGDG.rhel6 will be installed
---> Package python-six.noarch 0:1.9.0-2.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================
Installing:
barman noarch 2.3-1.rhel6 pgdg96 284 k
Installing for dependencies:
postgresql96-libs x86_64 9.6.8-1PGDG.rhel6 pgdg96 286 k
python-dateutil noarch 1.4.1-7.el6 base 84 k
python-psycopg2 x86_64 2.7.4-1.rhel6 pgdg96 151 k
python-six noarch 1.9.0-2.el6 base 28 k
rsync x86_64 3.0.6-12.el6 base 335 k
Transaction Summary
========================================================================================================================================================================
Install 6 Package(s)
Total download size: 1.1 M
Installed size: 4.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): barman-2.3-1.rhel6.noarch.rpm | 284 kB 00:00
(2/6): postgresql96-libs-9.6.8-1PGDG.rhel6.x86_64.rpm | 286 kB 00:00
(3/6): python-dateutil-1.4.1-7.el6.noarch.rpm | 84 kB 00:00
(4/6): python-psycopg2-2.7.4-1.rhel6.x86_64.rpm | 151 kB 00:00
(5/6): python-six-1.9.0-2.el6.noarch.rpm | 28 kB 00:00
(6/6): rsync-3.0.6-12.el6.x86_64.rpm | 335 kB 00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 270 kB/s | 1.1 MB 00:04
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Installing : postgresql96-libs-9.6.8-1PGDG.rhel6.x86_64 1/6
Installing : python-psycopg2-2.7.4-1.rhel6.x86_64 2/6
Installing : rsync-3.0.6-12.el6.x86_64 3/6
Installing : python-six-1.9.0-2.el6.noarch 4/6
Installing : python-dateutil-1.4.1-7.el6.noarch 5/6
Installing : barman-2.3-1.rhel6.noarch 6/6
Verifying : python-six-1.9.0-2.el6.noarch 1/6
Verifying : python-psycopg2-2.7.4-1.rhel6.x86_64 2/6
Verifying : rsync-3.0.6-12.el6.x86_64 3/6
Verifying : python-dateutil-1.4.1-7.el6.noarch 4/6
Verifying : barman-2.3-1.rhel6.noarch 5/6
Verifying : postgresql96-libs-9.6.8-1PGDG.rhel6.x86_64 6/6
Installed:
barman.noarch 0:2.3-1.rhel6
Dependency Installed:
postgresql96-libs.x86_64 0:9.6.8-1PGDG.rhel6 python-dateutil.noarch 0:1.4.1-7.el6 python-psycopg2.x86_64 0:2.7.4-1.rhel6 python-six.noarch 0:1.9.0-2.el6
rsync.x86_64 0:3.0.6-12.el6
Complete!
[root@lpara yum.repos.d]#
Server Configuration
The configuration is practically within the /etc/barman.conf, however it took my VERY LONG time to figure out the exact one :).
[barman] <- Specifies the barman general configuration barman_home = /var/lib/barman barman_user = barman log_file = /var/log/barman/barman.log compression = gzip bandwidth_limit = 0 immediate_checkpoint = true last_backup_maximum_age = 3 DAYS retention_policy = RECOVERY WINDOW OF 7 days minimum_redundancy = 3 [pg] <- Specify a PostgreSQL server, in this case our server will be called "PG" description = “PG Backup” conninfo = host=192.168.0.10 user=barman dbname=postgres port=5432 ssh_command = ssh [email protected] backup_method = rsync streaming_archiver = on streaming_conninfo = host=192.168.0.10 user=barman dbname=postgres archiver = on #slot_name = barman reuse_backup = link # For base backup retention retention_policy = RECOVERY WINDOW OF 7 DAYS # For archive logs retention wal_retention_policy = main # Retention policies are automatically enforced by cron retention_policy_mode = auto minimum_redundancy=2
As written above, the configuration file is structured in two areas:
- Barman Configuration
- Database Server Configuration
In order for the configuration above to work, there are several things which has to be done:
Configure passwordless configuration between barman(barman machine) and postgres(postgreSQL machine). Meaning that the barman user should be able to login, without password to the postgres user on he PostgreSQL. In order to do that, you have to:
- Make sure your barman user' public key is in the authorized_keys file on the remote machine (postgres user)
- Ensure your permissions are for the current user only in the home and .ssh directories (ie 700 or 644)
- Ensure your sshd config allows key authentication
Configuration Separation
You can also separate the configuration of the databases, by splitting the configurations as follows:
- Put all Barman config in /etc/barman.conf
- Put each database config in /etc/barman.d/config_name.conf
You have to of course point that in the /etc/barman.conf as follows:
[root@auxserver etc]# cat /etc/barman.conf [barman] configuration_files_directory = /etc/barman.d <- This line points to the configuration files, in our case: /etc/barman.d/pg.conf barman_home = /var/lib/barman barman_user = barman log_file = /var/log/barman/barman.log compression = gzip bandwidth_limit = 0 immediate_checkpoint = true last_backup_maximum_age = 3 DAYS retention_policy = RECOVERY WINDOW OF 7 days minimum_redundancy = 3
With this we set the configuration for the utility and with this:
[root@auxserver etc]# cat /etc/barman.d/pg.conf [pg] description = “PG Backup” conninfo = host=192.168.0.10 user=barman dbname=postgres port=5432 ssh_command = ssh [email protected] backup_method = rsync streaming_archiver = on streaming_conninfo = host=192.168.0.10 user=barman dbname=postgres archiver = on slot_name = repl_slot reuse_backup = link # For base backup retention retention_policy = RECOVERY WINDOW OF 7 DAYS # For archive logs retention wal_retention_policy = main # Retention policies are automatically enforced by cron retention_policy_mode = auto minimum_redundancy=2
We set up a configuration for the target database machine.
Client Machine
On the client machine there are 4+ things which we have to configure:
- The PostgreSQL utilities(pg_receivexlog)
- A PostgreSQL user: barman
- Access to this machine via pg_hba.conf
- Assure that we have at least +2 more wal_senders
Configuration
PostgreSQL Utilities
Utilities can be stalwartly installed via the YUM:
[root@lpara yum.repos.d]# yum install postgresql96.x86_64 Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile * base: ftp.hosteurope.de * extras: ftp.hosteurope.de * updates: ftp.hosteurope.de Resolving Dependencies --> Running transaction check ---> Package postgresql96.x86_64 0:9.6.8-1PGDG.rhel6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================== Package Arch Version Repository Size ======================================================================================================================================================================== Installing: postgresql96 x86_64 9.6.8-1PGDG.rhel6 pgdg96 1.4 M Transaction Summary ======================================================================================================================================================================== Install 1 Package(s) Total download size: 1.4 M Installed size: 7.0 M Is this ok [y/N]: y Downloading Packages: postgresql96-9.6.8-1PGDG.rhel6.x86_64.rpm | 1.4 MB 00:01 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : postgresql96-9.6.8-1PGDG.rhel6.x86_64 1/1 Verifying : postgresql96-9.6.8-1PGDG.rhel6.x86_64 1/1 Installed: postgresql96.x86_64 0:9.6.8-1PGDG.rhel6 Complete! [root@lpara yum.repos.d]# find / -name "pg_receivexlog" /usr/pgsql-9.6/bin/pg_receivexlog [root@lpara yum.repos.d]# /usr/pgsql-9.6/bin/pg_receivexlog -V pg_receivexlog (PostgreSQL) 9.6.8 [root@lpara yum.repos.d]#
User Creation
User creation is rather simple in PostgreSQL, however since barman will be able to provide replication, he should have a super user rights as follows:
postgres=# create user barman with password 'barman'; CREATE ROLE postgres=# alter user barman with superuser; ALTER ROLE postgres=#
Access to the Machine
Access to the machine is granted via the pg_hba.conf file. In order to grant access to the PostgreSQL machine, use the following syntax:
pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 ident # IPv6 local connections: host all all ::1/128 ident # Allow replication connections from localhost, by a user with the # replication privilege. #local replication postgres peer host postgres rep 127.0.0.1/32 trust host replication rep 192.168.0.11/32 trust #host postgres rep 192.168.0.10/32 trust host all rep 192.168.0.10/32 trust #Barman host all barman 192.168.0.19/32 trust <- For backup on all databases host replication barman 192.168.0.19/32 trust <- For Replication on all databases
PLEASE KEEP IN MIND THAT THIS IS VERY UNSECURE METHOD (Hint. Trust) FEEL FREE TO CHANGE IT TO MD5 or OTHER.
Wal_senders
The wal_senders settings is in the postgresql.conf file and can easily be changed:
[root@postegresqlmaster data]# cat /var/lib/pgsql/9.6/data/postgresql.conf | grep wal wal_level = replica # minimal, replica, or logical #wal_sync_method = fsync # the default is the first option #wal_compression = off # enable compression of full-page writes #wal_log_hints = off # also do full page writes of non-critical updates #wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers #wal_writer_delay = 200ms # 1-10000 milliseconds #wal_writer_flush_after = 1MB # measured in pages, 0 disables #max_wal_size = 1GB #min_wal_size = 80MB max_wal_senders = 4 # max number of walsender processes **<- This setting, right here, has to be increased with 2 :)** #wal_keep_segments = 0 # in logfile segments, 16MB each; 0 disables #wal_sender_timeout = 60s # in milliseconds; 0 disables #wal_receiver_status_interval = 10s # send replies at least this often #wal_receiver_timeout = 60s # time that receiver waits for #wal_retrieve_retry_interval = 5s # time to wait before retrying to
Barman In Action
Once everything is set up, you are ready to do your first backup using barman. From the barman server, issue the following command from the barman user:
-bash-4.2$ barman backup pg
Starting backup using rsync-exclusive method for server pg in /var/lib/barman/pg/base/20180316T100626
Backup start at LSN: 0/28000028 (000000010000000000000028, 00000028)
This is the first backup for server pg
WAL segments preceding the current backup have been found:
000000010000000000000025 from server pg has been removed
000000010000000000000026 from server pg has been removed
Starting backup copy via rsync/SSH for 20180316T100626
Copy done (time: 3 seconds)
This is the first backup for server pg
Asking PostgreSQL server to finalize the backup.
Backup size: 28.9 MiB. Actual size on disk: 28.9 MiB (-0.00% deduplication ratio).
Backup end at LSN: 0/280000F8 (000000010000000000000028, 000000F8)
Backup completed (start time: 2018-03-16 10:06:27.256874, elapsed time: 4 seconds)
Processing xlog segments from streaming for pg
000000010000000000000027
-bash-4.2$
Once you do the backup you can issue a check to see how is the server:
[root@auxserver yum.repos.d]# barman check pg
Server pg:
PostgreSQL: OK
is_superuser: OK
PostgreSQL streaming: OK
wal_level: OK
directories: OK
retention policy settings: OK
backup maximum age: OK (interval provided: 3 days, latest backup age: 23 minutes, 8 seconds)
compression settings: OK
failed backups: FAILED (there are 7 failed backups)
minimum redundancy requirements: FAILED (have 1 backups, expected at least 2)
ssh: OK (PostgreSQL server)
not in recovery: OK
archive_mode: OK
archive_command: OK
continuous archiving: OK
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archiver errors: OK
You see here that we fail to provide redundancy (we have only 1 backup) So I did one more :)
[root@auxserver yum.repos.d]# su - barman
Last login: Fri Mar 16 10:05:06 EDT 2018 on pts/0
-bash-4.2$ barman backup pg
Starting backup using rsync-exclusive method for server pg in /var/lib/barman/pg/base/20180316T102932
Backup start at LSN: 0/2A000028 (00000001000000000000002A, 00000028)
Starting backup copy via rsync/SSH for 20180316T102932
Copy done (time: less than one second)
Asking PostgreSQL server to finalize the backup.
Backup size: 28.9 MiB. Actual size on disk: 8.8 KiB (-99.97% deduplication ratio).
Backup end at LSN: 0/2A000130 (00000001000000000000002A, 00000130)
Backup completed (start time: 2018-03-16 10:29:32.804536, elapsed time: 1 second)
Processing xlog segments from streaming for pg
000000010000000000000029
-bash-4.2$ barman check pg
Server pg:
PostgreSQL: OK
is_superuser: OK
PostgreSQL streaming: OK
wal_level: OK
directories: OK
retention policy settings: OK
backup maximum age: OK (interval provided: 3 days, latest backup age: 31 seconds)
compression settings: OK
failed backups: FAILED (there are 7 failed backups)
minimum redundancy requirements: OK (have 2 backups, expected at least 2)
ssh: OK (PostgreSQL server)
not in recovery: OK
archive_mode: OK
archive_command: OK
continuous archiving: OK
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archiver errors: OK
-bash-4.2$ logout
[root@auxserver yum.repos.d]#