Basic instruction for Linux Mint 17.1 setup as a BackupPC server
for hypervisor client VMs.
Author: Ken Zahorec
date: 2015-02-17-00
=========================================================================
--- System Hardware planning.
I recommend a decent multi-core 64-bit Intel or AMD system. This could be
anything from an enterprise grade server to a moderate desktop system. A lower
end 32-bit system might work as well, but I have not set one up and do not plan
to do so.
Planning the hard drive and boot arrangement can simplify setup later in the
process. In this case we are using two 1TB hard drives to create a single large
data store volume, and then go on with instructions on how to use them.
Alternatively a single drive could be used. Use of LVM, during the Linux Mint
installation, is suggested either way. LVM provides the abilty to easily expand
storage across multiple hard drives such that it looks like a single
volume--when it becomes necessary to do so.
Be careful of boot ordering in the BIOS. Use the first enumerated drive as
first boot order hard drive device. Optionally, for convenience and
simplicity, adjust drive connections to mainboard or drive slots. Arrange
drives such that the desired host system drive will be first device /dev/sda
drive 1: smaller hard drive, for pc-backup host system drive, will become /dev/sda
drive 2: large hard drive, planned for backup storage, will become /dev/sdb
drive 3: large hard drive, planned for backup storage, will become /dev/sdc
Insert all drives into the system including the planned storage drives.
Storage requirements for backuppc server
Generally speaking, and with information from limited testing, we are finding
that we require about 1/5 to 1/10 unit amount of backup storage for each unit
of client storage. Needs may differ for others of course. This means that in
order to backup 150GB of VMs will require about 15GB to 30GB of storage volume
at the backuppc storage host. We could conceivably and safely backup 5 1TB
virtualization servers, with moderate VM loads, using a single 1TB storage
drive. The solution can be used to directly backup server, desktop, and laptop
hosts as well. We are not sure about the storage needs in this regards, but
based on the de-dupe and compression it should be relatively minimal--more
efficient than backing up full VM images.
--- Uninterruptible Power Supply (UPS) is recommended
Use of a UPS is recommended to avoid backup system interruptions during power
failures. One of easiest UPS systems to configure with Linux is an APC UPS
model with a USB connection. There is a driver and daemon "apcupsd" for APC
UPS models readily available from the default repositories. Some other
manufactures use the same USB interface as the APC units.
========================================================================
--- Basic Installation of Linux Mint.
Install Mint 17.1 Cinnamon 64-bit edition
Run full desktop installation
- Boot the installation media to a running trial desktop
- Double click the installation object on the desktop
To start installation from behind a proxy server (if necessary):
- Boot the installation media to a running trial desktop
- Set Proxy in Menu - Preferences - Network
- On desktop, get command line properties of installer, sh -c 'ubiquity gtk_ui'
- Open a terminal and use sudo -E with the command from above
Use sudo -E sh -c 'ubiquity gtk_ui'
During the install, Make sure to use the required system destination drive, not
a planned storage drive. Use defaults, but additionally also include LVM for
main system. We want LVM packages added to the system.
----------------------------------------------
--- Linux Mint setup items
System Settings - Effects
- Disable all desktop effects (uncheck), we don't really need special effects
display and processor loading on a backup server.
System Settings - Network
- Set manual fixed network address
- Set DNS server(s) (optional)
- If business proxy is required, set it as needed.
Menu - Preferences - Network - Proxy
If proxy is used, then all sudo commands using apt-get from this point
on will require the "-E" option to get the users environment passed on. With
this approach, you will only get the proxy settings in your environment if you
run gnome-terminal. This means that if you remotely access the server using
ssh -X, then you will need to additionally start gnome-terminal to use apt-get.
This also means that you will need a local X server to display the graphical
gnome-terminal program UI.
# gnome-terminal &
Update system after initial installation is complete and you have rebooted the
host.
sudo apt-get update
sudo apt-get upgrade
Install graphical LVM management tool
sudo apt-get install system-config-lvm
Install openssh server to provide remote access to host from ssh clients
sudo apt-get install openssh-server
Optionally, after this step we can continue all of the remaining steps from
remote client login through ssh -X. If you are using a proxy to get to
Internet, then this can be accomplished by first logging in to standard
terminal, THEN opening up gnome-terminal, a secondary terminal that contains
the proxy settings in the environment.
ssh -X user@pc-backup_ip_address
gnome-terminal &
Create a temporary backup-storage mount directory in the system. This will be
used as a target to copy the initial backuppc data before the mount point is
moved.
sudo mkdir /mnt/backup-storage
--------------------------------------------
---Creating a very large capacity storage volume with a temporary mountpoint.
Use system-config-lvm to create the large data store for the backup server
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/s1-system-config-lvm.html
Make sure any existing file systems from sdb and sdc are unmounted. Unmount
them if you have to. Assuming we are not using these volumes for the system. If
you have them as part of the system, then go back and install again without
using them.
Menu - Administration - Logical Volume Management
- Init /dev/sdb
- Init /dev/sdc
- Create a new backup storage logical volume group called "storage-vg"
- Put both sdb and sdc into storage-vg
- Create a new backup storage logical volume "storage-lv"
Format storage-lv using XFS
- Use all space
- Apply checkbox to mount on next boot
- Specify the mountpoint /mnt/backup-storage
--------------------------------------------
---Install and configure backuppc solution.
https://www.digitalocean.com/community/tutorials/how-to-use-backuppc-to-create-a-backup-server-on-an-ubuntu-12-04-vps
Install backuppc enterprise grade backup solution
(Added this apache2-utils later on in the first attempt, might instead be
better here. Try it to confirm.)
sudo apt-get install apache2-utils
sudo apt-get install backuppc
Postfix configuration popup during install...
use "Local only" for mail server type
...creates random password for apache e.g. "eWnYdU3a"
To later change password for apache2 access to something more friendly. This
was required on the first try because apache2-utils appears to be a missing
dependency for backuppc install.
sudo apt-get install apache2-utils (if not done previously above)
sudo htpasswd /etc/backuppc/htpasswd backuppc
For running backuppc pre-dump command via rsync we will also need perl module
rsyncp. Without this you will get errors in the log when the backup pre-dump
rsync command is attempted at the client.
sudo apt-get install libfile-rsyncp-perl
--------------------------------------------
--- Adjusting the backuppc server's large storage volume mount point
performing the "switcheroo"
The general idea here is that the filesystem at /var/lib/backuppc needs to be
mounted to the large storage volume. We have to deal with the file system
objects that are currently at this directory before we make the switch. We
could try to just config-change the backuppc storage location over to
/mnt/backup-storage, but this approach is not recommended. We will instead
build the default backuppc storage location into our filesystem--relatively
easy to do.
" 1. mount - as already suggested - your backup file-system directly at the
mount point /var/lib/backuppc"
reference: http://sourceforge.net/p/backuppc/mailman/message/21267701/
The backup target directory /var/lib/backuppc is currently rooted in the root
"/" filesystem--not at the large LVM data store. We need it mounted at the
large LVM data store created earlier to take advantage of our high capacity
hard drives. The large data store volume is currently mounted at
/mnt/backup-storage. We need it mounted at /var/lib/backuppc. We will stop the
backuppc daemon, copy /var/lib/backuppc files over to the current LVM data
store at /mnt/backup-storage. We will then shift the mount point to
/var/lib/backuppc. This will change the backup target over to the large
LVM data store located at the new mount position /var/lib/backuppc with all of
the original file system data objects intact. We are, in essence, pulling a
"switcheroo" on the backuppc storage area.
First stop the backuppc daemon...
sudo /etc/init.d/backuppc stop
Copy the backuppc storage area over to the large volume...
The p preserves all file system attributes, v is verbose, and r is recursive.
sudo cp -pvr /var/lib/backuppc/* /mnt/backup-storage/.
Now we can unmount the backup-storage by issuing the umount command as follows.
sudo umount /mnt/backup-storage
Edit the mount point in /etc/fstab to use the new filesystem mount point at
/var/lib/backuppc (replace "/mnt/backup-storage" with "/var/lib/backuppc")
sudo gedit /etc/fstab
The significant line in fstab should look like the following
/dev/storage-vg/storage-lv /var/lib/backuppc xfs defaults 1 2
Now we can mount the new /var/lib/backuppc mount point
sudo mount /var/lib/backuppc
Voila! we now have the large storage volume mounted at /var/lib/backuppc
The var/lib/backuppc directory has root user root group set. We need to change
it. We must adjust the new backuppc mountpoint to provide "backuppc" user and
group access. This is why we could not mount directly earlier in the
process--there was no backuppc user and group at that point in time.
sudo chown backuppc /var/lib/backuppc
sudo chgrp backuppc /var/lib/backuppc
Umount and mount the fstab "backuppc" mount point. This is done to prove to
ourselves that we can maintain filesystem attributes through a mount.
sudo umount /var/lib/backuppc
sudo mount /var/lib/backuppc
Verify mounting, owner, group, and exec rights.
ls -lah /var/lib/backuppc
....total 12K
drwxr-xr-x 8 backuppc backuppc 118 Jan 20 11:25 .
drwxr-xr-x 70 root root 4.0K Jan 19 19:09 ..
....
(note owner and group on the backuppc directory above in the "." entry
Start the backuppc daemon
sudo /etc/init.d/backuppc start
We are in business! BackupPC should be up and running.
--------------------------------------------
--- Configuring the APC UPS daemon for automated power management
This section provide details on setting up an APC brand uninterupptable
power supply (UPS) for use with the BackupPC server. Most APC models
can simply be connected to the host using a USB cable--allowing the host
to monitor the UPS condition continuously and provide a soft system
shutdown when power goes down for an extended period. This control capability
required the apcupsd power supply daemon be installed and configured in
the host system.
Install the apcupsd power supply daemon.
sudo apt-get install apcupsd
Follow instructions in the CentOS Hypervisor server hypervisor document for
configuration of apcupsd. These instructions cover both dedicated and shared
use of the UPS. Basically you need to edit the following:
sudo gedit /etc/apcupsd/apcupsd.conf
Use the descriptions provided in the file to set the paramaters. For example
suppose you have a dedicated USB connected UPS:
UPSNAME my-ups-name
UPSCABLE usb
etc...
Set the UPS as being configured in the apcupsd defaults
vim /etc/default/apcupsd
ISCONFIGURED=yes
Afterwards, start the apdupsd daemon.
sudo /etc/init.d/apcupsd start
Verify power supply connection and status using the apcaccess command.
apcaccess
Output of apaccess will provide details about the connected power supply.
For example:
APC : 001,023,0596
DATE : 2015-02-09 15:38:52 -0500
HOSTNAME : m17p1-00
VERSION : 3.14.10 (13 September 2011) debian
UPSNAME : local-UPS-CS-350
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2015-02-09 15:04:33 -0500
MODEL : Back-UPS 350
STATUS : ONLINE
BCHARGE : 100.0 Percent
TIMELEFT : 43.0 Minutes
MBATTCHG : 10 Percent
MINTIMEL : 5 Minutes
MAXTIME : 0 Seconds
NUMXFERS : 0
TONBATT : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
STATFLAG : 0x07000008 Status Flag
SERIALNO : BB0323008746
FIRMWARE : 5.1.D USB FW: c1
END APC : 2015-02-09 15:38:59 -0500
--------------------------------------------
---notes on some alternative partitioning for the Linux Mint backuppc host:
An alternative and avoidance to using the "switcheroo" would be to mount the
large storage volume at "/var", as part of the original installation of Mint.
Unfortunately this would complicate the simple Mint installation procedure by
requiring a more complicated partitioning scheme using LVM with additional
volumes. The relative ease of using "system-config-lvm" was the predominant
draw and advantage for putting up with the tradeoff in this next section.
Yet another approach might suggest to mount the large LVM storage to /var
**after** the install of Mint. This method was not verified--**mileage may
vary**. We could run into other, more cumbersome, filesystem and ownership
issues with this approach--operating on a live system. It might possibly
require us to manage the filesystem using an alternative boot media. Do-able,
but we would rather avoid this sticky web.
Another approach would be obviously to just use one very large hard drive for
the Linux Mint backuppc system. This would be the most simple to implement for
sure. As long as LVM is used, adding another drive for expansion of storage
would be quite easy to accomplish. Again, we would use "system-config-lvm" as a
somewhat simple means to do this, but this would not have to be done until
backup storage space started to become an issue.
==========================================================================
--- Configuring backuppc host ssh access to client computers
To use host names for client hosts in backuppc, we need these client hostnames
defined to IP addresses. If we have an available DNS server, we can add some
forward and reverse host records to get the job done. Editing DNS records is
beyond the scope of this instruction set.
As an alternative to DNS, we can add the required definitions directly to
/etc/hosts at the backuppc host. This will provide host name resolution within
the browser view of the backuppc website interface. We would do this only if
we are not providing records for hostnames in a DNS server but want to use
named hosts in the backuppc website interface.
$ sudo gedit /etc/hosts
Without hostnames defined we could forgo hostname usage and instead use direct
IP addresses to add backup clients to backuppc.
--- Create a public key pair for backuppc user
Access the backuppc host from remote client, or simple open a terminal on the
backup host Mint desktop.
$ ssh user@ip_address
We will need a SSH key pair, public and private, for the backuppc user. We
need to create this only **once** on the pc-backup host. The public key will be
sent to the hypervisor client hosts that we plan to provide backup services
for.
Change user, su to backuppc user...
$ sudo su - backuppc
From backuppc user, not sudo, generate a key pair using the ssh-keygen
utility...
$ ssh-keygen
Accept all defaults, we do not want a passphrase in this case.
From backuppc user, transfer the backuppc user's public SSH key to any client
host that will need to be accessed for backup purposes. This will need to be
done **for each client system** that will need to be accessed for backup
purposes.
$ ssh-copy-id root@client_name
You can use the fully qualified or short version of the name. If you are not
using /etc/hosts names or DNS hostnames then, instead of the client_name, use
the direct IP address of the client.
The ssh-copy-id command must be done **for each** client system that require
backup services.
After sending the public key, check to be sure you can access the client, from
backuppc user, without a password.
$ ssh root@client_name
You want to make sure to use the same form of the name you are planning to use
when you add this client host to BackupPC server. For example, if you are
planning to add the host as short name "my-hypervisor4", then use the same form
here. Reason for this is the first time it is done it will add the server name
to the known hosts list for the backuppc user. If you use a different form,
then you will get a preauth failure when the ssh job runs which looks like this
in CentOS /var/log/secure "Connection closed by 10.9.xxx.xxx [preauth]"
Exit from the backuppc user.
$ exit
Exit from the remote host or terminal session.
$ exit
==========================================================================
---Preparation of A CentOS 7 Hypervisor Client for backup via BackupPC
Here we will describe a typical backuppc setup for a CentOS 7 virtualization
server. This is a CentOS 7 hypervisor client host that will be backed up using
backuppc server. Refer here, as an example, on how to set up CentOS 7 as a
hypervisor server. There are many other sources as well.
http://runlevel-6.github.io/blog/2014/12/11/centos-virtualization-server/
We have the vm-clone-dump.sh script copied into the directory below. This
directory, and below, we will use for collection of data for backup purposes.
The virt-backup directory will be used for VM dump and VM dump cleanup of
before and after a backuppc job has completed.
/var/lib/libvirt/images/virt-backup/virt-backup/.
...contains scripts, lockfile, log files
/var/lib/libvirt/images/virt-backup/virt-backup/dump/.
...contains VM cloned images and VM XML definitions for both clone and
original image.
--- dumping VMs to ready them for transfer to backuppc host.
The "vm-clone-dump.sh" script is called as a *pre-command* by the backuppc
process.
vm-clone-dump.sh --dump
This creates a the dump directory and then creates temporary clones of running
and inactive VMs which are are placed in the dump directory. It also copies
the cloned VM XML and original VM XML definition files into the dump directory
before exitting. Each VM is cloned into a folder named the same as the VM name.
The clone image can be used with the original XML to restore and/or move a VM.
--- cleaning up the VM dump after files have been transferred to backuppc
The "vm-clone-dump.sh" script is called as a *post-command* by the backuppc
process.
vm-clone-dump.sh --clean
This cleans up the dump directory area, deletes the dump directory, and readies
the host for another dump operation. These files can be used to recreate the
original VM with the same data at the time it was dumped.
We created, on the hypervisor client, the planned VM backup prep area here.
/var/lib/libvirt/imgages/virt-backup
This location was chosen because on our hypervisor clients we have a large LVM
partition mounted at /var/lib/libvirt/images. We want to use part of the same
LVM partion to store local VM dumps in preparation for transfer to backup
storage. We are logged into the client system as root.
Future considerations: The choice of this directory was due to the large LVM
storage volume on the hypervisor being mounted at /var/lib/libvirt/images/. An
improvement, at this point, might to be instead mount the large LVM storage
volume in the hypervisor instead to the /var directory This would permit a more
attractive and independent path for VM backups, such as /var/backuppc/. This
would allow the backuppc scripts and dump area to be established in a separate
directory area independent of the libvirt specific sub-path. Future iterations
of the hypervisor installation could be implemented in this way allowing for
the following:
/var (large LVM storage volume)
/var/lib/libvirt/images (default libvirt related VM images)
/var/lib/libvirt/iso-images (libvirt related iso-images)
/var/backuppc (backuppc related scripting)
/var/backuppc/dump (backuppc VM dump data)
==========================================================================
---Tuning the solution to fit the hypervisor capabilities and loading
Through testing on several server systems I have found that running the VM dump
operation sequentially on each running VM, while other VMs are running, can
cause tremendous strain on a hypervisor resulting in VMs operating **very
slowly** during the dump operation. On older server systems this could look
like a broken server. This issue depends on the loading and capabilities of
the hypervisor. You get very different capabilities between an older desktop
system or older Dell 2950 and a brand shiny new Cisco blade with 32 Xeon cores,
1TB RAM, and full solid-state RAID storage system. The 2 options below can be
used to provide regular backup data with the tradeoff of halting VM usability
at the hypervisor for the time it takes to dump the VM data.
---Option: Disable all automated backups.
Disable all host_name automated backups and require the use of admin "backuppc"
user manual backup request only. This essentially means that when a client
host backup is needed, the backuppc user must go into the website and request a
host backup.
host_name - Edit Config - Schedule
BackupsDisable: 1
---Option: Throughput Increase for Clone Dump Processing
1. The vm-clone-dump.sh will, by default, pause **all** running VMs in the
hypervisor before creating dumped clones. Preventing concurrent use of VMs
while dumping may shorten the dump processing time in the hypervisor. This
however will prevent use of VMs while the hypervisor dump is being prepared and
backup files transferred--a reasonable tradoff.
2. Change vm-clone-dump.sh processing to be --concurrent, such that each VMs is
individually paused, and then resumed after it is cloned and dumped. This will
generally improve the uptime of VMs during a hypervisor clone dump
operation--assuming the server can handle the load. The tradoff may be a
degradation of performance on VMs during the clone creation--VM users may
become frustrated. This approach is not recommended on older servers with
relatively slower disk IO. Newer solid state storage may be required here to
obtain the desired concurrent effect while keeping the other VMs humming along.
==========================================================================
---BackupPC Administrative website interface
Browser access to backuppc administration site to add and configure clients
Access the backup host using HTTP from a remote client with a browser. You
could also access backuppc administration directly from the Mint desktop using
Firefox.
http://host_name_or_ip_address/backuppc
provide credentials:
user: backuppc
pass: the_password
Edit Hosts - Delete "localhost"
Remove the localhost from the backup configuration.
Once you have deleted the localhost from backuppc administration you can then
delete the localhost backup files from the backuppc host.
From the backuppc host we can remove the localhost backup files.
$ sudo rm -rf /var/lib/backuppc/pc/localhost
Before we can add any hypervisor host clients we need to prepare them using the
CentOS client instructions above as a working example.
In the backuppc administrative interface we will want to add the hypervisor
host clients we want to have backed up.
Edit Hosts - Add button
Provide the name of the hypervisor client host and then "Save". Keep in mind,
if you are using names, then you need to have either a DNS server entry or a
hosts file entry as detailed above in BackupPC setup instructions.
Choose the dropdown "Select a host..."
Now you can select the host from the pull down in the upper left of the
administration website. This will bring the new host into context. Now you
can edit the client host specific config by selecting the "Edit Config" link
directly below the host in context.
All of the configuration for the host is available globally in the general host
settings. Editing a specific host configuration can be done when you need to
configure specific parameters for a given client host. By creating a
configuration for the system, each host can then use the same configuration.
Host - Edit Config
Host config "Xfer" tab settings are as follows:
XferMethod: rsync
RSyncShareName: /var/lib/libvirt/images/virt-backup
Host config "Backup Settings" tab settings
DumpPreUserCmd:
$sshPath -q -x -l root $host /var/lib/libvirt/images/virt-backup/vm-clone-dump.sh --dump
DumpPostUserCmd:
$sshPath -q -x -l root $host /var/lib/libvirt/images/virt-backup/vm-clone-dump.sh --clean
Most of the other settings can remain as default. Schedule can be reviewed and
adjusted as necessary. We are now ready to start a full backup.
Select the "hostname Home" then select "Start Full Backup". Check the host and
server logfiles for errors and/or progress. The backup should complete. It
may take an hour or more, depending on the number of VMs, throughput of the
hypervisor client, etc.
---------------------------------------------------------------------
- end of document