Using rclone to transfer backups to Google Drive

[wpinsites]

Basic how-to for installing Rclone, configuring it and setting up a script to transfer backups to your Google Drive.
The script can be called via cron to automate the process.
Includes ability to specify how many remote backups you wish to keep.

Why? Some of the other options for storing backups on Google Drive require building from source on more recent versions of Debian. Rclone is a relatively easy solution.

Install and Configure Rclone

Install Rclone:

Code: Select all

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Accepting the defaults throughout the process is fine.

Configure Rclone:

Code: Select all

rclone config

Note: There are a lot of questions in the configuration process, most of which can be left at their default values by simply pressing "Enter."

As it is a fresh install, rclone tells us there are no remotes configured.
Press "n" then "Enter" to create a new remote. It will prompt you for a name. For this example we're going to call it "google-drive" but you can use whatever name you like.

You will be presented with a long list of storage options. Scroll through and find the number which corresponds to Google Drive, type that number then "Enter".

You will be prompted for a Google Application Client ID. You can just press "Enter" to use the built-in ID and secret, but these are low performance, so creating your own is highly recommended. If you aren't familiar with that process, visit the URL in the prompt for instructions.

You will then be prompted for the Google Application Client Secret. If you're using the built-in ID, just press "Enter" again, or enter the Secret for your own ID then press "Enter".

You will be asked for the Scope for the application to use. We want the first option so press "1" then "Enter".

If you are asked for the ID of the root folder just leave it blank and press "Enter". For some reason this is not always asked.

At the "Edit Advanced Config?" prompt just press "Enter".

This is where it gets a little tricky. You will be prompted to use Autoconfig, but autoconfig is designed for systems with a GUI and web browser installed, which isn't usually the case for servers. To get around this and be able to do the authorisation on your desktop computer we use an SSH tunnel:

Mac / Linux users simply open a terminal and enter:

Code: Select all

ssh -L localhost:53682:localhost:53682 username@remote_server

Replacing username and remote_server with your server FQDN or IP and your username for the server.

For Windows users this example uses PuTTY. Open a Command Prompt window and enter:

Code: Select all

plink.exe -N -L localhost:53682:localhost:53682 username@remote_server

Replacing username and remote_server with your server FQDN or IP and your username for the server.

Once the tunnel is in place, on the server type "Y" then "Enter" for the autoconfigure question. Some text will be displayed, amongst which you will see a URL in this format:
http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx

Copy that URL and paste it into a browser on your computer to complete the authorisation. You will see a standard Google Auth window asking you to select your Google account etc. Complete the authorisation and you will see a message appear on the server indicating that the auth has been successful.

Create the Remote Backup script

It is up to you where and exactly how you do this part. Ultimately you would want this script to be called as a cron task so that it transfers your backups periodically and purges old ones from your Google Drive. You can also optionally have it remove the local backups to save space on the local storage (default configuration). The following is the content of a basic BASH script that you can paste into any editor and save with a .sh extension, maybe something like remote-backups.sh or whatever fits your needs. You will need to modify key parts of this script to reflect your configuration, including the name you gave your Google Drive during the Rclone setup and the folder name(s) on your Google Drive where you want the backups stored. All options are quite straightforward.

Code: Select all

#!/bin/bash

# BEGIN CONFIGURATION ==========================================================

	  # The local directory in which the vesta backups reside
	  BACKUP_DIR="/backup"
	  # The Gsuite user that owns the Google Drive
	  GDRIVE_USER="<YOUR GOOGLE ACCOUNT EMAIL>"
	  # The rclone designation for the Google Drive
	  GDRIVE_NAME="Gdrive"
	  # The path on the Google Drive where the backups will reside
	  GDRIVE_DIR="<REMOTE FOLDER NAME>"
	  # The period to keep backups remotely
	  RETENTION="7d"
	  
	  # Set to --dry-run to test script
	  DRY_RUN=""

	  # You probably won't have to change these
	  THE_DATE="$(date '+%Y-%m-%d')"
	  RCLONE_PATH="$(which rclone)"

# END CONFIGURATION ============================================================

# Announce the start time
echo "Started: $(date)"

# Rclone backups to Google Drive
echo "------------------------------------"
echo "Sending backups to Google Drive..."
$RCLONE_PATH copy $DRY_RUN --ignore-existing --drive-impersonate $GDRIVE_USER $BACKUP_DIR $GDRIVE_NAME:"$GDRIVE_DIR" --include "*.tar"
RC=$?
if [ $RC -ne 0 ]; then
    echo "Sending backups to Google Drive FAILED!!!"
    exit $RC
else
	echo "------------------------------------"
	echo "Removing backups older than retention period..."
	$RCLONE_PATH delete $DRY_RUN --min-age $RETENTION --drive-impersonate $GDRIVE_USER $GDRIVE_NAME:"$GDRIVE_DIR" --include "*.tar"
	RC=$?
	if [ $RC -ne 0 ]; then
		echo "Removing old backups FAILED!!!"
		exit $RC
	else
		echo "------------------------------------"
		echo "Removing local backups..."
		$RCLONE_PATH delete $DRY_RUN $BACKUP_DIR --include "*.tar"
		RC=$?
	fi
fi

# Announce the end time
echo "Finished: $(date)"
exit $RC

Once the script has been saved don't forget to make it executable:

Code: Select all

chmod +x filename.sh

That's it, other than adding it to cron, which you can do via the MyVesta Dashboard under the CRON tab.

Have fun.