Backup

Contents

  1. Backup
    1. Windows 10
      1. Requirements
      2. Setup
      3. Data Restoration
        1. Restoration of a single file
        2. Custom Restoration
      4. Troubleshooting
    2. GNU/Linux, macOS, other Unix-like operating systems
      1. Backup for data less than ≈200 GB
        1. Common Setup
        2. Automated backups on GNU/Linux
        3. Automated backups on macOS
      2. Backup for data more than ≈200 GB
  2. Instructions for admins
    1. Restoring data from borg
    2. Restoring data from restic
    3. Attachments

Below, the term ‘back up’ refers to storage of data in such a way, that the state of that data at any given day in the past year can be recovered.
Conversely, ‘mirroring’ or ‘synchronising’ keeps a copy of the current data. Changes to the data in the past are lost.

Windows 10

Requirements

The ssh client must be enabled, see Settings -> "Manage Optional Features" -> "Add a feature" -> "OpenSSH Client" -> "Install".
For earlier Windows versions, install OpenSSH for Windows.

Setup

  1. Create the directory restic-backup in your home directory. (In fact, you may choose any name, and any directory.)

  2. Download the restic backup program from https://github.com/restic/restic/releases/latest. (Scroll down to the very bottom of the page.) Extract the restic_*.exe file from the zip archive and store it in the restic-backup folder.

  3. Rename the restic_*.exe file to restic.exe.

  4. Download the script restic_backup.bat and store it in the directory restic-backup.

  5. Run the script from the command line, i.e., open a Windows Command Shell, navigate to the restic-backup directory (type cd restic-backup) and type restic_backup <yourTUusername>. Here, <yourTUusername> refers to your upTUdate username, examples are tloimer, kcernoho, hkuhlman. You will be asked to confirm a connection, answer with "yes", not only "y". You need to type your upTUdate password. The latter is the password that is accepted by the web mail interface.

  6. Optionally, modify the file exclude.txt in the directory restic-backup. This file contains patterns for files that should be excluded from backup.

  7. To test, again type restic_backup <yourTUusername>. This should create your first backup and take a while, about 4 minutes per 10 GB of data.

  8. Under the Windows System Control, open the Task Scheduler (Aufgabenplanung) and create a task, to run the command %HOMEPATH%\restic-backup\restic_backup yourusername once a day. Replace %HOMEPATH% by the actual path to restic_backup.bat.

The batch file restic_backup.bat is amply commented.
Help for the restic program is available on the command line by typing restic help or at https://restic.readthedocs.io.

Data Restoration

Restoration of a single file

In case the full path to the file is known, and the version of the file in the most recent backup is needed:

  1. open a Windows Command Shell,
  2. copy the following lines to the shell, editing USER (your short account name) and FILEPATH as needed.

The commands will ask for the password to the repository. Find the password as RESTIC_PASSWORD in the batch file restic_backup.bat. That file was created when the backup program was installed. 

SET USER=wjandl
SET "FILEPATH=/C/Users/Werner Jandl/Documents"
SET REPOSITORY=sftp:b_backup:/mnt/backup/%USER%/%USER%.restic
restic -r %REPOSITORY% restore latest -i "%FILEPATH%"

Custom Restoration

Open a Windows Command Shell. Fore ease of use, set the shell variable REPOSITORY. As above, the commands below will ask for the password to the repository.

Find the identity (ID) of the snapshot which contains the data to restore. A snapshot is a collection of data that existed at a point in time. 

SET REPOSITORY=sftp:b_backup:/mnt/backup/wjandl/wjandl.restic
restic -r %REPOPSITORY% snapshots

The above command prints a table of available snapshots. As an example, see a table of four snapshots below:

The first 8 letters in each line are the ID of a snapshot.

With the ID of the snapshot, list or restore files from that snapshot. The ID ‘latest’ is an alias for the most recent snapshot. 

# list all files in a snapshot (you may want to redirect output to a file, >out)
restic -r %REPOSITORY% ls <ID>
# list files (recursively, with the option --recursive) in a directory
restic -r %REPOSITORY% ls [--recursive] <ID> /C/USERS/dir/path

# restore all data from a snapshot to the current directory
# (or the target directory given with the -t option)
restic -r %REPOSITORY% restore [-t /target/dir] <ID>
# restore only the files that match the include pattern (-i or --include).
# The directory hierarchy will be restored under the target directory.
restic -r %REPOSITORY% restore <ID> -i '/C/Users/Werner Jandl/lost file.docx'

Troubleshooting

At setup, if an error message regarding permissions of .ssh/config appears, make .ssh/config readable only by the user.

GNU/Linux, macOS, other Unix-like operating systems

Backup for data less than ≈200 GB

The back up is done in two steps,

  1. the data is syncronised, using rsync, to your home directory on b.fluid.tuwien.ac.at,
  2. all data in the home directories on b.fluid is backed up once a day.

In order to use this kind of backup, a correct rsync command must be created. Then, this command is written to your crontab file, such that the rsync-command is invoked once an hour. Follow the instructions below to do this.

Your home directory on b.fluid must be created by an adminstrator, Thomas Loimer or Werner Jandl. You can log in to b.fluid using the credentials of your TU e-mail account.

Please note, that directories which have names ending in ".nobackup", "cache", "Cache", or contain a cachedir-tag, are excluded from the backup.

Common Setup

On your desktop computer, 

bash    # the commands below must be executed in the bash shell
export TU_USER=your_tu_username


export TARGET_HOST=b.fluid.tuwien.ac.at
export TARGET_DIR=$(hostname -s)

# run the backup script once (enter your TU password when asked to do so)
~/bin/sync-to-host ${TU_USER} ${TARGET_HOST} ${TARGET_DIR}

The top of the file sync-to-host contains some comments on how the script works.

Automated backups on GNU/Linux

In order to set up a cron job for automated client backups on GNU/Linux systems, please copy and paste the following lines into the same terminal as above. Set SYNC_CMD according to the path where you saved the sync-to-host script: 

export SYNC_CMD="${HOME}/bin/sync-to-host ${TU_USER} ${TARGET_HOST} ${TARGET_DIR}"
export SYNC_TIME="$((RANDOM % 60)) * * * *"
export CRON_ENTRY="${SYNC_TIME} ${SYNC_CMD}"
crontab -l 2>/dev/null | { cat; echo "${CRON_ENTRY}"; } | crontab -

Automated backups on macOS

In order to set up a launchd job for automated client backups on macOS systems, please copy and paste the following lines into a terminal: 

#!/bin/bash
export BASENAME=at.ac.tuwien.fluid.b_backup-${USER}
export PLIST_PATH=${HOME}/Library/LaunchAgents/${BASENAME}.plist
cat > ${PLIST_PATH} <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>${BASENAME}</string>

  <key>ProgramArguments</key>
  <array>
    <string>${HOME}/bin/sync-to-host</string>
    <string>${TU_USER}</string>
    <string>${TARGET_HOST}</string>
    <string>${TARGET_DIR}</string>
  </array>

  <key>RunAtLoad</key><true/>
  <key>StartInterval</key><integer>3600</integer>
</dict>
</plist>
EOF
launchctl load ${PLIST_PATH}

Backup for data more than ≈200 GB

The borg backup program is used to back up the data from the client machine.

  1. Install the borg backup program. Under ubuntu or debian, on the command line type sudo apt install borgbackup.

  2. Download the script borg-user-backup.

  3. Make the script executable, run it and follow the instructions. Comments in the script describe what is done.
  4. Store the script at a convenient location and create a cron-job to invoke it once a day.

For a default setup, after step (ii) above,

The setup script will ask for your password, see the comments in borg-user-backup. A bash shell is required. 

bash
TU_USER=myusername
SOURCE_DIRS="/home/myname/dir1 /home/myname/dir2 /usr/local"
chmod +x ~/bin/borg-user-backup
~/bin/borg-user-backup $TU_USER b.fluid.tuwien.ac.at "$SOURCE_DIRS"
crontab -l 2>/dev/null | { cat; echo -n "$((RANDOM % 60)) $((RANDOM % 7 + 10)) * * * "
echo "$HOME/bin/borg-user-backup $TU_USER b.fluid.tuwien.ac.at $SOURCE_DIRS"
  } | crontab -

Instructions for admins

Restoring data from borg

In order to restore data from a backup, backups can be mounted to a user's home directory. Users can then access their data using an SFTP client (e.g. Nautilus on GNU/Linux, Cyberduck on macOS and WinSCP on Windows).

For example, in order to expose backup data from b.fluid to the user oswat, the following commands might be used: 

export BACKUP_REPO=/mnt/backup/b/home/oswat.borg
export RESTORE_DIR=/home/oswat/restore-$(date -Idate)
mkdir ${RESTORE_DIR}
borg mount -o default_permissions,allow_other ${BACKUP_REPO} ${RESTORE_DIR}
ls ${RESTORE_DIR}

Note: As soon as users have restored their data, backups should be unmounted again: 

umount ${RESTORE_DIR}

Restoring data from restic

From the restic-repository, a pseudo file-system can be generated as well. This pseudo filesystem exposes the entire data in the repository. For example, for the user kcernoho: 

export RESTORE_DIR=/home/kcerneho/restore
mkdir $(RESTORE_DIR}
RESTIC_PASSWORD=… restic mount -r /mnt/backup/kcernoho/kcernoho.restic --snapshot-template "2006-01-02_15" ${RESTORE_DIR}
# probably, use --allow-other?
# Unmount after restore
umount ${RESTORE_DIR}

Attachments

The attachments to this page were copied from b:/home/oswat/backup-scripts/. A crontab entry for oswat@b once a day calls a script that checks whether the attachments to this page are identical to the files in b:/home/oswat/backup-scripts/. The latter is a git-worktree, clones exist in b:/opt/borgscripts, s15:/opt/borgscripts and s16:/opt/borgscripts.

backup (last edited 2025-06-25 09:50:07 by www)