#language en #refresh 999999 = Backup = [[TableOfContents(4)]] Below, the term ‘back up’ refers to storage of data in such a way, that the state of that data at any given day in the past year can be recovered.[[BR]] Conversely, ‘mirroring’ or ‘synchronising’ keeps a copy of the current data. Changes to the data in the past are lost. == Windows 10 == === Requirements === The ssh client must be enabled, see Settings -> "Manage Optional Features" -> "Add a feature" -> "OpenSSH Client" -> "Install".[[BR]] For earlier Windows versions, install OpenSSH for Windows. === Setup === 1. Create the directory `restic-backup` in your home directory. (In fact, you may choose any name, and any directory.) 1. Download the `restic` backup program from https://github.com/restic/restic/releases/latest. (Scroll down to the very bottom of the page.) Extract the restic_*.exe file from the zip archive and store it in the `restic-backup` folder. 1. Rename the restic_*.exe file to `restic.exe`. 1. Download the script [attachment:restic_backup.bat restic_backup.bat] and store it in the directory `restic-backup`. 1. Run the script from the command line, i.e., open a Windows Command Shell, navigate to the restic-backup directory (type `cd restic-backup`) and type `restic_backup `. Here, `` refers to your upTUdate username, examples are `tloimer`, `kcernoho`, `hkuhlman`. You will be asked to confirm a connection, answer with "yes", not only "y". You need to type your upTUdate password. The latter is the password that is accepted by the [https://mail.intern.tuwien.ac.at web mail] interface. 1. Optionally, modify the file `exclude.txt` in the directory `restic-backup`. This file contains patterns for files that should be excluded from backup. 1. To test, again type `restic_backup `. This should create your first backup and take a while, about 4 minutes per 10 GB of data. 1. Under the Windows System Control, open the Task Scheduler (Aufgabenplanung) and create a task, to run the command `%HOMEPATH%\restic-backup\restic_backup yourusername` once a day. Replace %HOMEPATH% by the actual path to restic_backup.bat. The batch file [attachment:restic_backup.bat restic_backup.bat] is amply commented.[[BR]] Help for the restic program is available on the command line by typing `restic help` or at https://restic.readthedocs.io. === Data Restoration === ==== Restoration of a single file ==== In case the full path to the file is known, and the version of the file in the most recent backup is needed: i. open a Windows Command Shell, i. copy the following line to the shell, editing[[BR]] (your short upTUdate account name, e.g., wjandl),[[BR]] (the path to the file you want to retrieve, e.g., /C/Users/Werner Jandl/Document/!DantecManual.pdf) and[[BR]] (the directory where to store the retrieved file to, e.g., C:\Users\Werner Jandl\Desktop) as needed. If or contain spaces, these words must be sourrounded by apostrophes.[[BR]] The command will ask for the password to the repository. Find the password as RESTIC_PASSWORD in the batch file `restic_backup.bat`. That file was created when the backup program was installed. {{{ restic -r sftp:b_backup:/mnt/backup//.restic restore latest -i "" -t "" }}} As an example, the command[[BR]] restic -r sftp:b_backup:/mnt/backup/wjandl/wjandl.restic restore latest -i "/C/Users/Werner Jandl/Documents/!DantecManual.pdf" -t "C:\Users\Werner Jandl\Desktop"[[BR]] would restore the file !DantecManual.pdf to[[BR]] C:\Users\Werner Jandl\Desktop\C\Users\Werner Jandl\Documents/!DantecManual.pdf Note: The name b_backup is an alias that was set up by the script restic_backup.bat, see ~/.ssh/config. The alias enables passwordless login into b.fluid. ==== Custom Restoration ==== Open a Windows Command Shell. Fore ease of use, set the shell variable REPOSITORY. As above, the commands below will ask for the password to the repository. Find the identity (ID) of the snapshot which contains the data to restore. A snapshot is a collection of data that existed at a point in time. {{{ SET REPOSITORY=sftp:b_backup:/mnt/backup//.restic restic -r %REPOSITORY% snapshots }}} The above command prints a table of available snapshots. As an example, see a table of four snapshots below: .`ID Time Host Tags Paths` .`------------------------------------------------------------------------------` .`bf929b85 2020-02-19 14:30:02 LAPTOP-FULA C:\Users\Werner Jandl` .`5e976bd4 2021-01-14 14:30:01 LAPTOP-FULA C:\Users\Werner Jandl` .`1216d749 2021-01-15 14:30:02 LAPTOP-FULA C:\Users\Werner Jandl` .`9da76a45 2021-01-19 14:30:02 LAPTOP-FULA C:\Users\Werner Jandl` .`------------------------------------------------------------------------------` The first 8 letters in each line are the ID of a snapshot. With the ID of the snapshot, list or restore files from that snapshot. The ID ‘latest’ is an alias for the most recent snapshot. {{{ # list all files in a snapshot (you may want to redirect output to a file, >out) restic -r %REPOSITORY% ls # list files below a given directory (recursively, with the option --recursive) # example for : "C/Users/Werner Jandl/Desktop" restic -r %REPOSITORY% ls [--recursive] # restore all data from a snapshot to the current directory # (or the target directory given with the -t option) restic -r %REPOSITORY% restore [-t /target/dir] # restore only the files that match a pattern (-i ). # The directory hierarchy will be restored under the target directory. restic -r %REPOSITORY% restore -i }}} === Troubleshooting === At setup, if an error message regarding permissions of .ssh/config appears, make .ssh/config readable only by the user. == GNU/Linux, macOS, other Unix-like operating systems == === Backup for data less than ≈200 GB === ==== Common Setup ==== * Ask an administrator, T. Loimer or Werner Jandl, to create a home directory for you on b.fluid. * On your desktop computer, check that the program `rsync` is installed (should be installed by default on most systems). * Please note, that directories which have names ending in ".nobackup", "cache", "Cache", or contain a [http://www.bford.info/cachedir/spec.html cachedir-tag], are excluded from the backup. Move voluminous data into such directories! * Save the script [attachment:sync-to-host sync-to-host] to a directory of your choice, for example `~/bin/`; further below, the path to this script must be set accordingly. * Make the script [attachment:sync-to-host sync-to-host] executable (e.g., `chmod +x ~/bin/sync-to-host`). * Copy the following lines to a terminal, adjusting `TU_USER` according to your TU username: {{{ bash # the commands below must be executed in the bash shell export TU_USER=your_tu_username }}} * copy/paste the following lines to a terminal, adjusting `~/bin/sync-to-host` to the actual path to the script, {{{ export TARGET_HOST=b.fluid.tuwien.ac.at export TARGET_DIR=$(hostname -s) # run the backup script once (enter your TU password when asked to do so) ~/bin/sync-to-host ${TU_USER} ${TARGET_HOST} ${TARGET_DIR} }}} * Please, edit the file `~/.config/backup/exclude.txt` and set the patterns of files and directories which should be ignored by the backup. See `man rsync` under `INCLUDE/EXCLUE PATTERN RULES` for the syntax of these patterns. The top of the file [attachment:sync-to-host sync-to-host] contains some comments on how the script works. ==== Automated backups on GNU/Linux ==== In order to set up a cron job for automated client backups on GNU/Linux systems, please copy and paste the following lines into the same terminal as above. Set `SYNC_CMD` according to the path where you saved the [attachment:sync-to-host sync-to-host] script: {{{ export SYNC_CMD="${HOME}/bin/sync-to-host ${TU_USER} ${TARGET_HOST} ${TARGET_DIR}" export SYNC_TIME="$((RANDOM % 60)) * * * *" export CRON_ENTRY="${SYNC_TIME} ${SYNC_CMD}" crontab -l 2>/dev/null | { cat; echo "${CRON_ENTRY}"; } | crontab - }}} ==== Automated backups on macOS ==== In order to set up a launchd job for automated client backups on macOS systems, please copy and paste the following lines into a terminal: {{{ #!/bin/bash export BASENAME=at.ac.tuwien.fluid.b_backup-${USER} export PLIST_PATH=${HOME}/Library/LaunchAgents/${BASENAME}.plist cat > ${PLIST_PATH} < Label ${BASENAME} ProgramArguments ${HOME}/bin/sync-to-host ${TU_USER} ${TARGET_HOST} ${TARGET_DIR} RunAtLoad StartInterval3600 EOF launchctl load ${PLIST_PATH} }}} === Backup for data more than ≈200 GB === The [https://www.borgbackup.org borg] backup program is used to back up the data from the client machine. i. Install the borg backup program. Under ubuntu or debian, on the command line type `sudo apt install borgbackup`. i. Download the script [attachment:borg-user-backup borg-user-backup]. i. Make the script executable, run it and follow the instructions. Comments in the script describe what is done. i. Store the script at a convenient location and create a cron-job to invoke it once a day. For a default setup, after step (ii) above, * copy the script [attachment:borg-user-backup borg-user-backup] to ~/bin/, * copy the first three lines below to a terminal, editing the content of SOURCE_DIRS and TU_USER according to your needs. Press enter. * Then, copy the third to the last line all at once to your terminal and press enter. The setup script will ask for your password, see the comments in [attachment:borg-user-backup borg-user-backup]. A bash shell is required. {{{ bash TU_USER=myusername SOURCE_DIRS="/home/myname/dir1 /home/myname/dir2 /usr/local" chmod +x ~/bin/borg-user-backup ~/bin/borg-user-backup $TU_USER b.fluid.tuwien.ac.at "$SOURCE_DIRS" crontab -l 2>/dev/null | { cat; echo -n "$((RANDOM % 60)) $((RANDOM % 7 + 10)) * * * " echo "$HOME/bin/borg-user-backup $TU_USER b.fluid.tuwien.ac.at $SOURCE_DIRS" } | crontab - }}} = Instructions for admins = == Restoring data from borg == In order to restore data from a backup, backups can be made accessible as a (pseudo-) file system. Users can then access their data using an SFTP or SSH client (e.g. Nautilus on GNU/Linux, Cyberduck on macOS and WinSCP on Windows). For example, in order to expose backup data from `b.fluid` to the user `oswat`, the following commands might be used: {{{ export BACKUP_REPO=/mnt/backup/b/home/oswat.borg export RESTORE_DIR=/mnt/restore/oswat-$(date -Idate) mkdir ${RESTORE_DIR} borg mount -o default_permissions,allow_other --strip-components 2 ${BACKUP_REPO} ${RESTORE_DIR} # --strip-components 2 strip the usually existing '/home/userdir' path components ls ${RESTORE_DIR} }}} In the example above, all snapshots created at different times are available in the first directory below /mnt/restore/oswat-$(date -I date). Note: Unmount as soon as users have restored their data. Otherwise, backups are not possible while the virtual file system is mounted. {{{ umount ${RESTORE_DIR} }}} Alternatively, the entire data, as present at a given date, can be restored. E.g., again for the user oswat on b, {{{ # List available backups borg list --short /mnt/backup/b/home/oswat.borg # extract all data contained in one of the listed backups to /mnt/restore/oswat mkdir /mnt/restore/oswat; cd /mnt/restore/oswat borg extract --strip-components 2 --umask 022 /mnt/backup/b/home/oswat.borg::2022-02-06_05:52 [path] # --strip-components 2: strip home/oswat from the restored path # --umask 022 otherwise, directories might be not searchable # [path] path might be omitted, or, e.g.: home/oswat/machine/home/oswat/dir_in_home }}} To mount a backup repository to s15, from which it was created, as root on s15 do {{{ mkdir /mnt/restore borg mount -o default_permissions,allow_other --strip-components 2 s15-root@b.fluid.tuwien.ac.at:home/oswat.borg /mnt/restore }}} == Restoring data from restic == From the restic-repository, a pseudo file-system can be generated as well. This pseudo filesystem exposes the entire data in the repository. For example, for the user `kcernoho`: {{{ export RESTORE_DIR=/home/kcerneho/restore.cache mkdir $(RESTORE_DIR} RESTIC_PASSWORD=… restic mount -r /mnt/backup/kcernoho/kcernoho.restic --snapshot-template "2006-01-02_15" ${RESTORE_DIR} # probably, use --allow-other? # Unmount after restore umount ${RESTORE_DIR} }}} == Creating home directories on b.fluid == {{{ mkdir /home/tu_user chown tu_user:E322 /home/tu_user install -m 644 -o tu_user -g E322 /etc/skel/.??* /home/tu_user/ }}} == Attachments == The attachments to this page were copied from `b:/home/oswat/backup-scripts/`. A crontab entry for oswat@b once a day calls a script that checks whether the attachments to this page are identical to the files in b:/home/oswat/backup-scripts/. The latter is a git-worktree, clones exist in `b:/opt/borgscripts`, `s15:/opt/borgscripts` and `s16:/opt/borgscripts`.