backups should be simple and easy.
easily create immutable, trustless backups with revision history, compression, and file deduplication.
the index, tracked in git, contains filesystem metadata.
the index is a sorted tsv file of:
path, tarball, hash, size, mode
for every line of metadata in the index, there is one and only one tarball containing a file with that hash.
duplicate files, by blake2b hash, are never stored.
the index is encrypted with git-remote-gcrypt.
the ignore file, tracked in git, contains one regex per line of file paths to ignore.
a clean restore will clone the git repo, checkout a revision, select file paths by regex, gather needed tarball names, fetch tarballs from storage, and extract the selected files.
backup-add- scan the filesystem for changes.
backup-diff- inspect the uncommitted backup diff.
backup-ignore- if needed, edit the ignore regexes, then goto
backup-commit- commit the backup diff to remote storage.
backup-find- search for files in the index by regex at revision.
backup-restore- restore files from remote storage by regex at revision.
sudo mv bin/* /usr/local/bin
add some environment variables to your bashrc:
export BACKUP_ROOT=~ - root directory to backup
export BACKUP_RCLONE_REMOTE=$REMOTE - a remote setup with
export BACKUP_DESTINATION=$BUCKET/backups/$(hostname) - where to rclone data to
export BACKUP_CHUNK_MEGABYTES=100 - approximate size of each tarball before compression
have a gpg key and a gpg.conf that looks like the following:
>> cat ~/.gnupg/gpg.conf default-key YOUR@EMAIL.COM default-recipient YOUR@EMAIL.COM personal-cipher-preferences AES256 personal-digest-preferences SHA512 personal-compress-preferences Uncompressed default-preference-list SHA512 AES256 Uncompressed cert-digest-algo SHA512 s2k-cipher-algo AES256 s2k-digest-algo SHA512 s2k-mode 3 s2k-count 65011712 disable-cipher-algo 3DES weak-digest SHA1 force-mdc
modify backup state:
backup-add- scan the filesystem for changes
backup-commit- commit the backup diff to remote storage
backup-ignore- edit the ignore file in $EDITOR
backup-reset- clear uncommited backup state
view backup state:
backup-additions-sizes- show large files in the uncommited backup diff
backup-additions- inspect the uncommited backup diff, additions only
backup-diff- inspect the uncommited backup diff
backup-find- find files by regex at revision
backup-index- view the backup index
backup-log- view the git log
restore backup content:
backup-restore- restore files from remote storage by regex at revision
export BACKUP_TEST_RCLONE_REMOTE=$REMOTE export BACKUP_TEST_DESTINATION=$BUCKET/test tox