Backups should be simple and easy.
Easily create immutable, trustless backups with revision history, compression, and file deduplication.
The index, tracked in git, contains filesystem metadata.
The index is a sorted TSV file of: path, tarball, hash, size, mode
For every line of metadata in the index, there is one and only one tarball containing a file with that hash.
Duplicate files, by BLAKE2b hash, are never stored.
The index is encrypted with git-remote-aws.
Chunked tarballs are compressed with lz4 then encrypted with git-remote-aws.
Tarballs are stored on S3 and optionally mirrored to other remotes.
The ignore file, tracked in git, contains one regex per line of file paths to ignore.
A clean restore will clone the git repo, checkout a revision, select file paths by regex, gather needed tarball names, fetch tarballs from storage, and extract the selected files.
backup-add
- Scan the filesystem for changes.backup-diff
- Inspect the uncommitted backup diff.backup-ignore
- If needed, edit the ignore regexes, then goto backup-add
.backup-commit
- Commit the backup diff to remote storage.backup-find
- Search for files in the index by regex at revision.backup-restore
- Restore files from remote storage by regex at revision.bin/
on $PATH
or
sudo mv bin/* /usr/local/bin
Add some environment variables to your bashrc:
export BACKUP_ROOT=~
- Root directory to backup
export BACKUP_S3=s3://${bucket-name}/${backup-name}
- S3 storage for the tarballs
export BACKUP_GIT=aws://${bucket-name}+git-remote-aws/${backup-name}
- Git storage for the index
export BACKUP_CHUNK_MEGABYTES=100
- Approximate size of each tarball before compression
Modify backup state:
backup-add
- Scan the filesystem for changesbackup-commit
- Commit the backup diff to remote storagebackup-ignore
- Edit the ignore file in $EDITORbackup-reset
- Clear uncommited backup stateView backup state:
backup-additions-sizes
- Show large files in the uncommited backup diffbackup-additions
- Inspect the uncommited backup diff, additions onlybackup-diff
- Inspect the uncommited backup diffbackup-find
- Find files by regex at revisionbackup-index
- View the backup indexbackup-log
- View the git logRestore backup content:
backup-restore
- Restore files from remote storage by regex at revisionTests require libaws
export BACKUP_TEST_S3=s3://${bucket-name}/${backup-name}
export BACKUP_TEST_GIT=aws://${bucket-name}+git-remote-aws/${backup-name}
tox
To mirror tarballs to R2 and/or local filesystem, define these optional env vars:
export BACKUP_FS=/mnt/${backup-name}
export BACKUP_R2=s3://${bucket-name}/${backup-name}
On backup, define up to all three.
On restore, define one, or they will be tried in this order:
fs
r2
s3
To use R2, you must define these env vars:
R2_ACCESS_KEY_ID
R2_ACCESS_KEY_SECRET
R2_ACCOUNT_ID