Let’s say you have access to a remote machine and use it to copy backups occasionally, eg with rsync. Your local machine has credentials stored that allow write access on the remote machine, however if the local account was compromised that could also allow access to the remote machine and the data stored there.
How can you grant access to an account to write remotely, but also protect the data from this account? One possibility could be to change the permissions on the data after it is copied to prevent deletion/interference, although I’m just making this up. Is there a standard practise for this?
Just a small sidenote: If you do not trust your local machine you should think about why and how to change that.
Personally as some extra spice as I worry about ransomeare, I have a few key files I check across my array that should never change. If any of their hashes are off, I abort immediately.
Scheduled snapshots (btrfs or zfs). If the compromised account deletes or modifies files, they’re still there in the past snapshots
Filesystem-level snapshots are quite space-efficient because they don’t make copies of all the files or even whole files; just the blocks that changed.
I have solved that by giving the distant machine the credentials to connect to the local machine. And the distant machine can’t be accessed from the outside.
Do pull backups instead of push backups: Backup server connects to local machine.
What’s the rationale for this? Genuinely curious.
The reasoning is that your backup server should be more secure than production. Production has to have a bunch of stuff open in order to be useful and convenient. The backup server does not. It can be basically fully locked down.
To add - by doing pulls the backup server uses different credentials to run than the credentials used to perform pulls.
Backup server has it’s own credentials database, machines being backed up have their own database. Backup service in backup server uses appropriate credentials from machine being backed up to access the data there (shares, etc). So credentials from compromised machine are unrelated to credentials for backup server.
And if backups are done properly (full on a schedule, daily incrementals, or something similar) you should be able to revert to a known-good state with minimal data loss.
If the main site gets compromised the credentials there must be considered lost and known to che attackers.
with a pull backup that’s not an issue because the main site has no access to the remote system; it is a process on the remote site that has credentials to access the main site and not the other way around.
the remote system may
receiveretrieve a compromised copy of the data, but the attacker cannot tamper with previous backups so recovery is still possible.This is the main reason I had in my head about pull backups. Thanks for the explanation.
A system like proxmox backup server can do this scurely. There you can create a user that can only add new backups and read the existing ones, but cannot delete any or read anything else on the remote host.
Otherwise if you only care to protect the remote machine, then something like an ssh chroot jail would also work.
Encrypt before send, and if you want to have protection against deletions of the data have a cold backup offline other than during the copy.
Append only mode.
Restic has quite a good solution for this: https://restic.readthedocs.io/en/stable/060_forget.html#security-considerations-in-append-only-mode
Funny thing is that blockchains are actually good for this type of thing. Too bad crypto bros got a hold of it.
Append only, like others are saying.
I think you could do it somewhat like hetzner does for their storage boxes. You get an account that has read and write access to a directory and nothing outside. The accound can only run a limited set of commands, like ls, cat, nano, rsync etc. but has no access to commands that modify the filesystem.
Then you can use a copy on write fs like btrfs and make scheduled staggered snapshots.
I usually do 1x per year, 1x per month of current year, 4 per week of current montg, 7 per day in current week.
I have no clue what they use to limit the user accounts like that btw. but maybe that gives you a new jump off point for further research.
What you are talking about is Immutability and an append only backup. The s3 file system and some others has Immutability built in. Not all backups can do append only.
WORM: write once, read many. Any good backup software supports this.
You could also keep offline backups. You can’t compromise what you can’t reach.
You’ve got the right idea with the permission change… the key is that you can have code executing on the remote side with different permissions. So the writer process has permission to write in one directory, and the turnsyle procees (often the root super-user) rotates the files or directories at a different time (or on a signal, sometimes).
I mean depends on the solution you are using, but you can have multiple accounts on the remote backup. IE so upon completion of the backup. The remote machine moves the backup to an offline or read only share (depending if you need those credentials to access the data again later),
Obviously most important thing is your credentials that make the backup… should be very limited in scope to just doing those backups.
