Category: Backup

KVM VM Image Crash

So, something bad happened! My Virtual Machine running on my NAS crashed.

And it looked like this…

2020-05-08T19:51:09.582896Z qemu-system-x86_64: -drive file=/share/Storage/VM/Windows 10 Enterprise/Windows 10 Enterprise.img,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback: qcow2: Image is corrupt; cannot be opened read/write

So I did what I normally do, I started googling, and found a bunch of old articles saying that I had to install some “nbd” module in the kernel and then run “ddrescue”, most of the articles I found pointed to one very sad solution, scrap your VM and reinstall, because you’re never getting your data back.

Anyway, all of that seemed pretty old (and sad), and I thought there must be a better way. And guess what, there was!

First I ran a command called “qemu-img” with the command “check”, and It looked like this:

./qemu-img check /share/Storage/VM/Windows\ 10\ Enterprise/Windows\ 10\ Enterprise.img

When that was done it gave this:

<WALL OF TEXT> (list of all errors in the Image), and then a Summary:

2047 errors were found on the image.
Data may be corrupted, or further writes to the image may corrupt it.
17593 leaked clusters were found on the image.
This means waste of disk space, but no harm to data.
802489/4096000 = 19.59% allocated, 4.83% fragmented, 0.00% compressed clusters
Image end offset: 53754200064

I liked the part about “no harm to data”. So I ran the second command that I had found, which was:

./qemu-img check -r all /share/Storage/VM/Windows\ 10\ Enterprise/Windows\ 10\ Enterprise.img

And after that command finished:

The following inconsistencies were found and repaired:
17593 leaked clusters
1024 corruptions
Double checking the fixed image now…
No errors were found on the image.
802489/4096000 = 19.59% allocated, 4.83% fragmented, 0.00% compressed clusters
Image end offset: 53754331136

And that solved it!

Lesson learned: Scheduled Backup of your VM = good thing.

AWS Glacier!

Hard drives will fail!

Manufacturers of hard drives like to boast about “MTBF” (Mean Time Between Failures) and “AFR” (Annualized Failure Rate) rates that indicates that the drive would last anywhere from 300 000 hours (~34 years) to having a likely hood of failing during a year as low as 0,8%.

But real life tests, performed by companies like BackBlaze show that disk drives can fail already after 1 year, actually they had a 5.1% AFR during the first year, 1.4% between 1,5 years to 3 years and a staggering 11.8% between the third and forth year. Their study also shows that after 4 years, it was only 80% of the original disk drives that where still “alive”. Meaning that 20% of the drives had failed after 4 years. 

So, why does this matter?

I have videos, photos and important documents that I need to backup and in some cases archive. Archiving being “removing the local copy and storing the item in a secure place” (think of book keeping, where it’s required to archive the books, it’s not required that you have multiple copies in case of a fire), where as backup is a second copy of the object in another location in case of a fire. We’re talking wedding photos, wedding videos, photos of my son growing up etc. that, in the case of an unforeseen catastrophic event in my home, could potentially be lost for ever!

Companies like BackBlaze offer products for backing up your local machine (and also files stored on a server, like in my case), but I optioned for another setup using Amazon Web Services S3 Glacier!

So what is S3 Glacier? Glacier is an archiving service, where you can upload all you data and pay a very small service fee per month for Amazon to keep your data safe. Amazon S3 Glacier is designed to provide average annual durability of 99.999999999% for an archive. The service redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon S3 Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives. S3 Glacier performs regular, systematic data integrity checks and is built to be automatically self-healing.

My setup consists of a software running on my local NAS that monitors specific places on the NAS where I can put important files and documents, these are then uploaded to Amazon S3 Glacier.

Putting files into Glacier is actually quite easy and the price for storing your data in Glacier is very modest, it’s when you want to retrieve the files that you might need to pay (it also depends on how much data you want to retrieve and how fast you need your files).

As there is no GUI for exploring AWS Glacier Vaults (where the files are stored), I found a really great application online called Freeze that makes life much easier for anyone who wants to explore their files and retrieve them from a UI. 

So are you in the market for a simple, easy to use and very secure way of storing files, documents, pictures, videos etc. of your beloved ones, and you want to be sure that even if your house burns down you will still have access to those files? Try out AWS Glacier with the Freeze App!

(This is not a payed advertisement, BackBlaze, AWS and Freeze App have not been consulted and/or informed about this blog post)

Backup your VPS to Google Drive

If you only have a small amount of data to backup, Google Drive is actually quite a decent alternative, but you will need to install a few tools to you VPS.

With the tool rclone (rsync for cloud), you can upload your files to Google Drive, Amazon S3, Dropbox and a few more. The procedure is almost the same no matter of the cloud provider you go for.

Just goto:

Install rclone as described and then setup a cronjob to sync the files that you want to Backup.

My crontab to backup the automysqlbackup folders:

# m h  dom mon dow   command
  0 *   *   *   *    export PATH=$PATH:/home/siho/bin && rclone -q sync /var/lib/automysqlbackup/ remote:Backup/automysqlbackup