So you might find yourself saying this one day:

HELP! I just deleted a file on my btrfs! It wasn’t old enough to be in a backup, snapshot, or git It was old enough that I don’t want to retype it. How do I get it back? Also, my btrfs is on an SSD, and it might kick off a garbage collection routine at any time!

Well, as luck would have it, I just did that to a blog post I’ve spent all evening writing.

Don’t panic, but don’t wait. Since this is an SSD, the former location of the file was probably marked as garbage for the SSD to clean up. That said, the SSD doesn’t immediately run it’s garbage collection routine whenever you delete a random 8K file. That would be just as bad for wear as not having TRIM/discard at all.

Typical recovery steps

So, lets go through all the typical steps, and why I didn’t do them.

Umount your drive immediately

Can’t. It’s my root and home are subvolumes on that drive.

Shut down the machine, boot from live media

If I was on a hard disk, I would. However, I don’t know how the firmware on my SSD works. Will it decide a powercycle is a great time to do a garbage collection sweep?

Do some fancy btrfs generation rollback magic

I don’t know how. I found Jörg Walter’s btrfs-undelete. Unfortunately, it mentions that if the drive is mounted, it might not work.

Ask on IRC or mail list or reddit

Remember, this file was in my $HOME, is currently “unallocated” space that could be overwritten or garbage collected at any time. “Sooner” is really better than “later”.

How I got my file back

So I remembered I had referenced the path “/var/www/ssl” in the file. This is important, because:

• It’s short, and I know it’s in the file. Trying to remember a longer string potentially means not finding your file because your grammar or punctuation is incorrect.
• This path does not exist on this computer or any other files on this computer, so I should only find copies of this file.

While I’m not umounting the device, it would be stupid to deliberately write data to it. I already had a second drive connected, so I used it for recovery:

$ cd /my/other/drive

Now, I’m going to grep the raw block device for my short, unique string. I’m going to tell grep to output 1000 characters of context (thats 1000 characters before and after a match), and write that to a file. This needs to be done as root, obviously.

$ grep --text -C 1000 "/var/www/ssl" /dev/mapper/vg_w520-btrfs_ssd > pleasework.txt

Once it finishes:

$ ls -lh pleasework.txt 
-rw-r--r--. 1 root root 92M 2015-01-16 03:02 pleasework.txt

Oh boy. 92MB of matches for a 8K file. Lots of matches, since I :w after every sentence. vim isn’t going to like such a big file…

That said, I couldn’t think of a tool I’d rather use than vim for searching through the file. When all you have is a hammer, every problem looks like a nail, after all. After opening it in vim, I I searched for instances of “/var/www/ssl”, and found that there were many, many instances. However, many had been partially overwritten, as in this screenshot below:

But I know my copy is in there, so now I’ll search for a more recent string. One that I typed in one of my most recent edits, just before my mv fiasco. Searching for this string found three copies:

pretty similar to my article covering [[SELINUX and apache (httpd)

They were all very close, but in my case, only one wasn’t corrupted, which is the copy I extracted. Otherwise, I’d have saved all copies and compared them.

Lessons Learned

• Backups, snapshots, and git won’t help you if files are too young to be in any of them.
• You can potentially recover a deleted file, quickly, using just tools you have on-hand (grep & vim).
• Be more careful when doing file operations. When doing a mv *.mdwn, be sure you type the destination. And if you think you typed ls, double check. Or make sure the file you care about is alphabetically first, at least ;)

Caveats

Lots of things could have prevented this from working. I think I was only able to recover due to:

• filesystem behaviour
• vim’s atomic file saves (so writing out a whole fresh copy of the file every time)
• SSD being mostly empty (potentially avoiding aggressive garbage collection cycles)

Or put more simply: I got lucky.

Your mileage may vary.

No warranties.

Best of luck.