Offlineimap repeatedly deleting All Mail

February 28, 2013

or “That’s why you read changelogs”

I was having some significant difficulties with offlineimap after upgrading to 6.5.4 (I’m using Ubuntu currently, so I upgraded straight from 6.3.4).

The original issue was a FolderValidity error, which is covered by the offlineimap FAQ. This affected my “All Mail” folder (It also affected my “Sent Mail”, but it got lost in the noise of the 67964 messages in All Mail). The recovery process is simply to remove that folder (and it’s sync history), and start over. Unfortunate, but fine. That’s why I’ve got a 28Mbit connection, after all.

Offlineimap happily re-downloads my messages:

$ du -sh "All Mail"
1.2G    All Mail

$ find "All Mail" -type f | wc -l
67964

However, the second sync deletes all my messages! (Only locally in this case! Remember to keep ~/.maildir in git!).

The third sync downloads them all again. WTF.

Several hours and gigabytes of troubleshooting later, it turns out that OfflineIMAP 6.4 (which I had skipped) introduced the ability to create remote folders. Previously, only folders that existed on the remote would be synced. Local-only folders were effectively an island. Nothing here should cause an issue.

However, I was using a feature called “nametrans”. Nametrans is applied at the “Repository” level (Remember: Offlineimap accounts consist of two repositories: typically a local and remote). My nametrans rule was configured in the remote gmail repository.

nametrans = lambda foldername: re.sub('^(\[Gmail\]/)', '', foldername)

The result of this being that the “[Gmail]” prefix applied to Gmail’s built-in labels is removed when syncing the folder’s name. For example, the remote folder “[Gmail]/All Mail” gets translated to “All Mail” before being synced with your local repository.

Remote:"[Gmail]/All Mail -> nametrans -> Local:“All Mail”

The change is that now non-existent remote folders can now be created. In previous versions, this would never have occured:

Local:“All Mail -> nametrans -> Remote:“All Mail”

As you can see, a new remote folder called “All Mail” gets created.

The problem here becomes obvious: One local folder effectively maps to two remotes, and both are conflicting (one is empty, one is full)

If I had actually read the changelog, I would have seen this under “OfflineIMAP v6.3.5-rc2”:

Folders will now also be automatically created on the REMOTE side of an account if they exist on the local side. Use the folderfilters setting on the local side to prevent some folders from migrating to the remote side. Also, if you have a nametrans setting on the remote repository, you might need a nametrans setting on the local repository that leads to the original name (reverse nametrans).

The giant red warning on the nametrans documentation also helps, if you know what you’re looking for.

The “reverse nametrans” for my local folder is:

nametrans = lambda foldername: re.sub('^(Sent Mail|Drafts|Trash|All Mail|Spam)$', '[Gmail]/\\1', foldername)

Also note that my configuration is assuming sep = /. Also note that if your language is not en_us (in the website), your folder names will be in your local language or terms (“Bin” instead of “Trash” for en_uk, etc.).

You’ll want to remove the duplicate labels on the Gmail side before attempting to sync, otherwise you’ve not fixed the issue (multiple remotes map to one local). Gmail actually prefixes these bad folders with “[imap]”, which is nice.

Ideally, offlineimap would do two additonal things:

Warn if a nametrans is only applied to one side.
Pre-calculate and preserve the nametrans mappings before performing the sync, so a single folder can’t be referenced twice.