I have just switched over from offlineimap to mbsync (part of isync) for synchronising my local email with Gmail for use with mu4e. I found offlineimap worked well enough but it would hang frequently if I had connection problems (e.g. on my laptop) and mbsync is much faster. I couldn’t find a guide for migrating, and I encountered a few issues along the way, so I thought I’d document the process here.
The isync mailing list has some discussion of how to convert the maildir folders and related files from offlineimap to mbsync format, but I didn’t see any examples of this being done successfully so I decided to play it safe and sync a new maildir directory with mbsync, downloading a new copy of all of my emails from Gmail’s servers.
I’m using a Mac, and built the newest version of isync from source without problems (the version on Mac Ports is older). I based my
mbsyncrc file on the example on this page, and set up the structure of the maildir to match the one created by offlineimap (not really needed but makes things simpler). Here is my
# mbsyncrc based on # http://www.ict4g.net/adolfo/notes/2014/12/27/EmacsIMAP.html # ACCOUNT INFORMATION IMAPAccount gmail # Address to connect to Host imap.gmail.com User email@example.com Pass ************ AuthMechs LOGIN SSLType IMAPS SSLVersions SSLv3 CertificateFile /opt/local/share/curl/curl-ca-bundle.crt # THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE # - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE # SPECIFICATION OF AN IMAP ACCOUNT) # - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER # REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE) IMAPStore gmail-remote Account gmail # LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail) MaildirStore gmail-local Path ~/mbsync/ Inbox ~/mbsync/INBOX # CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS # # CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl # FOLDERS. SOME COMMONLY USED PATTERS INCLUDE: # # 1 "*" TO MATCH EVERYTHING # 2 "!DIR" TO EXCLUDE "DIR" # 3 "DIR" TO MATCH DIR Channel gmail-inbox Master :gmail-remote: Slave :gmail-local: Patterns "INBOX" Create Both Expunge Both SyncState * Channel gmail-trash Master :gmail-remote:"[Gmail]/Bin" Slave :gmail-local:"[Gmail].Bin" Create Both Expunge Both SyncState * Channel gmail-sent Master :gmail-remote:"[Gmail]/Sent Mail" Slave :gmail-local:"[Gmail].Sent Mail" Create Both Expunge Both SyncState * Channel gmail-all Master :gmail-remote:"[Gmail]/All Mail" Slave :gmail-local:"[Gmail].All Mail" Create Both Expunge Both SyncState * Channel gmail-starred Master :gmail-remote:"[Gmail]/Starred" Slave :gmail-local:"[Gmail].Starred" Create Both Expunge Both SyncState * # GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE # MBSYNC ON A GROUP TO SYNC ALL CHANNELS # # FOR INSTANCE: "mbsync gmail" GETS MAIL FROM # "gmail-inbox", "gmail-sent", and "gmail-trash" # Group gmail Channel gmail-inbox Channel gmail-sent Channel gmail-trash Channel gmail-all Channel gmail-starred
Now I synced with Gmail using
mbsync -V gmail
which worked fine for a while but then kept returning errors from Gmail about throttling and my quota being exceeded. My email is about 6GB in size, so large but not ridiculous, but it seems google didn’t like me trying to download it all in one go (possibly a limitation of my employer’s corporate google service). Anyway, I checked with the isync mailing list and there should be no problems with syncs being interrupted and resumed so it would be possible to keep repeating the sync until it completed. Instead of that I used Apple’s network link conditioner (part of Hardware IO Tools for Xcode) to limit my bandwidth to 750 kbps and left the sync to run overnight, which completed without problems.
Once this was done, I had my offlineimap copy of my maildirs in
~/offlineimap and my mbsync maildirs in
~/mbsync. To make it easy to switch between them if needed, I made a symbolic link from
~/Maildir to the mbsync maildir.
Next I deleted my mu index and re-indexed
rm -rf ~/.mu mu index
I then set some
mu4e variables to point to my new maildir
;;location of my maildir (setq mu4e-maildir (expand-file-name "~/Maildir")) ;;command used to get mail ;; use this for testing (setq mu4e-get-mail-command "true") ;; use this to sync with mbsync ;;(setq mu4e-get-mail-command "mbsync gmail") ;;rename files when moving ;;NEEDED FOR MBSYNC (setq mu4e-change-filenames-when-moving t) ;;set up queue for offline email ;;use mu mkdir ~/Maildir/queue to set up first (setq smtpmail-queue-mail nil ;; start in normal mode smtpmail-queue-dir "~/Maildir/queue/cur")
Now, after restarting mu4e, I was using the mbsync maildir.
Note that I initially used a dummy command to sync the mail in mu4e so that I could run the sync manually on the command line at first. I really only needed this because I kept hitting problems with duplicate UIDs, which are solved by getting mu4e to rename files when moving them, as in the code above.
The switch over has been very smooth. In fact I was surprised to find that the message IDs were preserved so that the org-links I stored to emails when I was using offlineimap still take me to the correct email!