Migrating from offlineimap to mbsync for mu4e

I have just switched over from offlineimap to mbsync (part of isync) for synchronising my local email with Gmail for use with mu4e. I found offlineimap worked well enough but it would hang frequently if I had connection problems (e.g. on my laptop) and mbsync is much faster. I couldn’t find a guide for migrating, and I encountered a few issues along the way, so I thought I’d document the process here.

The isync mailing list has some discussion of how to convert the maildir folders and related files from offlineimap to mbsync format, but I didn’t see any examples of this being done successfully so I decided to play it safe and sync a new maildir directory with mbsync, downloading a new copy of all of my emails from Gmail’s servers.

I’m using a Mac, and built the newest version of isync from source without problems (the version on Mac Ports is older). I based my mbsyncrc file on the example on this page, and set up the structure of the maildir to match the one created by offlineimap (not really needed but makes things simpler). Here is my mbsyncrc

# mbsyncrc based on
# http://www.ict4g.net/adolfo/notes/2014/12/27/EmacsIMAP.html
# ACCOUNT INFORMATION
IMAPAccount gmail
# Address to connect to
Host imap.gmail.com
User name@address.com
Pass ************
AuthMechs LOGIN
SSLType IMAPS
SSLVersions SSLv3
CertificateFile /opt/local/share/curl/curl-ca-bundle.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore gmail-remote
Account gmail

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore gmail-local
Path ~/mbsync/
Inbox ~/mbsync/INBOX

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR

Channel gmail-inbox
Master :gmail-remote:
Slave :gmail-local:
Patterns "INBOX"
Create Both
Expunge Both
SyncState *

Channel gmail-trash
Master :gmail-remote:"[Gmail]/Bin"
Slave :gmail-local:"[Gmail].Bin"
Create Both
Expunge Both
SyncState *

Channel gmail-sent
Master :gmail-remote:"[Gmail]/Sent Mail"
Slave :gmail-local:"[Gmail].Sent Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-all
Master :gmail-remote:"[Gmail]/All Mail"
Slave :gmail-local:"[Gmail].All Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-starred
Master :gmail-remote:"[Gmail]/Starred"
Slave :gmail-local:"[Gmail].Starred"
Create Both
Expunge Both
SyncState *

# GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE
# MBSYNC ON A GROUP TO SYNC ALL CHANNELS
#
# FOR INSTANCE: "mbsync gmail" GETS MAIL FROM
# "gmail-inbox", "gmail-sent", and "gmail-trash"
#
Group gmail
Channel gmail-inbox
Channel gmail-sent
Channel gmail-trash
Channel gmail-all
Channel gmail-starred

Now I synced with Gmail using

mbsync -V gmail

which worked fine for a while but then kept returning errors from Gmail about throttling and my quota being exceeded. My email is about 6GB in size, so large but not ridiculous, but it seems google didn’t like me trying to download it all in one go (possibly a limitation of my employer’s corporate google service). Anyway, I checked with the isync mailing list and there should be no problems with syncs being interrupted and resumed so it would be possible to keep repeating the sync until it completed. Instead of that I used Apple’s network link conditioner (part of Hardware IO Tools for Xcode) to limit my bandwidth to 750 kbps and left the sync to run overnight, which completed without problems.

Once this was done, I had my offlineimap copy of my maildirs in ~/offlineimap and my mbsync maildirs in ~/mbsync. To make it easy to switch between them if needed, I made a symbolic link from ~/Maildir to the mbsync maildir.

Next I deleted my mu index and re-indexed

rm -rf ~/.mu
mu index

I then set some mu4e variables to point to my new maildir

;;location of my maildir
(setq mu4e-maildir (expand-file-name "~/Maildir"))

;;command used to get mail
;; use this for testing
(setq mu4e-get-mail-command "true")
;; use this to sync with mbsync
;;(setq mu4e-get-mail-command "mbsync gmail")

;;rename files when moving
;;NEEDED FOR MBSYNC
(setq mu4e-change-filenames-when-moving t)

;;set up queue for offline email
;;use mu mkdir  ~/Maildir/queue to set up first
(setq smtpmail-queue-mail nil  ;; start in normal mode
      smtpmail-queue-dir   "~/Maildir/queue/cur")

Now, after restarting mu4e, I was using the mbsync maildir.

Note that I initially used a dummy command to sync the mail in mu4e so that I could run the sync manually on the command line at first. I really only needed this because I kept hitting problems with duplicate UIDs, which are solved by getting mu4e to rename files when moving them, as in the code above.

The switch over has been very smooth. In fact I was surprised to find that the message IDs were preserved so that the org-links I stored to emails when I was using offlineimap still take me to the correct email!

Advertisement

Fixing duplicate UID errors when using mbsync and mu4e

This is a particularly niche post but hopefully will save someone the time I spent trying to solve this problem.

I have migrated from offlineimap to mbsync (part of isync) for synchronising my local email with Gmail. mbsync is much faster and I’ve been happy with the switch but I kept encountering error messages from mbsync about duplicate UIDs. The form of the message was Maildir error: duplicate UID 12 (or some other number).

After a lot of digging around I found the solution was in the mbsync and mu4e manuals (of course). The problem is caused because mu4e does not by default rename files when moving them to a new directory and this then causes problems for mbsync. This is easily fixed by setting

(setq mu4e-change-filenames-when-moving t)

Move cursor to top/middle/bottom of window

As mentioned previously you can use C-l to recentre the window on the line containing the cursor (point in emacs terminology). A related command is move-to-window-line-top-bottom which is bound to the short-cut M-r. Use this repeatedly to cycle the position of the cursor (point in emacs terminology) between the top, middle and bottom of the window.

In other words, C-l recentres the window while keeping the point on the same line, while M-r moves the point without recentring the window.

Tweaking email contact completion in mu4e

The excellent emacs email client mu4e has very good completion of email contacts, which are ranked in a smart way. I have made a few tweaks to this to better suit my needs which I’ll describe below. In a nutshell, they let me

• add favourites to the start of the completion list
• offer a list of contacts as soon as I compose an email
• insert a contact anywhere

For my favourite contacts, I create a text file with contacts in the form

First Last <name@blah.com>

I then create a list of contacts by appending the sorted list of mu4e contacts to this list of favourites and removing duplicates. This forms the final contact list that is offered for completion.

The favourite list used to be more useful before the sorting of contacts in mu4e was improved in the last release. It still has one nice use though, which is that I can specify the name I want to appear for each contact, which otherwise mu4e takes from the message header. This is useful since my email templates make use of this name to address the email, but sometimes these are unhelpful. For example “B Maughan <benmaughan@blank.com>” in the default mu4e contact list would cause my template to start “Hi B,” but if I put “Ben Maughan <benmaughan@blank.com>” in my favourites file, this is the one that will end up in the TO field and so I will get a nice “Hi Ben,” in my template.

I wrap this in a function using the ivy completion library, based on an example by Jon Kitchin that used helm, and bind it to S-TAB so that when I hit SHIFT and TAB in the TO field of an email I get this tweaked version of the address completion.

;;need this for hash access
(require 'subr-x)

;;my favourite contacts - these will be put at front of list
(setq bjm/contact-file "/full/path/to/fave-contacts.txt")

(defun bjm/read-contact-list ()
  "Return a list of email addresses"
  (with-temp-buffer
    (insert-file-contents bjm/contact-file)
    (split-string (buffer-string) "\n" t)))

;;ivy contact completion
;;based on http://kitchingroup.cheme.cmu.edu/blog/2015/03/14/A-helm-mu4e-contact-selector/
(defun bjm/ivy-select-and-insert-contact (&optional start)
  (interactive)
  ;;make sure mu4e contacts list is updated - I was having
  ;;intermittent problems that this was empty but couldn't see why
  (mu4e~request-contacts)
  (let ((mail-abbrev-mode-regexp mu4e~compose-address-fields-regexp)
        (eoh ;; end-of-headers
         (save-excursion
           (goto-char (point-min))
           (search-forward-regexp mail-header-separator nil t)))
        ;;append full sorted contacts list to favourites and delete duplicates
        (contacts-list
         (delq nil (delete-dups (append (bjm/read-contact-list) (mu4e~sort-contacts-for-completion (hash-table-keys mu4e~contacts)))))))
    (when (and eoh (> eoh (point)) (mail-abbrev-in-expansion-header-p))
      (let* ((end (point))
             (start
              (or start
                  (save-excursion
                    (re-search-backward "\\(\\`\\|[\n:,]\\)[ \t]*")
                    (goto-char (match-end 0))
                    (point))))
             (contact
              (ivy-read "Contact: "
                        contacts-list
                        :re-builder #'ivy--regex
                        :sort nil
                        :initial-input (buffer-substring-no-properties start end))))
        (unless (equal contact "")
          (kill-region start end)
          (insert contact))))))

;;bind it
(define-key mu4e-compose-mode-map (kbd "<S-tab>") 'bjm/ivy-select-and-insert-contact)

Next I add this function to the hook that runs when I compose an email in mu4e, which launches me straight into address completion:

;;launch automatically
(add-hook 'mu4e-compose-mode-hook 'bjm/ivy-select-and-insert-contact)

Finally, here is a function that lets you insert a contact from your list anywhere in your email (not just the address fields). It will also work in any other buffer once you have started mu4e for the first time to initialise the contacts list.

;;ivy contacts for use anywhere
;;based on http://kitchingroup.cheme.cmu.edu/blog/2015/03/14/A-helm-mu4e-contact-selector/
(defun bjm/ivy-select-and-insert-contact-anywhere ()
  (interactive)
  (let (contacts-list contact)
    ;;append full sorted contacts list to favourites and delete duplicates
    (setq contacts-list
          (delq nil (delete-dups (append (bjm/read-contact-list) (mu4e~sort-contacts-for-completion (hash-table-keys mu4e~contacts))))))
    (setq contact
          (ivy-read "Contact: "
                    contacts-list
                    :re-builder #'ivy--regex
                    :sort nil))
        (unless (equal contact "")
          (insert contact))))

Quickly preview images and other files with peep-dired

The package peep-dired lets you quickly scan through images (or any other documents) with previews in dired. In the example below, I invoke peep-dired (I bind it to P) and scan through a set of images in a dired listing. When I am finished I hit P again to quit peep-dired.

Here is my code to install and configure peep-dired with use-package

;;preview files in dired
(use-package peep-dired
  :ensure t
  :defer t ; don't access `dired-mode-map' until `peep-dired' is loaded
  :bind (:map dired-mode-map
              ("P" . peep-dired)))

Reschedule multiple items in org agenda

I (all too) often find myself failing to complete all the tasks I schedule for a particular day and so need to reschedule them in my org agenda. To do this (and other operations) on multiple items, mark the items in your agenda view using m then hit B to bring up the bulk action list and then s to reschedule. This will set the new scheduled date to all marked items.

Use scroll margins to give context to your cursor

I add the following line to my emacs config file so the when I move the cursor near to the top or bottom of the window, the window will scroll when I am 3 lines away from edge. The effect is that (however it is moved) the cursor is always given 3 lines of context before the top or bottom of the window, which I find very useful.

(setq scroll-margin 3)

Dynamically filter directory listing with dired-narrow

If you install the package dired-narrow (one of several neat packages described on the dired-hacks github page), then you can type filter strings to dynamically filter down a dired listing to match the filter.

For example, in the animation below I have a dired buffer containing lots of files. I invoke dired-narrow by hitting / and then type “png” to narrow to filenames matching “png”. I then hit RET to complete the filtering. At that point I can operate on the filtered dired list as I would for any dired buffer. I can even narrow further by simply invoking dired-narrow again and typing a new filter (I use “mirror” in the example below). Once I have finished with the filter, I hit g to revert back to the full directory listing.

As an aside, I have started using use-package to install and configure packages (see the update on this page). Here is the code I use to install and set up dired-narrow.

;;narrow dired to match filter
(use-package dired-narrow
  :ensure t
  :bind (:map dired-mode-map
              ("/" . dired-narrow)))