During large cPanel to cPanel migration using Transfer Tool, Maildir format is not converted to mdbox format

Synopsis

cPanel uses Dovecot as it’s IMAP storage. Dovecot can be configured for either Maildir or mdbox format.

During a large (1 TB) transfer from the one continent to another one, we observed the following behaviour during a botched transfer:

Server A: Reliable

Server B: Mostly reliable but crashed a few times during the transfer due to poor NFS connectivity and the NAS bombing. Bandwidth was also pretty dire.

Network link: Fine, but not super fast. Transfer took almost 5 days due to slow disks and a bad network on the destination side.

Symptoms

Client observed some emails disappearing before their eyes.

Those same clients observed that certain emails were lost, between set dates. Everything else was there and one could see in Track Delivery that all those emails were delivered.

Explanation and Fix

Somewhere along the line the Transfer Tool decides to convert Maildir to mdbox. Since the transfer was botched there were batches on the destination side where Maildir messages were still stuck in cur and new and all the other traditional Maildir places. Email works, it’s just messages are missing but found in the file system at these locations.

Once off fix per mailbox

Replacements in the script example below:

  • mailbox is username before the @ sign
  • cpanel_user = the cPanel user directory
  • example.com = the domain name

Check your work before running the script. Check by going into Roundcube at the missing dates, and make sure they are missing. Then when you run the script below, Roundcube should display the older messages first.

doveadm import -u mailbox@example.com maildir:/home/cpanel_user/mail/example.com/mailbox INBOX ALL

Batch script to fix for everyone

I called this recover-maildir.sh

#!/bin/bash

# Users to skip
skip_users=("user1" "user2" "user3" "user4")

for dir in /home/cpanel_user/mail/example.com/*; do
    user=$(basename "$dir")

    # Check if user is in the skip list
    if [[ " ${skip_users[*]} " == *" $user "* ]]; then
        echo "Skipping $user@example.com (already processed)"
        continue
    fi

    echo "Importing mail for $user@example.com..."
    doveadm import -u "$user@example.com" maildir:"$dir" INBOX ALL

Errors during batch script

Doing this en-mass for 1000s of users especially legacy ones will produce some or all of the following self explanatory errors:

User doesn’t exist, corrupted indexes, INBOX.spam or INBOX.whatever doesn’t exist

doveadm(user1@example.com): Error: User doesn’t exist

doveadm(user2@example.com): Warning: fscking index file /home/cpanel_user/mail/example.com/user2/storage/dovecot.map.index

doveadm(user2@example.com): Warning: mdbox /home/cpanel_user/mail/example.com/user2/storage: rebuilding indexes

doveadm(user3@example.com): Error: Couldn’t create mailbox INBOX.spam: Mailbox doesn’t exist: INBOX.N.G

There might be more serious indexing, inode, or user mailbox full errors too. Carefully save and then analyse the output.

Remaining tasks

The script doesn’t clean up the old Maildirs, and doing so it out of the scope of this article. However, as a responsible network administrator you may not leave those old files around. If you can’t script the deletions  yourself, maybe ask cPanel via their ticketing system for help, get a specialist, or use AI. Make sure you have backups first.

Good luck!

Share this article

Leave a Reply

Your email address will not be published. Required fields are marked *