27 November 2013

Raspberry Pi USB RAID1 root partition

UPDATE: I have since switched back to using the Pi foundation kernel, so I have updated the post accordingly.

I run a webserver on my Raspberry Pi and wanted to get the benefits of the high reliability of a RAID1 on the root (/) partition. I easily found how to hand-off boot to a USB device and also found how to setup a RAID1 as a non-root partition, but I wanted to combine the two. I found one other forum where this had been attempted but it didn't look like they ever completed it so I will share with you how I did this. This method will also work for other raid levels if you adjust the steps accordingly.

First, I'm going to assume you have a recent stock Raspbian install already running on an SD card on your pi and you have two identical USB thumb-drives or SD cards in adapters plugged in (/dev/sda and /dev/sdb). I used two of these plugged directly into the ports on the rpi.

The key to all of this is going to be switching to the Raspbian repository kernel rather than the kernel that comes on the Raspbian image from the Rpi foundation. This is because the Rpi foundation kernel doesn't have the raid1 module compiled in and it doesn't use an initramfs, so there is no way to get your RAID recognized out of the box. The Raspbian repository image uses an initramfs and makes this process incredibly easy. The latest Rpi foundation image supports initramfs and raid1.

Setup the kernel

Update the kernel image

# rpi-update
At this point reboot and make sure that you are booted from the latest kernel

Configure initramfs

Add the following lines to /etc/initramfs-tools/modules
raid1
md_mod
ext4

Update the initramfs

# update-initramfs -c -k `uname -r`
You may see some warnings about symlinks. You can safely ignore these since it is trying to create a symlink on a vfat filesystem which does not support them.

Edit boot configuration

Backup the original config.txt
# cp /boot/config.txt /boot/config.txt_orig 
Use the following command to figure out what kernel version you are running

# uname -r

It will be something in the form of 3.10.XX+. Substitute your version below.

Add the following 2 lines to the end of /boot/config.txt:
kernel=kernel.img
initramfs initrd.img-3.10.XX+ followkernel

Reboot and test

# reboot
# uname -r
You should still see the above kernel version.

Setup RAID1

Install mdadm
# apt-get install mdadm
# modprobe md
# modprobe raid1
Partition drives using fdisk and create a primary partition that uses the whole drive and has a partition system id of fd Linux raid auto

# fdisk /dev/sda

Repeated for /dev/sdb

Destroy any existing superblocks on the drives
# mdadm --zero-superblock /dev/sda1 /dev/sdb1

Create RAID
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
# cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
# mdadm --examine --scan >> /etc/mdadm/mdadm.conf
# dpkg-reconfigure mdadm

Respond with the following
Ok
all
Yes
Yes
root

Format RAID
# mkfs.ext4 /dev/md0

Mount raid to temporary location
# mount /dev/md0 /mnt/md0

Copy data from live system over to md0
# cp -afxv / /mnt/md0

Make /boot/cmdline.txt look like the following

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/md0 rootfstype=ext4 elevator=deadline boot_delay=32 rootdelay=10 rootwait

Backup fstab
# cp /etc/fstab /etc/fstab_orig

Comment the following lines in /etc/fstab

/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

Add the following line to /etc/fstab
/dev/md0 / ext4 defaults,noatime,errors=remount-ro 0 1

Reboot

Optionally delete the original partition on the SD card.

Now the system will boot from the SD card into initramfs, load the raid modules and assemble then mount the raid. Then it will boot into the system running from the RAID.

Great care must be taken if you decide you want to use rpi-update to upgrade your kernel. After doing so, you must use initramfs-update -c -k xxxxxx+ to create a new initramfs for that kernel. Then you have to go into /boot/config.txt and update the last line to point to the new initramfs.

14 comments:

  1. Hello,

    I tried to follow your procedure on a fresh raspbian image (7/4/2014) and couldn't get it working...
    The "apt-get install linux-image-rpi-rpfv" command has now a different release (3.10-3-rpi) and I changed it accordingly in /boot/config.txt.
    When booting, the system becomes unresponsive and the last entry in the dmesg is:
    [ 25.234201] md: raid1 personality registered for level 1

    Under a "normal" boot (switching back to the SD) the message next to this usually is:
    [ 25.454434] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)

    So I assume something is wrong with the RAID.

    The new kernel is loaded.
    Linux raspberrypi 3.10-3-rpi #1 Debian 3.10.11-1+rpi5 (2014-03-17) armv6l GNU/Linu

    What could it be wrong?
    Thank you,
    Joaoabs

    ReplyDelete
  2. Hello,

    I was able to connect the RPI to a screen and got more information.
    https://www.dropbox.com/s/g8to8qcdylxc4np/2014-04-08%2021.17.30.jpg

    It seems something is wrong with the md's, but before trying these unsuccessful boots from it, I tried several reboots (still using /dev/mmcblk0p2 and mounting /dev/md0 in /mnt/tmp) to make sure the md was there propperly built and monted.

    Any idea?
    Thanks,
    Joaoabs

    ReplyDelete
  3. I really need to update this post. I actually am using the pi foundation image now, not the one from the repository. The pi foundation image now has initramfs and raid1 support so you don't need the debian images.

    I had a similar issue to what you are describing where everything was setup right but it would just hang at boot like it couldn't see the md. I think I had to run the following:

    # dpkg-reconfigure mdadm

    I responded to the prompts with the following:
    all
    Yes
    Yes
    root

    ReplyDelete
  4. Hi,

    Just letting you know, this will only work on Wheezy. Not on Jessie. I believe it's a bug in how Jessie reads fstab.

    ReplyDelete
  5. Hi,

    I just updated my pi kernel from 3.18.11+ to 4.4.6+ and spent hours figuring out why the md* devices are not visible during booting, even I did nitramfs-update. The solution was to run dpkg-reconfigure mdadm, which updated the initr.d.img*. After that, everything ran fine. Therefore I propose to change the Sentence "After doing so, you must use initramfs-update -c -k xxxxxx+ to create a new initramfs..." to After doing so, you must use dpkg-reconfigure mdadm to create a new initramfs...".

    Regards,

    Zoltan

    ReplyDelete
  6. Thanks! I set up raid for my PI. Your post was very helpful.
    Rolandas

    ReplyDelete
  7. My RPI won`t boot.


    Possible error:
    # dpkg-reconfigure mdadm
    update-initramfs: deferring update (trigger activated)
    update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
    update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
    Processing triggers for initramfs-tools (0.120+deb8u1) ...
    ln: failed to create hard link `/boot/initrd.img-4.4.9-v7+.dpkg-bak' => `/boot/initrd.img-4.4.9-v7+': Operácia nie je povolená
    update-initramfs: Generating /boot/initrd.img-4.4.9-v7+

    Any ideas?

    ReplyDelete
  8. Hi,

    I updated my pi kernel again, and have to correct my previous comment. After updating to the new kernel (and before rebooting), first you have to call "update-initramfs -c -k xxxxxx+" to create a new initramfs, and then you must use "dpkg-reconfigure mdadm" to update it with your raid configuration. It seems to me, that "dpkg-reconfigure madadm" updates the newest initramfs it finds. As long as there is no initramfs for the newly installed kernel, it will only update the initramfs of the actual kernel (not the new one, which will be loaded only after reboot).

    Regards,

    Zoltan

    ReplyDelete
    Replies
    1. Zoltan, this did not work for me. My pis get bricked after EVERY kernel upgrade. Each one of them runs USB RAID1 root.

      The only solution to bring them up again is to boot off sdcard, update that as well, then chroot to your md, generate initramfs from there again and viola - it works. Not much sense into why.

      Delete
  9. Use kernel=kernel7.img parameter for pi 2/3 machines; using kernel=kernel.img results in an unbootable machine.

    ReplyDelete
  10. Well, I finally got my system to boot off the RAID1 setup. The key points I learned after constant reboots was:

    a) Ensure the filename of initrd.img doesn't have any typos or inconsistencies. In my case it was initrd.img-4.4.9-v7+ that "update-initramfs xxx" was looking for and also ensure this is spelled correctly in your config.txt file. There is some inconsistency in the instructions that state to use a period where a - should be used and utilizing 'uname -r' also did NOT work since it took that string as a literal. so be careful, as your array will NOT boot due to this simple issue alone to which your eyes can easily overlook. double check your /boot drive to find the files !!

    b) Watch out when using the dpkg-reconfigure mdadm because this will add ANOTHER identical entry to your mdadm.conf and that will lead to boot failure !! check the file and ensure there is only ONE entry. I had to manually delete the duplicate entry and then it booted..

    c) do not dare upgrade the kernel if you are physically away from the system as you could end up being locked out unable to restore until you are face to face with the machine.

    ReplyDelete
  11. This tutorial is excellent, but where there is Linux there are caveats. This one cost me several hours.

    I duplicated my server on another pi with RAID1 and got it working after an acceptable amount of time. Two WD 1 TB drives with 10 and 932GB partitions. 10 for root fs and the other for file shares.

    I moved the disks to the server, deleted and created new identical partitions on both drives:


    /dev/sda1 2048 20973567 20971520 10G fd Linux raid autodetect
    /dev/sda2 20973568 1953525167 1932551600 921.5G fd Linux raid autodetect

    /dev/sdb1 2048 20973567 20971520 10G fd Linux raid autodetect
    /dev/sdb2 20973568 1953458175 1932484608 921.5G fd Linux raid autodetect

    But after creating /dev/md0 (/dev/sda1 /dev/sdb1) and /dev/md1 (/dev/sdb1 /dev/sdb2) and performing # mdadm --examine --scan >> /etc/mdadm/mdadm.conf I ended up with three entries in mdadm.conf. Two for md0 and one for md1. One entry for md0 was bearing the hostname of my test machine and the other had the correct hostname like md1. I had already performed the update-initramfs -c -k `uname -r` and had done the mdadm --zero-superblock so I was quite puzzled as to why I was ending up with a RAID signature from the test machine. I ended up connecting both drives to a Win 10 machine and performing a DISKPART> clean command on both. Probably could have done a dd if=/dev/zero of=/dev/sd(a and b) bs=4M count=1000 or something but the DISKPART> was clean and fast.

    I returned the drives to the pi and created the md0 and md1 again and deleted all of the md entries in /etc/mdadm/mdadm.conf and repeated the # mdadm --examine --scan >> /etc/mdadm/mdadm.conf again. After that, the phantom entry in mdadm.conf was gone. I then deleted the new initrd image from /boot and recreated it again using same command used earlier.

    The best practice I learned from this "all nighter" was to make sure that your disks are completely zeroed before you even start.

    I also learned that adding echo > 0 /proc/sys/raid/speed_limit_max to /etc/rc.local is a good idea otherwise the RAID re-syncs will eat the performance of even your pi 3 alive.



    ReplyDelete
    Replies
    1. Just a follow-up re: /proc/sys/raid/speed_limit_max

      On my system echo > 0 /proc/sys/raid/speed_limit_max slowed the rebuild rate to 1148K/sec. setting it to 1 (default) yields 18197K/sec.

      I found that setting it to 0 was very useful for making the system much more responsive while resyncing the /dev/md0 10GB root filesystem; however, 10 days to resync /dev/md1 (932GB) is unacceptable so I set it back to 1.

      Turns out that the resync of the large non-root filesystem at max speed does not have a detrimental impact on system performance so I removed the hack in /etc/rc.local. Further, I found that echo > 0 /proc/sys/raid/speed_limit_max slows an active max resync immediately but setting it back to 1 does not take it back to full speed. I found that I have to reboot to do that. In my case, 12 hours to re-sync /dev/md1 beats the heck out of 10 days with minimal impact on my server.

      Maybe there will be a Raspberry pi with USB 3.0 and Gigabit Ethernet someday.

      Delete
  12. Hi Drew: I have to thank you for the missing link I was looking for. The md is now loading up early and my md0 raid 5 now is working with my file system loaded on it. I am now able to boot from my Cruser fit 16G usb's (30G available) and with out an sd card also. I have 4 usb's. sda1 is my boot. sda2,(was my original file system) sdb2, sdc2, sdd2 are now my 3 drive raid. It all boots great now. I run 30 sims on a docker swarm raid system now on OSGRID, using 3 raspberry pi's. Again thank you for this post..
    MovLab :))

    ReplyDelete