Configuring MDADM Raid10 for Ubuntu 9.10

So you want to setup software raid 10 for an Ubuntu 9.10 system?

I'm going to assume you already know what raid is, even what raid10 is. But setting up an OS to use it is mildly complicated.

Planning

First, I'm going to recommend you have 4 or 5 drives. You could do 6 or 7... but thats a space issue more than a redundancy one. For this tutorial I'm going to assume 4 or 5. Technically raid10 will work with 2 drives, but then its really just a raid1, and with 3 it will work, but I wouldn't consider this ideal. Stick with an even number of mirrors and slaves, and use odd numbers for spares.

The actual array will consist of 4 drives, two stripes, two mirrors.  If you use a 5th drive, it will be configured as a hot-spare.

Decide if you want a near or far layout. Near layouts aren't quite as fast for reading as a far layout, however, in a degraded state will be faster than a far layout in a degraded state. Since choosing a far layout requires some serious tweaking, and because its performance will suck if you lose a drive [which could make your applications crash if you're doing this for a loaded server] I'm going to recommend a near layout, which is the default.

You will need the Ubuntu 9.10 Alternate installer cd for your architecture. Let me say that again ALTERNATE -- which is a textual installer. If you try to use the standard gui installer cd, you wont be able to create the raid arrays from within the installer.

You also need to know that /boot cannot be safely installed on raid10. It is therefore recommended that you install boot to a raid1 device. Don't worry, you just need to put boot on a separate md device, and i'll show you how to do that below.

Next, swap. If you've got insufficient memory to complete the ubuntu install without swap, you'll need to make swap partitions. I have enough ram in my system to install without swap so I'm not going to use any swap partitions, but instead create a swapfile after the install is completed. For redundancy I'm going to stick this on the same raid1 array that I put /boot on. Some people put swap partitions on all the drives and tell the OS to use those -- however, if your system is swapping and a drive is lost, you will potentially lose your swap data, and crash your system. Putting the swap file on raid1 will probably impact swap performance, but at least you know if a drive dies your system will keep running along happily.

 

 

The Install

In your bios, make sure you have your hard drives set as boot devices... my bios says something like usb, cdrom, hard drive... the latter being a generic for 'any hard drive'... if your bios specifies devices individually, you need to make sure that all your drives are in the boot list in case you lose a drive and need to reboot. This install will add grub to all the drives, and we're going to set them as bootable so that if the main system drive fails, it will just boot from another -- at least until it can get to the raid1 /boot level. For clarity, we're talking about the MBR records.

Next, boot of the alternate install cd. Answer all the normal questions until you get to the partitioning screen. Choose manual as your partitioning method. (Not guided)

Refer to the screenshots here [I couldn't take screenshots as I was installing, so I cant share my exact settings]. https://help.ubuntu.com/community/Installation/SoftwareRAID

1) Delete all your partitions [this will delete all your data so if they're not clean harddrives, stop, get out and back up, cuz you will lose all data]

2) Create two partitions per disk. Make the first partition equal to 2 x <RAM IN GB> + 5GB. (for me this was about 2x16+5 = 37GB) This will hold your raid1 volume. Make the second partition equal to the free space on the drive minus 5GB rounded to a nice number. Note that G and GB are distinct measurement sizes on linux, use either, but be CONSISTENT! Why minus 5GB? Well, in my experience no two drives are exactly the same size. You want your raid partitions to be as close to the same size as you can get em, and since we wont be using a LVM, you dont want to create a partition that is too big for a replacement disk that might not have exactly the same specs. This isn't the place to push the envelope, so leaving yourself a nice margin to work with is key.

3) Set the volume labels to the same for each corresponding partitions. First I called RAID1BOOTSWAP and the second I called RAID10SYSTEM.

4) Set the bootable attribute on the first partitions.

5) Set the mount points to do not mount.

6) Once you have the drives partitioned, go to CONFIGURE SOFTWARE RAID and follow the dialogs until you get a choice to CREATE NEW MD DEVICE. Pick that, and chose raid1. Provide the number of devices (4) and the number of spare (0 or 1 if you have a 5th drive). Select all the first partitions. sda1 sdb1 sdc1 sdd1 for example.

7) Repeat step 6 choosing raid10 and the sda2 sdb2 sdc2 sdd2 partitions.

8) Go back to the partition summary screen (where you saw the option to configure software raid) ... pick your MD0 device.

9) Configure MD0 to use a filesystem (i suggest ext3 or ext4).  Set MD0 to mount /boot. Give the drive a label. I used RAID1BOOTSWAPARRAY.

10) Go back to the partition summary screen and pick MD1 device.

11) Configure MD1 to use a filesystem (i suggest ext3 or ext4). Set MD1 to mount the root filesystem /. Give the drive a label. I used RAID10SYSTEMARRAY.

12) Finish/save/write/whatever and continue with the install as normal. Ignore the warning about no swap partitions. (Unless you dont have enough ram, in which case create swap in those extra 5gb spaces, you can delete those partitions later). You will probably be asked if you want to boot a degraded array or not. I want my system to reboot and run on 3 disks. So I answered yes. This is dangerous though as if the system doesn't run its unlikely a second disk will fail, and you wont lose the array -- however if the system runs it could lose another drive and, if its the wrong one, your array could be lost. A four-disk raid10 array can survive one drive crash, and potentially a second depending on whether its a mirror or a stripe. Kinda like 1 for sure, the second is a 50:50 chance.

13) Reboot into the system when the install finishes as normal.

Monitoring

Ok, so you've got the system rebooted and everything is making a lot of noise as the drive syncs. To watch the status of your arrays type

watch -n 1 cat /proc/mdstat

When its done syncing the arrays you should see a display very similar to:

Personalities : [raid10] [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4]
md1 : active raid10 sdc2[2] sdd2[3] sdb2[1] sda2[0]
      566403456 blocks 64K chunks 2 near-copies [4/4] [UUUU]
     
md0 : active raid1 sdc1[2] sdd1[3] sdb1[1] sda1[0]
      24410624 blocks [4/4] [UUUU]
     
unused devices: <none>

The UUUU items tell you that the disks are happy. If they're failed they'll look like UU_U where the 3rd disk is failed. The active line will also show a (F) beside the drive name... eg sda1[0](F) ...indicating the drive that has failed.

Since you wont have a MTA up yet, we'll skip configuration of the monitoring daemon for now.

It will take a while for your drives to sync. For me it took about 2 hours. Go out for dinner now. The progress report will tell you how its doing. Do not close up your case because you will be testing a drive failure after the sync is done.

Setting up swap

Ok, so the drives are now synced, and everything appears to work. Lets setup a swap file.

Create an 8gb blank file. You can use 2xRAM as a good value -- though with larger amounts of ram this is probably excessive. I picked 8GB, but left the partitioning with enough space that I could create a larger swap file if desired.

dd if=/dev/zero of=/boot/8192Mb.swap bs=1M count=8192

mkswap /boot/8192Mb.swap

swapon /boot/8192Mb.swap

pico -w /etc/fstab

Add

/boot/8192Mb.swap none swap sw 0 0

reboot.

You should now be able to see available swap space in top / free and /proc/meminfo areas.

Testing your raid array (DO THIS!!!)

To test your raid array, pick a disk at random and unplug it. Some people don't recommend doing this as it could theoretically cause damage to a drive, but most drives are hotswappable and ive never had a problem. YMMV. If you're watching the mdstat, you'll notice one of the drives go into UU_U or _UUU or U_UU etc... so you're now running on three drives.

Reboot. Make sure it comes back with 3 drives and without manual intervention.

At this point you should tell the raid that the disconnected drive is to be considered faulty and remove it from the array.

mdadm --manage /dev/md0 --set-faulty detached

mdadm --manage /dev/md1 --set-faulty detached

mdadm --manage /dev/md0 -r detached

mdadm --manage /dev/md1 -r detached

poweroff and reconnect the drive. boot.

I wanted to test a real failure with a blank disk, so i nuked the partition table on the drive using fdisk and recreated the partitions from scratch.

I then did a mkfs.ext4 /dev/sdc1  ... and again for sdc2 ... im not sure it was necessary, but thats what I did and it worked.

Ok, so you've got a degraded array and the disk ready to go back into service.

mdadm --manage /dev/md0 -a /dev/sdc1

Wait for this to rejoin the array and syncronize (watch with mdstat) about 5 mins for me.

Then add it back to the raid10 array

mdadm --manage /dev/md1 -a /dev/sdc2

The array will recover and this will take some time. About 45 mins for me.

Verify your arrays are happy with

mdadm --detail /dev/md0

mdadm --detail /dev/md1

You're looking for State:clean and persistent superblocks with 4 active devices.

From here, continue to setup your system and install a MTA that works for mail from the command line.... test it with

mail user@example.org

enter the subject

enter a message

control-D

... replacing user@example.org with your email. Make sure you get the email

Configure smartd

apt-get install smartmontools

pico -w /etc/smartd.conf

Change the devicescan line (the one thats not commented) to

DEVICESCAN -m root -d sat -m user@example.org -M exec /usr/share/smartmontools/smartd-runner -M test

save.

pico -w /etc/default/smartmontools and uncomment start_smartd=yes

save

/etc/init.d/smartmontools start

smartd will start and will send you 4 emails about each of your drives having an issue (this is the test switch) once you're satisfied that you got these emails, remove the -M test from the end of the devicescan line in smartd.conf. If you dont get the emails debug, starting with your MTA.

Configure MDADM preferences to monitor your array

dpkg-reconfigure mdadm

Follow along, and set your email in there when prompted. I got a warning about runlevel arguments, but it seems to be working.

Conclusion

That is all there is to setting up an Ubuntu 9.10 system to use a raid 1 boot array and a raid 10 system array.