Frozen Industries

Create a single, aligned partition on a drive larger than 2TB.

% sudo parted /dev/sdx
(parted)% mklabel gpt
(parted)% mkpart primary 1 -1

Tests are done across four 7200RPM SATAII drives on a PCI-X card sitting on a PCI (32-bit, 133MB/sec theoretical max) bus, probably the slowest bus configuration possible, and then again after being moved to a motherboard with dual PCI-X slots. Server is running Ubuntu 9.10 AMD64 Server.

Benchmark is a simple ‘dd’ sequential read and write.

write: dd if=/dev/zero of=/dev/md2 bs=1M
read: dd if=/dev/md2 of=/dev/null bs=1M

mdadm –create /dev/md2 –verbose –level=10 –layout=n2 –raid-devices=4 /dev/sd[ftlm]1

mdadm –create /dev/md2 –verbose –level=10 –layout=f2 –raid-devices=4 /dev/sd[ftlm]1

mdadm –create /dev/md2 –verbose –level=10 –layout=o2 –raid-devices=4 /dev/sd[ftlm]1

And more comparisons:

RAID1 (PCI)

write: 38.9 MB/s
read: 64.8 MB/s

Single Disk (PCI)

write: 59.4 MB/s
read: 71.9 MB/s

Have an LVM device left on your system from a drive that was removed before pvremove was run?

1

$ sudo dmsetup remove /dev/mapper/removed-device

LVM allows you to hot add devices to expand volume space. It also allows you to hot remove devices, as long as there are enough free extents in the volume group (vgdisplay) to move data around. Here I’m going to replace a 400 GB drive (sdg) with a 750 GB one (sdf) from logical volume “backup” on volume group “disks”. It does not matter how many hard drives are in the volume group, and the filesystem can stay mounted.

1. Partition and create a physical volume on the device

   1	$ sudo pvcreate /dev/sdh1

2. Add the new drive to the volume group

   1	$ sudo vgextend disks /dev/sdh1

3. Move all extents from the old drive to the new one (this step may take hours)

   1	$  sudo pvmove -v /dev/sdg1

4. Remove the old drive

   1	$ sudo vgreduce disks /dev/sdg1

5. Expand the logical volume to use the rest of the disk. In this case, another 350GB.

   1	$ sudo lvextend -l+83463 /dev/disks/backup

6. Expand the file system

   1	$ sudo resize2fs /dev/disks/backup

Scan a system for RAID arrays and save findings so the array reappears across reboots:

1

# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Create a RAID5 array out of sdm1, sdj1, and a missing disk (all partitioned with raid-autodetect partitions)

1

# mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sd[mj]1 missing

Create a RAID1 array

1

# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd[ts]1

Remove a RAID array

1
2

# mdadm --stop /dev/md1
# mdadm --zero-superblock /dev/sd[ts]1

Replace a failed drive that has been removed from the system

1

# mdadm /dev/md3  --add /dev/sdc1 --remove detached

/dev/md0:
Timing buffered disk reads: 1248 MB in 3.00 seconds = 415.65 MB/sec

MDADM 2.x on kernels >2.6.17 supports online resizing of RAID 5 arrays

# cat /proc/mdstat

md0 : active raid5 sdf1[7] sdb1[0] sde1[5] sdg1[4] sdh1[3] sdd1[2] sdc1[1]
1465175424 blocks level 5, 64k chunk, algorithm 2 [7/6] [UUUUUU_]
[>....................] recovery = 1.3% (3331200/244195904) finish=2357.0min speed=1700K/sec

Ouch. Two files are used to control the speed of rebuilding RAID arrays in Linux.

/proc/sys/dev/raid/speed_limit_min
/proc/sys/dev/raid/speed_limit_max

Even though my _max file is set to 200,000K/sec and my system is not doing anything, my RAID 5 rebuild process is hovering around the _min rebuild speed, of 1,000K/sec. With my setup this will take approximately 40 hours to complete, which is too long for me to wait. So, I pushed the _min speed up to 10,000K/sec, which will now take 6 hours to finish, and use slightly more of my system’s idle resources.

root# echo “10000″ > /proc/sys/dev/raid/speed_limit_min

Later I set _min to 50,000K/sec, and the rebuild speed topped out at 25,000K/sec.

#cat /proc/mdstat

Personalities : [raid5]
md0 : active raid5 sdf1[7] sdb1[0] sde1[5] sdg1[4] sdh1[3] sdd1[2] sdc1[1]
1465175424 blocks level 5, 64k chunk, algorithm 2 [7/6] [UUUUUU_]
[=>...................] recovery = 5.1% (12661840/244195904) finish=149.9min speed=25726K/sec

The rebuild took less than 3 hours, down from the original 40.

I recently upgrade my main workstation’s hard drive under Ubuntu 6.10 and noticed a couple things changed during the process. Here are my instructions for a fast and reliably hard drive swap. I’m using SATA drives without LVM using the default Ubuntu install and partition options for this howto.

1. Prepare
1. Shutdown your machine and install your new hard drive. Don’t mess with your current hard drive (yet)
2. Find a LiveCD, I used the Ubuntu 6.10 LiveCD, it matched my OS, but it doesn’t have to. Knoppix should work fine.
3. Boot using your LiveCD
4. After booting, open a terminal and “sudo su” to become root
2. Setup your new drive
1. Use cfdisk /dev/sda to look at your partitions on your current drive. I have sda1 of type linux as most of my current drive, and a 6 GB sda5 as linux swap at the end
2. Duplicate this on your new drive using cfdisk /dev/sdb, adjusting for space as necessary. I created a new primary partition using cfdisk using all but 6 GB of space, then a new logical partition using the rest of the space. You must create the partitions in this order to get the right numbering
3. make the primary partition to bootable
4. set the swap partition as type 82 (linux swap)
5. save and quit
6. create filesystems on the new partitions using mkfs.ext3 /dev/sdb1 and mkswap /dev/sdb5
3. Copy data
1. Make directories to mount your old and new partitions, in this case, /mnt/sda1 and /mnt/sdb1
2. Mount your drives to these partitions using mount /dev/sda1 /mnt/sda1 and mount /dev/sdb1 /mnt/sdb1
3. Copy all your data from your old drive to your new drive using cp -a /mnt/sda1/* /mnt/sdb1/ . The -a will preserve owners, permissions, date, etc.
4. Get up and do something else. It took 70 minutes for my machine to copy about 150 GB of data from one drive to the other
4. Fix the boot options
1. This is where Ubuntu 6.10 differs from previous versions. Fstab and menu.lst both use UUID numbers to find partitions. To get the UUID number of your new partitions, run vol_id /dev/sdb1 and vol_id /dev/sdb5 . Copy these numbers into their appropriate places in your /mnt/sdb1/etc/fstab and /mnt/sdb1/boot/grub/menu.lst files. You may need to dig around the menu.lst to find all the entries.
2. Now install grub onto the MBR of the new drive to make it bootable. To do this I first chroot into my new system using chroot /mnt/sdb1 /bin/bash . Now that you’re in the new system, run grub. Inside grub, run setup (hd1,0) then root (hd1) . This will differ if you have a different drive setup. Quit grub (quit).
5. Finish up
1. Logout of your chroot (logout), unmount your mounted drives umount /dev/sda1 and umount /dev/sdb1, and shutdown your computer. Disconnect your old drive, plug your new drive into the old drive’s cable, and start your computer back up. If everything went well, it will boot back up as if nothing happened.

My server decided that an executable file didn’t really exist on the file system, or so I thought. Lack of sleep was the main problem, but here are some things I did to check my file system for errors. I setup this file system on a Ubuntu 6.06 AMD64 install with LVM, so everything is in LVM instead of standard partitions.

# sudo e2fsck -n /dev/mapper/Ubuntu-root

This was showing errors, but I ran it while the system was mounted and running, so there were open files, so this was normal. The -n kept e2fsck from attempting to fix anything, which was good because later I ran the command after booting from an Ubuntu LiveCD and found no errors.

Before booting from the LiveCD I tried to get the system to fix itself by running fsck on boot. Two methods I used to do this on Ubuntu were running these from the live system before rebooting, they both accomplish the same thing, so only one was really needed.

# sudo touch /forcefsck

# sudo tune2fs -C 40 /dev/mapper/Ubuntu-root

These appeared to have no affect, probably because the filesystem was fine, but I took down the system and ran fsck from a LiveCD instead. Of course, this wasn’t as simple as it should have been, the LiveCD did not detect my LVM volumes, so /dev/mapper/Ubuntu-root was missing. The fix was to install LVM2 and start it up.

# sudo apt-get install lvm2
# sudo /etc/init.d/lvm start

The /dev/mapper/ entries then appeared and I could run all the fscks I wanted. At this point my fsck checks were coming out clean, so file system corruption was not to blame.