Disclaimer: Use this info at your own risk, don’t come to me if all your data disappears one day. But it works for me.
Background
Want to use 4 different sized drives together as one large storage tank with 1 drive fault tolerance?
Do you also want to be able to upgrade the smallest capacity drive with a larger one without having to move all of your data to a temporary location then re-build a new array and then copying all your data back?
I’ve figured out how you can do it.
I’ve figured out how to make FreeNAS using ZFS (nightly build) to use an array of 4 drives (theoretically start with one drive but I’m not going to talk about that here) and allow the array to grow by replacing the smallest drive in the array with a larger one similarly to how a Drobo works. I’m going to describe the process then I’ll do a real world example.
Summary
This is the basic idea… I will be creating five raid5 sets inside one ZFS pool. I will be creating four partitions on each of the four drives and using these partitions as the storage containers for the raid arrays. This way I can control the sizes of each partition and get the best use out of the drive space within a raid5 array. ZFS allows you to expand the size of a raid array and the method I’m describing let’s you take advantage of that cool feature.
What I will be doing is slicing up the four hard drives into as many equal sized parts as I can. From there I’ll build five Raid5 arrays (one for each line of partitions in the phase 1 illustration) and create a zpool called tank0 containing all five raid5 arrays.
Details
As you can see below I will start with four drives:
Drive 0 – 80 Gig
Drive 1 – 120 Gig
Drive 2 – 200 Gig
Drive 3 – 200 Gig
The zpool will be five raid5 arrays and will give me about 329 Gigs of protected storage (54% use of the total space of the drives). Then I’ll take the 80 Gig drive out of the picture and put in a 250 Gig and I’ll end up with the following drives:
Drive 0 – 120 Gig
Drive 1 – 200 Gig
Drive 2 – 200 Gig
Drive 3 – 250 Gig
My zpool will increase in size to about 531 Gigs of protected storage (68% use of the total space of the drives). This will be done without doing any backing up, just swapping of drives. It is a four step process and each drive must be installed wiped repartitioned and resilvered (replaced) back into the zpool. I actually moved all the drives around one at a time. But I’m not sure if I had to bother. It might be possible and easier by just swapping out the smallest drive with the largest one and repartion all the drives in place one at a time… I’ll test that in the future. As for right now I’ve completed it the long way and it works!
Configurations
See how the configurations change below from phase 1 to phase 2 to get an idea of what I’m about to do.
Phase 1
Drive 0 | Drive 1 | Drive 2 | Drive 3 | Partition Size | ||||
ad0 | ad1 | ad2 | ad3 | |||||
Drive Size | 80 Gig | 120 Gig | 200 Gig | 200 Gig | ||||
Actual usable | 81918 | 123482 | 203884 | 203884 | ||||
ad0s1 | ad1s1 | ad2s1 | ad3s1 | Raid5a | 81726 | x 3 | 245178 | |
ad0s2 | ad1s2 | ad2s2 | Raid5b | 64 | x 2 | 128 | ||
ad1s3 | ad2s3 | ad3s2 | Raid5c | 41628 | x 2 | 83256 | ||
ad0s3 | ad1s4 | ad3s3 | Raid5d | 64 | x 2 | 128 | ||
ad0s4 | ad2s4 | ad3s4 | Raid5e | 64 | x 2 | 128 | ||
Unused Space | 0 | 0 | 80402 | 80402 |
Total Space (sum of all drives) | 613168 Megabytes = 613 Gigabytes |
Total available within Zpool | 328818 Megabytes = 329 Gigabytes |
Percent of usable storage | 54% |
Phase 2
Drive 0 | Drive 1 | Drive 2 | Drive 3 | Partition Size | ||||
ad0 | ad1 | ad2 | ad3 | |||||
Drive Size | 120 Gig | 200 Gig | 200 Gig | 250 Gig | ||||
Actual usable | 123482 | 203884 | 203884 | 250018 | ||||
ad0s1 | ad1s1 | ad2s1 | ad3s1 | Raid5a | 123290 | x 3 | 369870 | |
ad0s2 | ad1s2 | ad2s2 | Raid5b | 64 | x 2 | 128 | ||
ad1s3 | ad2s3 | ad3s2 | Raid5c | 80466 | x 2 | 160932 | ||
ad0s3 | ad1s4 | ad3s3 | Raid5d | 64 | x 2 | 128 | ||
ad0s4 | ad2s4 | ad3s4 | Raid5e | 64 | x 2 | 128 | ||
Unused Space | 0 | 0 | 0 | 46134 |
Total Space (sum of all drives) | 781268 Megabytes = 781 Gigabytes |
Total available within Zpool | 531186 Megabytes = 531 Gigabytes |
Percent of usable storage | 68% |
Procedure
The procedure using a FreeNAS nightly build (newest experimental version):
Step 1
First thing you need to do is figure out the best use of your four drives space. Use the chart in Phase 1. I suggest using a spreadsheet to work out the best sizes for the partitions. Remember if you are doing this yourself, make sure you keep the partition sizes the same size or larger. You can’t shrink them, ZFS only allows you to replace parts of a raid array with one the same size or larger.
Step 2
Using FreeNAS’s GUI Mount the four disks and enable SMART monitoring Disks|Management| +
Preformatted file system set as ZFS storage pool device.
This allows FreeNAS to monitor the drive using S.M.A.R.T. and can be setup to email when a drive is showing symptoms of problems.
Step 3
Now I create raid5a1 on drive0 (ad0) using fdisk-linux (which I removed the original FreeNAS fdisk and renamed the fdisk-linux to fdisk on my FreeNAS server). Get the fdisk-linux package [here]
Step 4
Delete or rename the original /sbin/fdisk and copy the fdisk-linux to your /sbin folder. I renamed the fdisk-linux to fdisk and that is what I used for the rest of this document.
Step 5
From the command prompt:
Partition Drives
Partition Drive 0 (ad0) (delete all partitions on the drives first)
Beginning the Partitioning
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
freenas:/sbin# fdisk /dev/ad0 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 9964. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-9964, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-9964, default 9964): +81726M Command (m for help): p Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad0s1 1 9937 79818921 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-9964, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-9964, default 9964): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-9964, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-9964, default 9964): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (9956-9964, default 9956): Using default value 9956 Last cylinder or +size or +sizeM or +sizeK (9956-9964, default 9964): Using default value 9964 Command (m for help): p Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad0s1 1 9937 79818921 83 Linux /dev/ad0s2 9938 9946 72292+ 83 Linux /dev/ad0s3 9947 9955 72292+ 83 Linux /dev/ad0s4 9956 9964 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad0: 81.9 GB, 81964302336 bytes 255 heads, 63 sectors/track, 9964 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad0s1 1 9937 79818921 a5 FreeBSD /dev/ad0s2 9938 9946 72292+ a5 FreeBSD /dev/ad0s3 9947 9955 72292+ a5 FreeBSD /dev/ad0s4 9956 9964 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Partition Drive 1 (ad1)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
freenas:/sbin# fdisk /dev/ad1 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 15017. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-15017, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +81726M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-15017, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-15017, default 15017): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-15017, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-15017, default 15017): +41628M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (15009-15017, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017): Using default value 15017 Command (m for help): p Disk /dev/ad1: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad1s1 1 9937 79818921 83 Linux /dev/ad1s2 9938 9946 72292+ 83 Linux /dev/ad1s3 9947 15008 40660515 83 Linux /dev/ad1s4 15009 15017 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad1: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad1s1 1 9937 79818921 a5 FreeBSD /dev/ad1s2 9938 9946 72292+ a5 FreeBSD /dev/ad1s3 9947 15008 40660515 a5 FreeBSD /dev/ad1s4 15009 15017 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now we Partition Drive 2 (ad2)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 |
freenas:/sbin# fdisk /dev/ad2 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 9937 79818921 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-24792, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +64M Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 9937 79818921 83 Linux /dev/ad2s2 9938 9946 72292+ 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (9947-24792, default 9947): Using default value 9947 Last cylinder or +size or +sizeM or +sizeK (9947-24792, default 24792): +41628M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (15009-24792, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 9937 79818921 83 Linux /dev/ad2s2 9938 9946 72292+ 83 Linux /dev/ad2s3 9947 15008 40660515 83 Linux /dev/ad2s4 15009 15017 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 9937 79818921 a5 FreeBSD /dev/ad2s2 9938 9946 72292+ a5 FreeBSD /dev/ad2s3 9947 15008 40660515 a5 FreeBSD /dev/ad2s4 15009 15017 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now we partition Drive3 (ad3)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
freenas:/sbin# fdisk /dev/ad3 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +81726M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (9938-24792, default 9938): Using default value 9938 Last cylinder or +size or +sizeM or +sizeK (9938-24792, default 24792): +41628M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (15009-24792, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-24792, default 24792): +64M Command (m for help): p Disk /dev/ad3: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad3s1 1 9937 79818921 83 Linux /dev/ad3s2 9938 14999 40660515 83 Linux /dev/ad3s3 15000 15008 72292+ 83 Linux /dev/ad3s4 15009 15017 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad3: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad3s1 1 9937 79818921 a5 FreeBSD /dev/ad3s2 9938 14999 40660515 a5 FreeBSD /dev/ad3s3 15000 15008 72292+ a5 FreeBSD /dev/ad3s4 15009 15017 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now let’s create our zpool containing our 5 (Raid5a to Raid5e) raid5 arrays (of the partitions)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
freenas:/sbin# zpool create -m /mnt/tank0 tank0 raidz1 /dev/ad0s1 /dev/ad1s1 /dev/ad2s1 /dev/ad3s1 freenas:/sbin# df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 494M 87M 367M 19% / devfs 1.0K 1.0K 0B 100% /dev tank0 224G 128K 224G 0% /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 errors: No known data errors |
Now we add the 2nd raid array to the same pool (tank0).
1 |
We must force since we are adding a 3 device raid to a pool that already has a 4 device raid. |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s2 /dev/ad1s2 /dev/ad2s2 freenas:/sbin# df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 494M 87M 367M 19% / devfs 1.0K 1.0K 0B 100% /dev tank0 224G 0B 224G 0% /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 errors: No known data errors |
Now we add the 3rd raid array:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad1s3 /dev/ad2s3 /dev/ad3s2 freenas:/sbin# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 0B 322G 0% /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 errors: No known data errors |
Now we add the 4th raid array:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s3 /dev/ad1s4 /dev/ad3s3 freenas:/sbin# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 0B 322G 0% /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 errors: No known data errors |
Now we add the 5th and final raid to our pool (tank0)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
freenas:/sbin# zpool add -f tank0 raidz1 /dev/ad0s4 /dev/ad2s4 /dev/ad3s4 freenas:/sbin# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 0B 322G 0% /mnt/tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
1 2 3 4 5 6 |
freenas:/sbin# ls -la /mnt total 6 drwxr-xr-x 3 root wheel 512 Apr 19 00:21 . drwxr-xr-x 19 root wheel 512 Apr 18 01:37 .. drwxr-xr-x 2 root wheel 2 Apr 19 00:21 tank0 freenas:/sbin# chmod 777 /mnt/tank0/ |
Now add the tank0 to samba and test the speed.
Testing
Over my 100Mb/s Ethernet connection.
Writing a 1,411,699,808 byte file to tank0 over the network took 152 seconds. Which is 74 Mb/s not too bad considering all the writing to the different raid5 arrays within the pool. ZFS must do a lot in memory… Ram usage was around 25% of my 1 Gig FreeNAS box and CPU varied from 25% to 50% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz
Reading a 1,411,699,808 byte file from tank0 over the network took 160 seconds. Which is 70 Mb/s not too bad either. Ram usage was around 14% of my 1 Gig FreeNAS box and CPU was consistently at 25% from my AMD Sempron(tm) Processor 3000+, running at 1800 Mhz
It is surprising that reading from the zpool tank0 was a little slower then writing… Either way these speeds aren’t too shabby and will do well for streaming HD quality movies or doing archival backups of your data.
Swapping Drives On The Fly
Now on to swapping out the 80 Gig drive for a 250 Gig without backing up to another drive and then rebuilding the array and copying it all back.
First scrub the zpool to make sure it’s all good.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
freenas:/sbin# zpool scrub tank0 freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: scrub in progress, 12.48% done, 0h1m to go config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
After a little while the scrub completes. This will take longer with more data in the pool.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
freenas:/sbin# zpool status pool: tank0 state: ONLINE scrub: scrub completed with 0 errors on Sun Apr 19 01:07:58 2009 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Let’s export the zpool.
1 |
freenas:/sbin# zpool export -f tank0 |
Shutdown the computer and swap out the 200 Gig drive3 with the new larger 250 Gig drive 3. We work our way down to drive0 in steps, swapping larger drives along the way.
With the new drive installed we have to import the pool to see it.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
freenas:~# zpool import pool: tank0 id: 1762523649679414016 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-3C config: tank0 UNAVAIL insufficient replicas raidz1 DEGRADED ad1s1 ONLINE ad2s1 ONLINE ad2s1 UNAVAIL cannot open ad3s1 ONLINE raidz1 UNAVAIL corrupted data ad2s2 ONLINE ad2s2 UNAVAIL cannot open ad3s2 ONLINE raidz2 DEGRADED ad2s3 UNAVAIL cannot open ad3s3 ONLINE ad2s4 UNAVAIL cannot open ad3s4 ONLINE pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank0 DEGRADED raidz1 DEGRADED ad0s1 ONLINE ad1s1 ONLINE ad2s1 ONLINE ad3s1 UNAVAIL cannot open raidz1 ONLINE ad0s2 ONLINE ad1s2 ONLINE ad2s2 ONLINE raidz1 DEGRADED ad1s3 ONLINE ad2s3 ONLINE ad3s2 UNAVAIL cannot open raidz1 DEGRADED ad0s3 ONLINE ad1s4 ONLINE ad3s3 UNAVAIL cannot open raidz1 DEGRADED ad0s4 ONLINE ad2s4 ONLINE ad3s4 UNAVAIL cannot open |
During testing I already used the name tank0 and built a similar raid array. ZFS is trying to get all the info it can. Since they both have the same name I must import the real one using it’s ID.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
freenas:~# zpool import 325956304794594593 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 01:25:45 2009 config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 UNAVAIL 0 0 0 cannot open raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 UNAVAIL 0 0 0 cannot open raidz1 DEGRADED 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 UNAVAIL 0 0 0 cannot open raidz1 DEGRADED 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 UNAVAIL 0 0 0 cannot open errors: No known data errors |
So far so good, now we partition Drive3 (the new 250 Gig drive) ad3 and replace the partitions into the raid arrays of our zpool.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
freenas:~# fdisk /dev/ad3 The number of cylinders for this disk is set to 30401. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-30401, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-30401, default 30401): +123290M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-30401, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-30401, default 30401): +80466M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (24775-30401, default 24775): Using default value 24775 Last cylinder or +size or +sizeM or +sizeK (24775-30401, default 30401): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (24784-30401, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-30401, default 30401): +64M Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad3: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad3s1 1 14990 120407143+ a5 FreeBSD /dev/ad3s2 14991 24774 78589980 a5 FreeBSD /dev/ad3s3 24775 24783 72292+ a5 FreeBSD /dev/ad3s4 24784 24792 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now we replace the partitions of old drive3 (200 Gig) with the new Drive3 (250 Gig).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
freenas:~# zpool replace tank0 ad3s1 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver in progress, 29.02% done, 0h0m to go config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad3s1/old UNAVAIL 0 0 0 cannot open ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 UNAVAIL 0 0 0 cannot open raidz1 DEGRADED 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 UNAVAIL 0 0 0 cannot open raidz1 DEGRADED 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 UNAVAIL 0 0 0 cannot open errors: No known data errors |
After it completes we replace the other 3 partitions:
1 2 3 |
freenas:~# zpool replace tank0 ad3s2 freenas:~# zpool replace tank0 ad3s3 freenas:~# zpool replace tank0 ad3s4 |
When they are complete we have a nice clean pool again:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 01:46:31 2009 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Testing
Copy some more files on tank0 for testing… No problem. I’ve got a 500 Meg .ZIP file on tank0 and I’ll do a test on it to make sure the data is still perfect, just to be 100% sure.
Test found no errors. As expected.
Let’s scrub again then when it completes without error, we export and shutdown the computer. Then we swap out drive 2 (200 Gig) with the old drive3 200 Gig drive, probably not needed since they are the same size, but I’m going to do it for completeness.
1 2 |
freenas:~# zpool scrub tank0 freenas:~# zpool export –f tank0 |
Shutdown and swap out drives reboot with new drive2 (200 Gig):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank0 DEGRADED raidz1 DEGRADED ad0s1 ONLINE ad1s1 ONLINE ad2s1 UNAVAIL cannot open ad3s1 ONLINE raidz1 DEGRADED ad0s2 ONLINE ad1s2 ONLINE ad2s2 UNAVAIL cannot open raidz1 DEGRADED ad1s3 ONLINE ad2s3 UNAVAIL cannot open ad3s2 ONLINE raidz1 ONLINE ad0s3 ONLINE ad1s4 ONLINE ad3s3 ONLINE raidz1 DEGRADED ad0s4 ONLINE ad2s4 UNAVAIL cannot open ad3s4 ONLINE freenas:~# zpool import tank0 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 02:05:57 2009 config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 UNAVAIL 0 0 0 cannot open ad3s1 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 UNAVAIL 0 0 0 cannot open raidz1 DEGRADED 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 UNAVAIL 0 0 0 cannot open ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 UNAVAIL 0 0 0 cannot open ad3s4 ONLINE 0 0 0 errors: No known data errors |
Now we repartition drive2 with our new sizes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
freenas:~# fdisk /dev/ad2 The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): d Partition number (1-4): 4 Command (m for help): d Partition number (1-4): 3 Command (m for help): d Partition number (1-4): 2 Command (m for help): d Selected partition 1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-24792, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (24784-24792, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792): Using default value 24792 Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 14990 120407143+ 83 Linux /dev/ad2s2 14991 14999 72292+ 83 Linux /dev/ad2s3 15000 24783 78589980 83 Linux /dev/ad2s4 24784 24792 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad2: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad2s1 1 14990 120407143+ a5 FreeBSD /dev/ad2s2 14991 14999 72292+ a5 FreeBSD /dev/ad2s3 15000 24783 78589980 a5 FreeBSD /dev/ad2s4 24784 24792 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now we replace drive2’s larger partitions in our pool:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
freenas:~# zpool replace tank0 ad2s1 freenas:~# zpool replace tank0 ad2s2 freenas:~# zpool replace tank0 ad2s3 freenas:~# zpool replace tank0 ad2s4 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 27.04% done, 0h0m to go config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad2s1/old UNAVAIL 0 0 0 cannot open ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad2s2/old UNAVAIL 0 0 0 cannot open ad2s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad1s3 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad2s3/old UNAVAIL 0 0 0 cannot open ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s4 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad2s4/old UNAVAIL 0 0 0 cannot open ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
After awhile resilvering should complete and we look clean again:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 02:14:01 2009 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Some more quick tests, TOK as they should.
Size is still the same:
1 2 3 4 5 |
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 3.4G 319G 1% /mnt/tank0 |
It will grow after the last part of any one raid array is replaced with a larger one.
So all is good now we need to swap out the Drive1 (120 Gig) drive with the old Drive2 (200 Gig) drive. So we scrub and export:
1 |
freenas:~# zpool scrub tank0 |
Got to 97.7% then my system gave some errors (didn’t have time to read the screen) and then it rebooted.
I wonder if it’s FreeNAS’s ZFS problem or something else?
After reboot the pool tank0 came up ok by itself. Maybe it did complete then the reboot happened. I’m not 100% sure.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
freenas:~# zpool status pool: tank0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Going to do another scrub…
1 |
freenas:~# zpool scrub tank0 |
Completed without any problems this time. Weird!
Now to export and swap out the drives.
1 |
freenas:~# zpool export tank0 |
Shutdown and swap Drive1 (120 Gig) with old Drive2 (200 Gig).
Reboot and check the zpool
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
freenas:~# zpool status no pools available freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank0 DEGRADED raidz1 DEGRADED ad0s1 ONLINE ad1s1 UNAVAIL cannot open ad2s1 ONLINE ad3s1 ONLINE raidz1 DEGRADED ad0s2 ONLINE ad1s2 UNAVAIL cannot open ad2s2 ONLINE raidz1 DEGRADED ad1s3 UNAVAIL cannot open ad2s3 ONLINE ad3s2 ONLINE raidz1 DEGRADED ad0s3 ONLINE ad1s4 UNAVAIL cannot open ad3s3 ONLINE raidz1 ONLINE ad0s4 ONLINE ad2s4 ONLINE ad3s4 ONLINE |
Now we import our zpool
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
freenas:~# zpool import tank0 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Sun Apr 19 09:19:02 2009 config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 UNAVAIL 0 0 0 cannot open ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 UNAVAIL 0 0 0 cannot open ad2s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad1s3 UNAVAIL 0 0 0 cannot open ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 UNAVAIL 0 0 0 cannot open ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
As you can see ad1 partitions need to be replaced. But first we have to resize our new drive1 (200 Gig) partitions.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
freenas:~# fdisk /dev/ad1 The number of cylinders for this disk is set to 24792. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): d Partition number (1-4): 4 Command (m for help): d Partition number (1-4): 3 Command (m for help): d Partition number (1-4): 2 Command (m for help): d Selected partition 1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-24792, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-24792, default 24792): +123290M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-24792, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-24792, default 24792): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-24792, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-24792, default 24792): +80466M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (24784-24792, default 24784): Using default value 24784 Last cylinder or +size or +sizeM or +sizeK (24784-24792, default 24792): Using default value 24792 Command (m for help): p Disk /dev/ad1: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad1s1 1 14990 120407143+ 83 Linux /dev/ad1s2 14991 14999 72292+ 83 Linux /dev/ad1s3 15000 24783 78589980 83 Linux /dev/ad1s4 24784 24792 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): p Disk /dev/ad1: 203.9 GB, 203928109056 bytes 255 heads, 63 sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad1s1 1 14990 120407143+ a5 FreeBSD /dev/ad1s2 14991 14999 72292+ a5 FreeBSD /dev/ad1s3 15000 24783 78589980 a5 FreeBSD /dev/ad1s4 24784 24792 72292+ a5 FreeBSD Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
New partitions have been created, we can now do the zpool replace thing.
1 2 3 4 |
freenas:~# zpool replace tank0 /dev/ad1s1 freenas:~# zpool replace tank0 /dev/ad1s2 freenas:~# zpool replace tank0 /dev/ad1s3 freenas:~# zpool replace tank0 /dev/ad1s4 |
I don’t wait for each partition to resilver. I type all four replace commands as soon as I can and I let it chug away.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 63.24% done, 0h0m to go config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad0s1 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad1s1/old UNAVAIL 0 0 0 cannot open ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s2 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad1s2/old UNAVAIL 0 0 0 cannot open ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 ad1s3/old UNAVAIL 0 0 0 cannot open ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 ad0s3 ONLINE 0 0 0 replacing DEGRADED 0 0 0 ad1s4/old UNAVAIL 0 0 0 cannot open ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Once it finishes we should look like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
freenas:~# zpool status pool: tank0 state: ONLINE scrub: resilver completed with 0 errors on Sun Apr 19 09:28:56 2009 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Still the same size:
1 2 3 4 5 |
freenas:~# df -H Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 518M 92M 385M 19% / devfs 1.0k 1.0k 0B 100% /dev tank0 322G 3.4G 319G 1% /mnt/tank0 |
Our tank0 pool still hasn’t changed Size, it should after we swap the last (fourth) drives partitions.
So now we get ready to replace the last drive. Drive0 (80 Gig, ad0) with our old drive1 (120 Gig) drive.
1 |
freenas:~# zpool scrub tank0 |
Once it completes (check with #zpool status) we export it and shutdown the computer.
1 |
freenas:~# zpool export tank0 |
Ok to shutdown and swap drives now.
With the last drive replaced, Drive0 (80 Gig) is Drive0 (120 Gig).
We do it all one last time:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
freenas:~# zpool status no pools available freenas:~# zpool import pool: tank0 id: 325956304794594593 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank0 DEGRADED raidz1 DEGRADED ad0s1 UNAVAIL cannot open ad1s1 ONLINE ad2s1 ONLINE ad3s1 ONLINE raidz1 DEGRADED ad0s2 UNAVAIL cannot open ad1s2 ONLINE ad2s2 ONLINE raidz1 ONLINE ad1s3 ONLINE ad2s3 ONLINE ad3s2 ONLINE raidz1 DEGRADED ad0s3 UNAVAIL cannot open ad1s4 ONLINE ad3s3 ONLINE raidz1 DEGRADED ad0s4 UNAVAIL cannot open ad2s4 ONLINE ad3s4 ONLINE freenas:~# zpool import tank0 |
Now repartition drive0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
freenas:~# fdisk /dev/ad0 The number of cylinders for this disk is set to 15017. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): d Partition number (1-4): 4 Command (m for help): d Partition number (1-4): 3 Command (m for help): d Partition number (1-4): 2 Command (m for help): d Selected partition 1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-15017, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-15017, default 15017): +123290M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (14991-15017, default 14991): Using default value 14991 Last cylinder or +size or +sizeM or +sizeK (14991-15017, default 15017): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (15000-15017, default 15000): Using default value 15000 Last cylinder or +size or +sizeM or +sizeK (15000-15017, default 15017): +64M Command (m for help): n Command action e extended p primary partition (1-4) p Selected partition 4 First cylinder (15009-15017, default 15009): Using default value 15009 Last cylinder or +size or +sizeM or +sizeK (15009-15017, default 15017): Using default value 15017 Command (m for help): p Disk /dev/ad0: 123.5 GB, 123521334784 bytes 255 heads, 63 sectors/track, 15017 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/ad0s1 1 14990 120407143+ 83 Linux /dev/ad0s2 14991 14999 72292+ 83 Linux /dev/ad0s3 15000 15008 72292+ 83 Linux /dev/ad0s4 15009 15017 72292+ 83 Linux Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): a5 Changed system type of partition 1 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): a5 Changed system type of partition 2 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): a5 Changed system type of partition 3 to a5 (FreeBSD) Command (m for help): t Partition number (1-4): 4 Hex code (type L to list codes): a5 Changed system type of partition 4 to a5 (FreeBSD) Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. |
Now we replace our new partitions into the pool:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
freenas:~# zpool replace tank0 /dev/ad0s1 freenas:~# zpool replace tank0 /dev/ad0s2 freenas:~# zpool replace tank0 /dev/ad0s3 freenas:~# zpool replace tank0 /dev/ad0s4 freenas:~# zpool status pool: tank0 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 14.32% done, 0h1m to go config: NAME STATE READ WRITE CKSUM tank0 DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 ad0s1/old UNAVAIL 0 0 0 cannot open ad0s1 ONLINE 0 0 0 ad1s1 ONLINE 0 0 0 ad2s1 ONLINE 0 0 0 ad3s1 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 ad0s2/old UNAVAIL 0 0 0 cannot open ad0s2 ONLINE 0 0 0 ad1s2 ONLINE 0 0 0 ad2s2 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad1s3 ONLINE 0 0 0 ad2s3 ONLINE 0 0 0 ad3s2 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 ad0s3/old UNAVAIL 0 0 0 cannot open ad0s3 ONLINE 0 0 0 ad1s4 ONLINE 0 0 0 ad3s3 ONLINE 0 0 0 raidz1 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 ad0s4/old UNAVAIL 0 0 0 cannot open ad0s4 ONLINE 0 0 0 ad2s4 ONLINE 0 0 0 ad3s4 ONLINE 0 0 0 errors: No known data errors |
Once it finishes, we should now have a nice clean pool: