How to Replace a Failed Drive in a ZFS Pool


So you have a failed disk in a ZFS pool and you want to fix it? Routine disk failures are really a non-event with ZFS because the volume management makes replacing them so dang easy. In many cases, unlike hardware RAID or older volume management solutions, the replacement disk doesn’t even need to be exactly the same as the original. So let’s get started replacing our failed disk. These instructions will be for a Solaris 10 system, so a few of the particulars related to unconfiguring the disk and device paths will vary with different flavors of UNIX.

First, take a look at the zpools to see if there are any errors. The -x flag will only display status for pools that are exhibiting errors or are otherwise unavailable.
Note: If the disk is actively failing (a process that sometimes takes a while as the OS offlines it), any commands that use storage related system calls will hang and take a long time to return. These include “zpool” and “format”, so just be patient; they will eventually return.

# zpool status -x

 pool: data
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
 scrub: none requested

        NAME        STATE     READ WRITE CKSUM
        data        DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            c1t4d0  ONLINE       0     0     0
            c1t5d0  FAULTED      1    81     0  too many errors
          mirror-1  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
            c1t3d0  ONLINE       0     0     0

errors: No known data errors

So we can easily see that c1t5d0 has failed. Take a look at the “format” output do get the particulars about the disk:
# format

Searching for disks...done

       0. c1t0d0 
       1. c1t1d0 
       2. c1t2d0 
       3. c1t3d0 
       4. c1t4d0 
       5. c1t5d0 
Specify disk (enter its number): 

Get your hands on a replacement disk that is as similar as possible to a SEAGATE-ST914602SSUN146G-0603-136.73GB. I was only able to dig up a HITACHI-H103014SCSUN146G-A2A8-136.73GB, so I’ll be using that instead of a direct replacement.

Next, use “cfgadm” to look at the disks you have and their configuration status:

# cfgadm -al

Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             scsi-sata    connected    configured   unknown
c1::dsk/c1t0d0                 disk         connected    configured   unknown
c1::dsk/c1t1d0                 disk         connected    configured   unknown
c1::dsk/c1t2d0                 disk         connected    configured   unknown
c1::dsk/c1t3d0                 disk         connected    configured   unknown
c1::dsk/c1t4d0                 disk         connected    configured   unknown
c1::dsk/c1t5d0                 disk         connected    configured   unknown

We want to replace t5, so we prepare it for removal by unconfiguring it:

# cfgadm -c unconfigure c1::dsk/c1t5d0

The “safe to remove” led should turn on and you can pull the disk, remembering to allow it several seconds to spin down. Replace it with the new disk and take a look at “cfgadm -al” output again to ensure that it has been automatically configured. If it has not, you can manually configure it like below:

# cfgadm -c configure c1::dsk/c1t5d0

Now, it’s a simple matter of a quick “zpool replace” to get things rebuilding:

# zpool replace data c1t5d0

You can use the output of zpool status to watch the resilver process… Version 2.0

We just started supporting Solaris 10 in our VMware cluster so I had to update my zone type script to detect if the OS is running there. I’m not sure how I feel about depending on the output of ptrdiag since the interface is labeled “unstable”, but it works for now, and I really don’t see Sun changing the first line of output where the system configuration is listed. Anyhow, when issued with the -v or –vmware flag, the script returns 0 if it’s running on the cluster and 1 if it is not.


# -g or –global
Return 0: The machine is a global zone with 1 or more local zones
Return 1: The machine is not a global zone

# -l or –local
Return 0: The machine is a local zone
Return 1: The machine is not a not a local zone

# -v or –vmware
Return 0: The machine is running on a VMware hypervisor
Return 1: The machine is not running in VMware

#! /bin/bash
# When issued with the -g or --global flag, this script will return:
# 0 if the machine is a global zone and has one or more local zones. 
# Otherwise, it will return 1
# When issued with the -l or --local flag, this script will return:
# 0 if if is a local zone and 1 if it is not
# When issued with the -v or --vmware flag, this script will return:
# 0 if it is a vmware host and 1 if not.

list=( `/usr/sbin/zoneadm list -civ | awk '{ print $1 }'`)

  case "$1" in
        # If the third element in our array is null, set it to 0
        if [ "${list[2]}" == ""  ]; then
        # This is a global zone only if it has one or more local zones.
        if [ ${list[1]} -eq 0 ] && [ ${list[2]} -ge 1 ]; then
        # 1 is returned if we have a global and local zone, 
        # otherwise, we return 0
                exit 0
                exit 1
        # If the second element in our array is = or > 1, it is a local zone.
        if [ ${list[1]} -ge 1 ]; then
        # Return 1 if this is a local zone, otherwise return 0.
                exit 0
                exit 1

        # Don't run our check on local zones... Prtdiag can't run there
        if [ ${list[1]} != 0 ]; then
                exit 1
                vmhost=( `/usr/sbin/prtdiag | grep System | awk '{ print $5 }'`)
                if [ $vmhost == VMware ]; then
                        #If the host is running on the vmware cluster return 0, 
                        # otherwise, return 1
                        exit 0
                        exit 1
        echo "Usage: /local/adm/ {-l | --local | -g | --global | -v | --vmware}"
        exit 1

UNIX – Find Files that Changed Within Time Window

Every so often us lowly UNIX admins find ourselves needing to search a file system for files that have been created or changed within a certain time window. In other words, those files that are newer than time “X”, but not newer than time “Y”. There are a number of ways to accomplish this, but my preferred method is to create two reference files to indicate the beginning and end of my window and use the “-newer” and “! -newer” flags to search for files that changed within that window.

# touch -amt 200910260000 /tmp/starttime
# touch -amt 200910262359 /tmp/endtime
# find / -type f -newer /tmp/starttime -a ! -newer /tmp/endtime

The guys at point out that it is more elegant to accomplish this without creating two files, but their solution does not work with operating systems that use strict POSIX compliant “find” implementations, making it of little use in some cases. For the curious, here is their example:

# find . -type f -newermt 2009-10-26 ! -newermt 2009-10-27

Happy 1234567890’th Second UNIX!

Today, Friday February 13, at 3:31 PM (PST), the UNIX time will read exactly 1234567890. So exacly what is all this excitement about UNIX being able to count to 10? Surely, the operating system that is slowly but steadily putting Microsoft out of business must be able to do that. Well, it’s actually the UNIX time stamp, and what has all of us nerds talking is really just the fact that the numbers have never lined up in sequence like this before.

So what the heck is this UNIX time anyhow? Well, simply put, it’s actually the exact number of seconds since the the Unix epoch. This was 00:00:00 UTC on January 1, 1970.

From Wikipedia:

It is not a linear representation of time nor a true representation of UTC (though it is frequently mistaken for both) as the times it represents are UTC but it has no way of representing UTC leap seconds (e.g. 1998-12-31 23:59:60).

Install Solaris Package in Alternate Base Directory

Unless you specify a different administrative file, the pkgadd command reads “/var/sadm/install/admin/default”, which specifies the base directory as “/opt”. Do not change the settings in this file, but rather create a custom admin file and enter an alternate “basedir” directive if you want to install your package into a different directory. We are going to install our package into “/var/applications”, and call our custom admin file “custom”.

First, create and edit “/var/sadm/install/admin/custom”, adding a line similar to this:

Next, issue the pkgadd command with the “-a” flag to call you alternative admin file:

pkgadd -d device -a custom PackageName

This really comes in handy when your customers want to retain control over their packages, but you don’t want to give them access to write packages into the system area. More detailed instructions can be found here.

Strange X11 Forwarding Problem

I started getting this error:
X11 connection rejected because of wrong authentication
when trying to forward X11 applications from a Linux server to my Mac. I had been forwarding the display on this server for years, so I was a little unsure what could be causing it. In the end, it turned out that I had filled up /var, and X11 could not write to “/var/log/XFree86.0.log”. It was an easy fix, but the error was certainly no help.

PHP and Sed for String Substitution

I needed to replace a string in several thousand files scattered all over the filesystem on one of our servers. I used find to create a list of files that needed to be changed, along with their complete path and called it “list.txt”. It looked something like this:

/path/to/directory with spaces/filefour.txt
and so on...

I worked out the “sed” command to do the in place editing, and Zach helped me whip up a quick PHP script to read the contents of “list.txt” into an array and iterate through it. He was also nice enough to show me how to use “str_replace” to escape any annoying spaces that happened to find their way into the names of directories.

  1. < ?php
  2. $files=file('list.txt');
  3.         foreach($files as $file)
  4.         {
  5.         $command='/bin/sed -i \'s/old-string/new-string/g\' '.str_replace(' ','\ ',$file);
  6.         exec($command);
  7.         }
  8. ?>

It’s a handy little script that I’m sure I will find a use for later, so I thought I would put it up here.