Registering Solaris CLARiiON Hosts With QLA 2310 HBAs

Sun Microsystems likes the QLA 2310 Fiber Channel HBA. It’s only a 2Gig card, but it works with the Sun native driver, which makes it wonderful for us Solaris Administrators. Unfortunately, it does not integrate perfectly with EMC CLARiiON SANs because it does not register properly with Navasphere. Even if you manually register the host, the LUNs will not be presented to the host because the agent can’t pass commands to the array.

To remedy this situation on my Solaris 8 host, I used the following procedure:

Edit the /etc/system file and add the following line:

set fcp:ssfcp_enable_auto_configuration=1

Next, I rebooted my Solaris host with the “-r” flag:

reboot -- -r

Next I checked Navisphere to make sure my paths have logged in. They were, so I logged into the Solaris host and ran the following commands:

cfgadm
devfsadm
format

I then saw the storage that was presented to my host. Finally, I restarted the Navisphere agent and started using my new LUNs.

How to Disable Automatic FSCK on EXT3 Filesystems

The e2fsck will regularly force a check of a filesystem even if the filesystem is marked clean. By default, this happens on every twenty mounts or 180 days, whichever comes first.

The ext3 filesystem does this as well, which can be annoying if you have a very large filesystem and a short downtime window. Therefore, it’s a good idea to disable this feature on large volumes. Keep in mind that you should still run fsck occasionally, by disabling the automatic checks, you get to Decide when, not the system.

Use the command:

tune2fs -i 0 /dev/hdxx

This disables periodic, automatic checking.

Setting Up The Automounter Service on RHEL

Mounting filesystems in RHEL is pretty straightforward and easy. Occasionally, however, you will not want the filesystem to remain mounted all the time, but rather to automatically mount for a set period of time only when it is needed. Because of networking overhead, and the general unreliability of networks, NFS mounts are a good example of when this can be especially useful.

In order to manage the automatic mounting and unmounting of filesystems on RHEL, we use the Automounter service. Here is how.

First, The main configuration file is “/etc/auto.master”. It should look something like this:

#
# $Id: auto.master,v 1.3 2003/09/29 08:22:35 raven Exp $
#
# Sample auto.master file
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# For details of the format look at autofs(5).
#/misc  /etc/auto.misc --timeout=60
#/misc  /etc/auto.misc
#/net   /etc/auto.net


Let’s assume that we want to set up an NFS mount on “/misc/backups”. We would first create an entry in this file that looks something like this:

/misc   /etc/auto.misc --timeout=120


This tells the autofs service that we want to use it to manage mounts from within “/misc”, that the configuration file is “/etc/auto.misc”, and that it should disconnect after 2 minuets of inactivity.

Now, let’s edit the “/etc/auto.misc” file. The file has three columns: the mount point from within the /misc directory, the options for mounting the filesystem, and the filesystem to be mounted. It also includes the remote server’s name since we are using NFS. It should look something like this when you are done:

#
# $Id: auto.misc,v 1.2 2003/09/29 08:22:35 raven Exp $
#
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage

cd              -fstype=iso9660,ro,nosuid,nodev :/dev/cdrom
backups         -rw,soft,intr remoteservername:/path/to/nfs/export

# the following entries are samples to pique your imagination
#linux          -ro,soft,intr           ftp.example.org:/pub/linux
#boot           -fstype=ext2            :/dev/hda1
#floppy         -fstype=auto            :/dev/fd0
#floppy         -fstype=ext2            :/dev/fd0
#e2floppy       -fstype=ext2            :/dev/fd0
#jaz            -fstype=ext2            :/dev/sdc1
#removable      -fstype=ext2            :/dev/hdd


Next, we create the directory for the mount point in /misc:

# mkdir /misc/backups

And finally we restart the autofs service:

# service autofs restart

That should pretty much do it. If you don’t have autofs configured to start up, you can use chkconfig to enable it. “/misc/backups” will now be mounted whenever a user or process attempts to access data on it, and it will be automatically disconnected after 120 seconds of inactivity. Last, but not least, you can always confirm that it is running with the “service” command:

# service autofs status

As always, change the details to match your own requirements.

Working With Disk Labels in RHEL

When you install RHEL, the filesystems are labeled for you. Usually you won’t have to mess with it anymore, but on occasion, you may want to change them to more accurately represent the data that is stored on that partition. If, for instance, you used to have all of your database files on a partition labeled “/database”, but you have now moved them somewhere else, and you now wish to house your user account data there, it would make sense to change the label to something like “/users”.

Labels are, of course, arbitrary, so there is no technical need to do this, and you could, instead simply change the mount point in the fstab file, mounting the partition by device name rather than label, but it is usually cleaner to change the label. Here is how you do it:

First, let’s figure out what the partition is currently labeled as:

[root@calvin /]# /sbin/e2label /dev/hda4
/database
[root@calvin /]#

It’s current label is “/database”, and, since we have moved the database data somewhere else, we now want to store our user account data here, we need to change it to “/users”.

[root@calvin /]# /sbin/e2label /dev/hda4 /users
[root@calvin /]#

That’s all there is to it, now we check to make sure we have done what we think we have done.

[root@calvin /]# /sbin/e2label /dev/hda4
/users
[root@calvin /]#

Sure enough, it’s now labeled “/users” and the data on the disk remains intact. Now all we have to do is change the appropriate entry in the “/etc/fstab” file to represent the change.

Change this:

LABEL=/database       /databases            ext3    defaults        1 2


To this:

LABEL=/users          /users                ext3    defaults        1 2


And you’re all set to go. Make sure you have unmounted “/databases” before making the change.

Now, just run:
[root@calvin /]# mount /users
[root@calvin /]#

And you’re all set to go. As always, change the values here to represent those in your environment.

Using Sort to List Directories by Size

If you manage a UNIX system with a large number of directories that vary in size, chances are that you’ve needed to figure out which ones are using up the most disk space. Of course if the directories are user accounts, the best way to do this is to enable quotas and use the “repquota” command. If you just have a bunch of directories, however, you can easily figure out which ones are largest by giving the correct arguments to “du” and “sort”. Here is how:

du -sk * | sort +0nr

This will display the size of all directories and sort them from largest to smallest. If you want to sort them from smallest to largest, simply remove the “r”.

du -sk * | sort +0n

If you have nested directories, you will need to incorporate foreach to recurse through and get all the directory names.

Taking Disk Cylinders From Swap on Solaris 8

Kids… DO NOT TRY THIS AT HOME! If this is not done exactly right, you will render your system unbootable and corrupt your data. That being said, under some circumstances you can take some space from your swap partition and add it to an unused one without initializing your entire disk. This is particularly useful if you decide you want to use DiskSuite to mirror your system disk, but have not allocated the 100MB partition that is needed to hold the state databases. As always, BACK EVERYTHING UP FIRST. Better yet, make two backups and store them on two different systems. This is a risky procedure, and you don’t want to lose any data!

You can also use my instructions for copying a Solaris boot drive to a disk with a different partition layout as a safer alternative.

The first thing you need to do is figure out if your disk layout will allow for this procedure. Usually the swap partition is the second one on the disk, making it partition number 1 (Partition number 0 is root). If partition number 1 is swap on your system, and partition number 3 or 4 are unused, you are in good shape, and this should work. To figure this out, you should do something like this:

# format
Select the boot disk – usually disk 0
Specify disk (enter its number): 0
format> partition
format> print

This will show you the current disk layout.


Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9436       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm       0                0         (0/0/0)            0
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

Here we see that partitions 3 and 4 are unused and directly after partition 1, so we can take some space from swap and assign it to one of these. Partition 2 is, of course the entire disk. I have not tried it, so I don’t know if you could assign non-sequential cylinders to a partition that is not directly after swap.

So to take some space from partition 1 and add it to partition 3, the first thing we have to do is disable swap, so the format utility will let us change it.

Comment out the following lines in your /etc/vfstab file and reboot the system.


#/dev/dsk/c1t0d0s1         -       -               swap    -       no      -
#swap    -       /tmp    tmpfs   -       yes     - 

This will bring the system up without swap enabled. You can now edit the disk label. Remember that our cylinders need to be sequential, so always work in cylinders when using the format utility.

Re-enter the format utility, select your system disk and view the partition table:

# format
Select the boot disk – usually disk 0
Specify disk (enter its number): 0
format> partition
format> print

Again we wee that partitions 3 and 4 are unused.


Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9436       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm       0                0         (0/0/0)            0
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

The first thing we need to do is take some cylinders away from partition 1. In this example, we are looking to make partition 3 roughly 100MB, so we need to take about 75 cylinders from partition 1 so that we can add it to partition 3. Parititon 1 ends at cylinder 9436, so we need to subtract 75 from that number. 9436 – 75 = 9361, so that is the new ending cylinder for partition 1. We then subtract the beginning cylinder (726) from that number to give us the new total number of cylinders for partition 1. 9361 – 726 = 8635, so this is the number we enter when format asks for the size of the partition. Like so:


partition> 1
Part      Tag    Flag     Cylinders         Size            Blocks
  1       swap    wu     726 -  9360       11.90GB    (8635/0/0)  24946515

Enter partition id tag[swap]: 
Enter partition permission flags[wu]: 
Enter new starting cyl[726]: 
Enter partition size[24946615b, 9436c, 12880.92mb, 12.00gb]: 8635c
partition>

Now we have to add these 75 cylinders to partition 3.


partition> 3
Part      Tag    Flag     Cylinders         Size            Blocks
  3 unassigned    wm       0                0          (0/0/0)            0

Enter partition id tag[unassigned]: 
Enter partition permission flags[wm]: 
Enter new starting cyl[0]:9361
Enter partition size[0b, 0c, 0.00mb, 0.00gb]:75c
partition>

Print out the new partition table to make sure everything lines up correctly:


partition> print
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9360       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm    9361 -  9436      107.21MB    (76/0/0)      219564
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

Partition 1 ends at cylinder 9360, and partition 3 picks right up at cylinder 9361. Partition 3 ends at cylinder 9436, and partition 5 begins at cylinder 9437. Partition 4, of course, remains unused. Since none of the cylinders overlap, we can go ahead and write the disk label out. DO NOT DO THIS if you have any doubt at all about what you have just done. By writing out the disk label, you could corrupt the data on your formated filesystems if any cylinders overlap into them. The format utility is usually pretty smart about keeping you from making mistakes, but be very careful anyway! You don’t want to end up with scrambled eggs on a disk that has valuable data on it.

partition> label
This writes out the disk label, so you can now exit the format utility and re-enable swap in your /etc/vfstab file. Simply uncomment out the following two lines and reboot the system.


/dev/dsk/c1t0d0s1         -       -               swap    -       no      -
swap    -       /tmp    tmpfs   -       yes     -

Reboot your system, and if all goes well, it will come up, and you will see that partition 3 will have a little over 100MB on it. Usually people want to do this so they can store the DiskSuite meta database on the newly created partition. If this is the case for you, you can now move on to mirroring the system disk.

Making RHEL 3 See Multiple LUNS

For some reason RHEL 3 comes out of the box configured to see only the first Lun on a SCSI channel. This is usually not a problem, as the first Lun is all you care about, but in some instances, you will need to configure the SCSI module to see multiple Luns.

In this case we are using an Adaptec DuraStor 6200S, which is set up to present the RAID controller as Lun 00, and the actual RAID array as Lun 01. Without any modifications to the system, we plug in in, and after a reboot check /proc/scsi/scsi. We can see the RAID controller, but since we can only see the first Lun on the channel, we never get to the array:

Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: Adaptec Model: DuraStor 6200S Rev: V100
Type: Processor ANSI SCSI revision: 03

The actual array would show up as “Channel: 00 Id: 00 Lun: 01”, but it’s not there. To resolve this, we have to first edit “/etc/modules.conf” and add the following line:

options scsi_mod max_scsi_luns=128 scsi_allow_ghost_devices=1

In our case, modules.conf looks like this after the modification:

alias eth0 e1000
alias eth1 e1000
alias scsi_hostadapter megaraid2
alias usb-controller usb-uhci
alias usb-controller1 ehci-hcd
alias scsi_hostadapter1 aic7xxx
options scsi_mod max_scsi_luns=128 scsi_allow_ghost_devices=1

Next we have to build a new initrd image. This is done with the “mkinitrd” command.

WARNING: MAKE DARN SURE you build this against the right kernel (the kernel you want to use). If you are going to replace your current initrd image with the new one, you should make a back-up copy first. The -f option will force or overwrite the current initrd image file.

cp /boot/initrd-2.4.21-47.ELsmp.img /boot/initrd-2.4.21-47.ELsmp.img.bak
mkinitrd -f -v /boot/initrd-2.4.21-47.ELsmp.img 2.4.21-47.ELsmp

Once this is done, you can reboot your machine, and check “/proc/scsi/scsi” to see confirm that it sees the second Lun. You should see something like this:

Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: Adaptec Model: DuraStor 6200S Rev: V100
Type: Processor ANSI SCSI revision: 03

Host: scsi2 Channel: 00 Id: 00 Lun: 01
Vendor: Adaptec Model: DuraStor 6200S Rev: V100
Type: Direct-Access ANSI SCSI revision: 03

Hat Tip: Alan Baker for help figuring this out.
UPDATE: RHEL 4 doest not have this problem.