X11 Forwarding Broken on Solaris

If you’re running Solaris 8 or 9 and an upgrade results in broken SSH X11 forwarding, the problem may be Sun’s socfs bug. The symptom will be SSH’s failure to set the $DISPLAY variable and an error in your system log looking something like this:

Jun 3 09:40:24 servername sshd[26432]: [ID 800057 auth.error] error: Failed to allocate internet-domain X11 display socket.

To fix this, you can either install Sun’s latest socfs patch for your version of the OS, or simply force sshd into IPv4 mode by doing the following:

Edit you sshd_config file, adding the following:

# IPv4 only
ListenAddress 0.0.0.0

Edit your sshd startup script to issue a “-4″ to sshd on start:

case “$1″ in
’start’)
echo ’starting ssh daemon’
/usr/local/sbin/sshd -4
;;

Restart sshd, and that should pretty much do it… Enjoy.

Problems Registering Solaris Hosts With QLA 2310 HBAs in Navasphere

Sun Microsystems likes the QLA 2310 Fiber Channel HBA. It’s only a 2Gig card, but it works with the Sun native driver, which makes it wonderful for us Solaris Administrators. Unfortunately, it does not integrate perfectly with EMC CLARiiON SANs because it does not register properly with Navasphere. Even if you manually register the host, the LUNs will not be presented to the host because the agent can’t pass commands to the array.

To remedy this situation on my Solaris 8 host, I used the following procedure:

Edit the /etc/system file and add the following line:

set fcp:ssfcp_enable_auto_configuration=1

Next, I rebooted my Solaris host with the “-r” flag:

reboot -- -r

Next I checked Navisphere to make sure my paths have logged in. They were, so I logged into the Solaris host and ran the following commands:

cfgadm
devfsadm
format

I then saw the storage that was presented to my host. Finally, I restarted the Navisphere agent and started using my new LUNs.

Another Used Sun Fire T2000 For Sale

STATUS: Sold

We have another used Sun Fire T2000 server. Since the first one was sold in less than 24 hours, I thought it would be nice to offer this one up to this site’s readers as well. It has never been used in production and is in like-new condition.

  • Sun Fire T2000 Server
  • 8 core 1.0 Ghz UltraSPARC T1 processor
  • 16GB DDR memory (16 1GB DIMMs)
  • 2X 73GB 10,000PRM disk drives
  • 1X DVD-ROM/CDRW drive
  • Serial Number: 0639VB0053
  • Starting Bid: $13,000

If you are interested, or would like any further information, please leave a comment or visit this link. Our university campus policy states that big ticket items like this must be sold using an auction style bidding system. I should add, however, that most items in the surplus system are sold with only one bid, so please don’t let the process discourage you if you are at all interested.

Used Sun Fire T2000 For Sale

STATUS: Sold

The University where I work is currently selling a used Sun Fire T2000 server. Since many of this site’s readers are fellow Solaris administrators, I thought it would be nice to offer it up to them. It has never been used in production and is in like-new condition.

  • Sun Fire T2000 Server
  • 4 core 1.0 Ghz UltraSPARC T1 processor
  • 8GB DDR memory (16 512MB DIMMs)
  • 2X 73GB 10,000PRM disk drives
  • 1X DVD-ROM/CDRW drive
  • Serial Number: 0617NNN1FY
  • Starting Bid: $6,000

If you are interested, or would like any further information, please leave a comment of visit this link. Our campus policy states that big ticket items like this must be sold using an auction style bidding system. I should add, however, that most items in the surplus system are sold with only one bid, so please don’t let the process discourage you if you are at all interested.

Taking Disk Cylinders From Swap on Solaris 8

Kids… DO NOT TRY THIS AT HOME! If this is not done exactly right, you will render your system unbootable and corrupt your data. That being said, under some circumstances you can take some space from your swap partition and add it to an unused one without initializing your entire disk. This is particularly useful if you decide you want to use DiskSuite to mirror your system disk, but have not allocated the 100MB partition that is needed to hold the state databases. As always, BACK EVERYTHING UP FIRST. Better yet, make two backups and store them on two different systems. This is a risky procedure, and you don’t want to lose any data!

You can also use my instructions for copying a Solaris boot drive to a disk with a different partition layout as a safer alternative.

The first thing you need to do is figure out if your disk layout will allow for this procedure. Usually the swap partition is the second one on the disk, making it partition number 1 (Partition number 0 is root). If partition number 1 is swap on your system, and partition number 3 or 4 are unused, you are in good shape, and this should work. To figure this out, you should do something like this:

# format
Select the boot disk - usually disk 0
Specify disk (enter its number): 0
format> partition
format> print

This will show you the current disk layout.


Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9436       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm       0                0         (0/0/0)            0
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

Here we see that partitions 3 and 4 are unused and directly after partition 1, so we can take some space from swap and assign it to one of these. Partition 2 is, of course the entire disk. I have not tried it, so I don’t know if you could assign non-sequential cylinders to a partition that is not directly after swap.

So to take some space from partition 1 and add it to partition 3, the first thing we have to do is disable swap, so the format utility will let us change it.

Comment out the following lines in your /etc/vfstab file and reboot the system.


#/dev/dsk/c1t0d0s1         -       -               swap    -       no      -
#swap    -       /tmp    tmpfs   -       yes     -

This will bring the system up without swap enabled. You can now edit the disk label. Remember that our cylinders need to be sequential, so always work in cylinders when using the format utility.

Re-enter the format utility, select your system disk and view the partition table:

# format
Select the boot disk - usually disk 0
Specify disk (enter its number): 0
format> partition
format> print

Again we wee that partitions 3 and 4 are unused.


Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9436       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm       0                0         (0/0/0)            0
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

The first thing we need to do is take some cylinders away from partition 1. In this example, we are looking to make partition 3 roughly 100MB, so we need to take about 75 cylinders from partition 1 so that we can add it to partition 3. Parititon 1 ends at cylinder 9436, so we need to subtract 75 from that number. 9436 - 75 = 9361, so that is the new ending cylinder for partition 1. We then subtract the beginning cylinder (726) from that number to give us the new total number of cylinders for partition 1. 9361 - 726 = 8635, so this is the number we enter when format asks for the size of the partition. Like so:


partition> 1
Part      Tag    Flag     Cylinders         Size            Blocks
  1       swap    wu     726 -  9360       11.90GB    (8635/0/0)  24946515

Enter partition id tag[swap]:
Enter partition permission flags[wu]:
Enter new starting cyl[726]:
Enter partition size[24946615b, 9436c, 12880.92mb, 12.00gb]: 8635c
partition>

Now we have to add these 75 cylinders to partition 3.


partition> 3
Part      Tag    Flag     Cylinders         Size            Blocks
  3 unassigned    wm       0                0          (0/0/0)            0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]:9361
Enter partition size[0b, 0c, 0.00mb, 0.00gb]:75c
partition>

Print out the new partition table to make sure everything lines up correctly:


partition> print
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -   725        1.00GB    (726/0/0)    2097414
  1       swap    wu     726 -  9360       11.90GB    (8635/0/0)  24946515
  2     backup    wm       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wm    9361 -  9436      107.21MB    (76/0/0)      219564
  4 unassigned    wm       0                0         (0/0/0)            0
  5        usr    wm    9437 - 10888        2.00GB    (1452/0/0)   4194828
  6        var    wm   10889 - 18148       10.00GB    (7260/0/0)  20974140
  7 unassigned    wm   18149 - 24619        8.91GB    (6471/0/0)  18694719

Partition 1 ends at cylinder 9360, and partition 3 picks right up at cylinder 9361. Partition 3 ends at cylinder 9436, and partition 5 begins at cylinder 9437. Partition 4, of course, remains unused. Since none of the cylinders overlap, we can go ahead and write the disk label out. DO NOT DO THIS if you have any doubt at all about what you have just done. By writing out the disk label, you could corrupt the data on your formated filesystems if any cylinders overlap into them. The format utility is usually pretty smart about keeping you from making mistakes, but be very careful anyway! You don’t want to end up with scrambled eggs on a disk that has valuable data on it.

partition> label
This writes out the disk label, so you can now exit the format utility and re-enable swap in your /etc/vfstab file. Simply uncomment out the following two lines and reboot the system.


/dev/dsk/c1t0d0s1         -       -               swap    -       no      -
swap    -       /tmp    tmpfs   -       yes     -

Reboot your system, and if all goes well, it will come up, and you will see that partition 3 will have a little over 100MB on it. Usually people want to do this so they can store the DiskSuite meta database on the newly created partition. If this is the case for you, you can now move on to mirroring the system disk.

Changes to Solaris ssh_config for Password-less SSH Login

Zach over at NoSheep has written a really nice article about how to configure a SSH trust relationship on UNIX systems. By doing this, you can prevent the system from prompting you for a password, thus allowing the user to use scripted solutions to move files around with sftp, etc. He didn’t mention, however, that to get this up and working correctly under Solaris, you have to uncomment the following lines in your /usr/local/etc/ssh_config file:

RSAAuthentication yes

And one of these lines. In most cases the Identity File will be id_rsa.

# IdentityFile ~/.ssh/identity
IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa

Changing Linux Mount Points

If you’re familiar with UNIX, you know that changing mount points is really pretty easy. All you have to do is go into “/etc/fstab”, “/etc/vfstab” (or whatever your flavor of UNIX happens to call its filesystem table) and change the mount directory.

If, for instance, you had a Solaris box, and you wanted to make the disk currently mounted as “/data” be mounted as “/database”, all you would have to do is the following:

# umount /data
# mv /data /database
Change this line in “/etc/vfstab” from something like this:
/dev/dsk/c1d0s6 /dev/rdsk/c1d0s6 /data ufs 1 yes -
to something like this:
/dev/dsk/c1d0s6 /dev/rdsk/c1d0s6 /database ufs 1 yes -
and remount it as “/database”.
# mount /database

With Linux, however, it’s not quite so clear anymore… It’s still easy, but it’s just not so clear what you have to do since they have now taken to mounting filesystems using the volume label. Rather than pointing directly to the disk device, Linux points to the label, and “/etc/fstab” look more like this:

LABEL=/data /data ext3 defaults 1 2

You can always simply change the disk label, but if you don’t care, you can just tell linux where the raw device is, bypassing the need to worry about the label. The easiest way to do this is simply to replace the “LABEL=/data” value to the “/dev” entry of the disk itself. Then, simply change “/data” to “/database” and you’re all set.

Here is an example of what you would do to change the mountpoint of “/data” to /database”:

# umount /data
# mv /data /database
Change this line in “/etc/fstab” from this:
LABEL=/data /data ext3 defaults 1 2
to this:
/dev/sda6 /database ext3 defaults 1 2
and remount it as /database
# mount /database

Remembering to change the example values here with those required for your situation.

How To Increase The Maximum Number of Processes Per UID in Solaris

If you are running solaris 8 or 9, and getting strange errors like:

“VFork failed”

when you try to run commands, or

“Mar 31 10:40:32 sauron genunix: [ID 748887 kern.notice] NOTICE: out of per-user processes for uid 1234″

from the dmesg command output, the most likely cause is that you have exceeded the maximum number of processes per user that you are allowed to run on your server.

To be sure, you can run:

ps -ef | grep <uid> | wc -l

and compare that number against the v.v_proc settings from the command:

sysdef -i

There will be a lot of output from “sysdef -i”, but you are looking for “v.v_proc” and “v.v_maxup”. “v.v_proc” is the max number of processes per user, and “v.v_maxup” is the max number of processes plus the reserved number for root.

You should see something like this:

4058 maximum number of processes (v.v_proc)
99 maximum global priority in sys class (MAXCLSYSPRI)
4053 maximum processes per user id (v.v_maxup)

Since the settings in this example is 4058, if 4058 is something like what you are seeing in the output of:

ps -ef | grep <uid> | wc -l

Than you will need to increase the value of v.v_proc and v.v_maxup. The good news is it’s easy to to. The bad news is that the settings you need to change are NOT v.v_proc and v.v_maxup. It is, instead, the “maxusers” value. The other bad news is that you can’t change them dynamicly.

Here is what you will most likely want to add to your /etc/system file. Tweak the setting as needed:

set maxusers = <This should be something like the amount of available physical memory in MB>

Then reboot the system, and your maximum number of processes per user should now be set to something much more reasonable than the Solaris defaults.

As you can see, setting maxusers to a value will control max_nprocs and maxuprc. The algorythms are:
max_nprocs = 10 + (16 x maxusers)
maxuprc = max_nprocs - reserved_procs (default is 5)

As a result, usually only maxusers is tuned.

Solaris Automounter

Whenever you’re using NFS mount points, it’s really nice to use some type of automounter. Linux and FreeBSD use AMD to accomplish this, but Solaris uses automountd, and it’s fun and easy to use… Here is an example of a configuration that will automatically mount an NFS share and unmount it after 5 minuets of inactivity.

We have a system called micky which has an NFS point shared to a system called minny as /shareme.
We can see that it is set up in the /etc/dfs/dfstab file on micky:

share -F nfs -o ro=minny.yourdomain.com -d “NFS ShareMe” /shareme

The above will share the directory read-only. If you would like to map the directory as root and be able to write to it, the command would look more like this:

share -F nfs -o rw,root=minny.yourdomain.com -d “NFS ShareMe” /shareme

You can run the share command on micky to check to make sure it is shared:

# share
- /shareme ro=minny.yourdomain.com “NFS ShareMe”

If it’s not shared, run shareall to share it:

# shareall

Now, jump on over to minny and add the following line to /etc/auto_master:

/- auto_direct

Automountd will now look in /etc/auto_direct for direct mount points.

Next edit /etc/auto_direct and add the following line:

/micky-shareme micky:/shareme

Now, create the directory for the NFS mount point on minny:

# mkdir /micky-shareme

Finally, run the auromount command on minny to inform the daemon of the changes:

# automount

That should do it… Have fun with your new automount NFS share.

More information on this can be found here

Solaris Systems With Multiple Names Have Trouble Mounting CD’s

If you have a Solaris box with multiple names, you might have trouble mounting CD’s if the primary name is different from the one you gave the machine at install time. This is because of the CRAZY way Solaris goes about auto-mounting its optical media.

Here is a 10,000 foot view of how Solaris automounts a CD:

The disk is inserted
vold checks and sees that there is a disk inserted
vold connects to inetd, which in turn, starts smserverd
smserverd mounts the disk and all is right with the world

All this depends, however, that:

1) Vold is running
# /etc/init.d/volmgt start

2) This line is not commented in /etc/inetd.conf
100155/1 tli rpc/ticotsord wait root /usr/lib/smedia/rpc.smserverd rpc.smserverd

And 3) The current hostname of the machine is listed in /etc/net/ticotsord/hosts.

This file should looks something like this:

——-SNIP——
#ident “@(#)hosts 1.2 92/07/14 SMI” /* SVr4.0 1.2 */
# RPC Hosts
micky micky
minny minny
——/SNIP——

Micky and minny, of course are the hostnames you have given the machine. You can pretty much just make it mirror the /etc/hosts file.

Once these three criteria are met, a machine with multiple names should be able to automount a CD with no problems.

Rebuilding the Solaris Device Tree

If you ever shift around any bootable drives within a Sun Solaris box, you may find that either the device names (cxtxd0sx) do not follow the disk position within the server, or, the system just fails to boot because it can’t mount the other disk slices.

Let’s assume you are booting off of target 8 (c1t8d0s0), but wish to move that disk to the appropriate slot to make it target 0 (c1t0d0s0). You have changed all references in the /etc/vfstab file to reflect the new disk position, physically moved the drive from the target 8 slot to the target 0 slot, and changed the boot-device variable within the OBP to the appropriate disk. You should now be all set to boot from the disk in target 0, right?

Not quite yet.

Solaris creates a device tree with links to all the disks it knows about, and these don’t get rebuilt upon reboot. If you simply tried to boot the disk now in target 0, it would find the kernel, but fail to mount any of the other filesystems, because these device links are still pointing to the disk slices on target 8.

In order to boot off the drive in the new position, you will have to remove these device links and rebuild them. Here is how we do that:

1. Insert a Solaris 8, 9 or 10 cd into the hosts cdrom

2. From the ok prompt, enter boot cdrom -s

ok> boot cdrom -s

3. fsck the boot disk

# fsck -y /dev/rdsk/c1t0d0s0

Remember that your boot disk may differ than the example above. Since in our example above, we have put the disk into the slot for target 0 (c1t0d0), that is what we are using here.

4. Mount the root slice on /mnt

# mount /dev/dsk/c1t0d0s0 /mnt

Note that your root slice may differ than the above example.

5. Move path_to_inst

# mv /mnt/etc/path_to_inst /mnt/etc/PATH_TO_INST_ORIG

6. Remove all old device links

# rm /mnt/dev/rdsk/c* ; rm /mnt/dev/dsk/c* ; rm /mnt/dev/rmt/* ; rm
/mnt/dev/cfg/c*

7. Rebuild path_to_inst and devices

# devfsadm -r /a -p /mnt/etc/path_to_inst

8. Unmount the root slice and reboot

# umount /mnt ; init 6

You should now be able to boot off your old drive in its new slot.

SUNWqlc Drivers Don’t Work With thw QLogic QLA2340 HBA


The Qlogic QLA2340 HBA is a 64-bit, 133MHz PCI-X to 2 Gb Fibre Channel adapter. It’s a cool card, but it doesn’t work with the standard SUNWqlc and SUNWqlcx drivers. You have to remove these driver packages and install the one directly from QLogic in order to get it working.

# pkgrm SUNWqlc
# pkgrm SUNWqlcx

Now, download the QLogic drivers (qla2300.sparc_pkg.Z) HERE. Or HERE if you are not using SPARC Solaris.

Then, it’s just a matter of unzipping them, and installing them using the instructions that come with the drivers. The only bummer is that you won’t be able to use the luxadm commands anymore.

Hopefully this has been helpful. Have fun with your speedy new HBA.

Solaris Virtual Network Interfaces

Sometimes it’s useful to create a virtual network interface on your Solaris box, so that you can associate multiple IP addresses with the same host and not have to go through all the trouble of buying another NIC.

Here’s a quick HOWTO. Let’s assume our network card is eri0, and we want to create a virtual interface called eri0:1

Create the virtual interface:
# ifconfig eri0:1 plumb

Configure the virtual interface:
# ifconfig eri0:1 179.164.83.161 netmask 255.255.255.0 broadcast 179.164.83.255

Check to make sure it worked:

# ifconfig -a

lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
eri0: flags=1000843
mtu 1500 index 2
inet 179.164.83.160 netmask ffffff00 broadcast 179.164.83.255
ether 0:3:ba:9:63:9b
eri0:1: flags=1000842 mtu 1500 index 2
inet 179.164.83.161 netmask ffffff00 broadcast 179.164.83.255

Finally bring up your new virtual interface:
# ifconfig eri0:1 up

To make it come up on start:

create /etc/hostname.eri0:1 with hostname in it
make sure the hostname is in /etc/hosts

NOTE: The IP addresses in this story have been changed to protect the innocent.

libnnz9.so: symbol nzdsi_initialize: referenced symbol not found

So you’re running the Oracle Application Server on Solaris 8, and you’ve decided to be a good systems administrator and install the latest 8_Recommended patch cluster from SunSolve. Well, I’m very sorry to inform you that you have just broken your application server. After bouncing the system, you will most likely get the following error as OAS tries to start:

ld.so.1: opmn: fatal: relocation error: file /u01/app/oracle/product/oas_midtier_9.0.4/lib/libnnz9.so: symbol nzdsi_initialize: referenced symbol not found
opmnctl: opmn start failed

If you’re anything like me, you found this during your scheduled downtime, it’s insanely early in the morning, and you really just want to go back to bed. Well, hopefully this will help.

The first thing you want to do is (optionally) make sure you are preloading your libraries:

LD_PRELOAD=$ORACLE_HOME/lib/libclntsh.so; export LD_PRELOAD

It won’t fix the problem, but it is a good idea, and it has the added advantage of making the error go away. The server still won’t start, but the error will go away and that has to be progress, right?

To really resolve the problem, however, you have to backout SunOS 5.8 linker patch 109147-39

patchrm 109147-39

Sun Document ID: 101995 identifies the problem.

On systems with certain patches installed (109147-39 on Solaris 8), applications that reference large numbers of “delayed” loadable shared objects (where the shared objects load slowly and do not explicitly define their required dependencies), may experience start-up failure or may terminate due to an error of the runtime linker.

Sun has recognized the problem, but according to them, a final resolution is pending completion.

Solaris X86 Compatible RAID Controller

Every time I have to spec a solution using Solaris, I always have to answer a bunch of questions in meetings about why Sun is so costly compared to Dell servers. Usually the reason for the higher price is not the servers (especially with X86 sun), but rather the storage. Since Sun does not offer a system with a RAID card, you always have to purchase a high-end disk enclosure that is capable of performing the RAID functions unless you want the performance degradation that comes with software RAID.

The good news is that there is finally a really nice PCI RAID card that works with Solaris! The bad news is that it only works with X86 Solaris, and Sun only goes so far as to say that it is”reported to work“.

Anyhow, no matter. Here is the deal:

According to Sun Big Admin, the Mylex Accelaraid 150 is reported to work with Solaris 9 04/04 to Solaris 10 03/05 (read Solaris 9 and 10 X86). The firmware and bios on the card needs to be: BIOS Version 4.10-50; Firmware 4.08-37.

Pity that there still does not seem to be a RAID controller that works with SPARC hardware. If someone would come up with that, it would make my life as a Solaris administrator a whole lot easier.

Next Page »