Recovering From a Corrupt NetInfo Database on OSX.4

I managed to corrupt my NetInfo database on an OS 10.4 server a few weeks ago by not cleanly unmounting the drive after booting from DVD and resetting the admin password. Long story short, this left me with no users on the system at all. With no users, I could not log in to create one, so I had to blow away the NetInfo database and restore it to factory defaults. This should only be done when you only have a small number of users, and don’t mind having to re-create them. Only the user account information is deleted, and the user directory is retained, but you will have to manually add any users you may have back into the system through the GUI, making sure that the new “user” references the old “user’s” account directory.

If you have more than just one or two users, you should use the procedure to recover from one of your NetInfo database backups. A backup of your this database is made at 3:15 every day so long as the computer is running. It is stored in “/var/backups/”, and here are some instructions on how to recover it from it. If, however, you don’t care about re-adding users, and simply want to get into your system quickly, or if you don’t have a backup to restore from, here is how you can do it:

BEWARE: THIS WILL COMPLETELY ERASE ALL USER ACCOUNT INFORMATION FROM THE SYSTEM!!! You are warned.

1) Start by booting your Mac into single user mode. To do this, hold down both the “Apple” and the “s” keys as the system boots.

2) The system will have mounted the “/” filesystem read-only to protect against data loss. To get “/” mounted read-write, we have to run two commands:

# /sbin/fsck -fy

# /sbin/mount -uw /

3) Now “/” is mounted read-write, so we can start with the real work. First, rename your existing NetInfo database to something else so the OS will not see it on the way up:

# mv /var/db/netinfo/local.nidb /var/db/netinfo/local.nidb.bad

# mv /var/db/netinfo/network.nidb /var/db/netinfo/network.nidb.bad

4) Next, remove the “.AppleSetupDone” file so the OS will kick you back into the installer upon boot and you can recreate your users.

# rm /var/db/.AppleSetupDone

5) Finally, reboot your system and recreate your users, making sure they are pointed towards their existing account directories.

# reboot

IFCONFIG Does Not give You Link Status; ETHTOOL Does

For some reason that is a complete mystery to me, RHEL does not give you the link status when you run # ifconfig -a. This makes it incredibly hard to debug link integrity issues! Buried amongst all of Red Hat’s proprietary commands, however, is a utility called ethtool, which does give you the status of your link.

Since ethtool is used for querying settings of an ethernet device and changing them, it does a lot more than just give link status. Amongst other things, you can use it to turn on or off autonegotiation on your network card. Run # /sbin/ethtool -h for full usage.

Here’s how you use it to see if your server has link:

# /sbin/ethtool eth0

You should see something like this:

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes

Getting ntpd to work correctly on RHEL

When many new servers are delivered from the factory, the system clock is way off. Most UNIX systems run “ntpd” to keep the time in sync with internet time servers, which are, in turn synchronized against an atomic clock. This results in a system time that is very very close to the “actual” time of day. The downside, however, is that even a properly configured “ntpd” will not synchronize the system clock if it is too far out of sync with the time server. To remedy this, we first have to run “ntpdate” to get the system clock close to the correct time, and then enable “ntpd” to keep it there.

The first thing we have to do is “ntpd” to free up the port for “ntpdate”:

[root@server /]# /sbin/service ntpd stop
Shutting down ntpd:                                        [  OK  ]

This frees up the port for ntpdate. Next we run:

[root@server /]# /usr/sbin/ntpdate time.apple.com

Now the time should be set correctly. We then change the default time servers to something like the following in /etc/ntp.conf:

# --- OUR TIMESERVERS -----
time-a.timefreq.bldrdoc.gov
time-b.timefreq.bldrdoc.gov
time-c.timefreq.bldrdoc.gov

We can use any time server we want, but I like these and find them to be reliable.

Finally, start backup up your “ntpd” service, and your all set to go.

[root@server /]# /sbin/service ntpd start
Starting ntpd:                                        [  OK  ]

Remember to use “chkconfig” to make sure “ntpd” is enabled to come up when the system starts.

Controlling Services With chkconfig

Many system 5 UNIX variants use scripts in the /etc/rcN.d/ directories to control which services should be started in the various runlevels. If, for instance, you wanted the secure shell daemon to run in runlevel 4, you would put a script named something like “S55sshd” in “/etc/rc4.d”. This script would usually accept the “start” “stop” and “restart” arguments, as well as the commands to perform these functions. When the system came up, it would execute “/etc/rc4.d/S55sshd start” when it transitioned into runlevel 4. On the way down, it would execute “/etc/rc4.d/S55sshd stop” as the system passed from runlevel 4 to runlevel 3. If you had made some changes to the sshd configuration file, and wanted to restart the service, you could manually execute “/etc/rc3.d/sshd restart” to kill and then restart the daemon.

Since this model involved having multiple copies of the same script in many different directories, Linux and others have adopted the standard of putting all service control scripts in “/etc/init.d/”, and using symbolic links to these scripts in the various “/etc/rcN.d/” directories. This allowed for the SGI IRIX innovation of the “chkconfig” command, which is command line tool that manages the symbolic links for you.

How to use “chkconfig” in Red Hat Enterprise Linux:

First, all your service control scripts need to be in the “/etc/init.d/” directory. They should reflect the name of the service they control. In our example, the file is named /etc/init.d/sshd”.

Secondly, they have a tag at the head of the script that looks something like this so that “chkconfig” understands that it can controll it:

# Basic support for IRIX style chkconfig
###
# chkconfig: 2345 55 25
# description: Manages the services you are controlling with the chkconfig command
###

The first set of numbers “2345″ is are the default runlevels for the service, and “55″ and “25″ represent the name of the “S” and “K” symbolic links, and the order in which the service will be started and stopped in the respective runlevel. You will need to change these last two numbers, making them unique.

Once these requirements are met, using the command is fairly simple. When we go into /etc/rc3.d, we see a file called “S55sshd”.

[root@calvin rc2.d]# cd /etc/rc3.d
[root@calvin rc2.d]# ls -al S55sshd
lrwxrwxrwx 1 root root 14 Nov 15 15:10 S55sshd -> ../init.d/sshd

We see this file is a symbolic link to “../init.d/sshd”. Let’s run the “chkconfig” command to turn sshd off.

[root@calvin init.d]# /sbin/chkconfig sshd off
[root@calvin init.d]# /sbin/chkconfig --list sshd
sshd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

chkconfig --list sshd confirms that sshd has been disabled in all runlevels, and the symbolic link has been removed from all “/etc/rcN.d/” directories.

Let’s turn sshd back on:

[root@calvin init.d]# /sbin/chkconfig sshd on
[root@calvin rc2.d]# /sbin/chkconfig --list sshd
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

chkconfig --list sshd confirms that sshd has now been enabled in runlevels 2, 3, 4 and 5, and we see s symbolic link to “/etc/init.d/sshd” named “S55sshd” in “/etc/rc2.d/”, “/etc/rc3.d/”, “/etc/rc4.d/” and “/etc/rc5.d/”.

Let’s imagine now that we only want sshd to be enabled in runlevel 5. We run the following command to accomplish this:

[root@calvin rc2.d]# /sbin/chkconfig sshd --level 234 off
cd /etc/[root@calvin rc2.d]# /sbin/chkconfig --list sshd
sshd 0:off 1:off 2:off 3:off 4:off 5:on 6:off

chkconfig --list sshd confirms that sshd has been disabled in all runlevels except 5, and the “S55sshd” has been removed from “/etc/rc2.d/”, “/etc/rc3.d/” and “/etc/rc4.d/”.

There is, of course, more to it, but this should get you well on your way to happily managing your system services with “chkconfig”.

Nice Clients > Nasty Clients

It occurs to me that when you work in IT, you can easily tell how how nice the person you are working with is by the extent to which he is pleasant to you when things are not going well.

All too often people act as nice as can be when they want something, but as soon as a service they want to use is down, they start spitting fire and calling managers. As if we can somehow resolve issues faster with all the added blood pressure of higher-ups breathing down our necks.

Anyone can be nice when they want something. If you really want to stand out from the crowd, remember to be nice and respectful when you are waiting for your IT department to resolve a problem. Outages are stressful enough for your IT guy without you making his life harder. Remember that he wants the service back up as much, or more than you do.

Changing Linux Mount Points

If you’re familiar with UNIX, you know that changing mount points is really pretty easy. All you have to do is go into “/etc/fstab”, “/etc/vfstab” (or whatever your flavor of UNIX happens to call its filesystem table) and change the mount directory.

If, for instance, you had a Solaris box, and you wanted to make the disk currently mounted as “/data” be mounted as “/database”, all you would have to do is the following:

# umount /data
# mv /data /database
Change this line in “/etc/vfstab” from something like this:
/dev/dsk/c1d0s6 /dev/rdsk/c1d0s6 /data ufs 1 yes -
to something like this:
/dev/dsk/c1d0s6 /dev/rdsk/c1d0s6 /database ufs 1 yes -
and remount it as “/database”.
# mount /database

With Linux, however, it’s not quite so clear anymore… It’s still easy, but it’s just not so clear what you have to do since they have now taken to mounting filesystems using the volume label. Rather than pointing directly to the disk device, Linux points to the label, and “/etc/fstab” look more like this:

LABEL=/data /data ext3 defaults 1 2

You can always simply change the disk label, but if you don’t care, you can just tell linux where the raw device is, bypassing the need to worry about the label. The easiest way to do this is simply to replace the “LABEL=/data” value to the “/dev” entry of the disk itself. Then, simply change “/data” to “/database” and you’re all set.

Here is an example of what you would do to change the mountpoint of “/data” to /database”:

# umount /data
# mv /data /database
Change this line in “/etc/fstab” from this:
LABEL=/data /data ext3 defaults 1 2
to this:
/dev/sda6 /database ext3 defaults 1 2
and remount it as /database
# mount /database

Remembering to change the example values here with those required for your situation.

How To Install Oracle 10g on RedHat Enterprise 3

So you’ve got Oracle 10G and you want to install it on your RedHat Enterprise 3 server. Well, since Oracle can’t manage to create tar files like everyone else in the world, you have to find a way of dealing with the .cpio they send you. Here’s how to get it extracted:

cpio -idmv < /path/to/ship-version.cpio

This extracts everything nicely into a Disk1 directory.

Now, before flying off and running the installer, you have a couple of things to do first. To start, you have to tweak your kernel a bit. There are a number of ways to do this, but I like to use the /etc/sysctl.conf file.

Edit /etc/sysctl.conf and add the following lines:

kernel.shmall = 134217728
kernel.shmmax = 2147483648
kernel.semopn = 100
semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

Next you have to add an oracle user and a dba group. Run the following commands as root:

groupadd dba
useradd -d /path/to/oracle/user/directory -g dba -c ‘Oracle User’ -s /path/to/fovorite/shell oracle
chown oracle:dba /path/to/oracle/user/directory
passwd oracle (set new password)

Add the following environmental settings to your oracle user’s .bashrc file. Feel free to change them if you are using a C-Type shell.

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.1.0/Db_1; export ORACLE_HOME
ORACLE_SID=YOUR_SID; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

That should just about do it. Restart the system, log in as the oracle user and run the oracle installer (/path/to/Disk1/runInstaller). Check to make sure that all the settings from your .bashrc file are picked up by the oracle installer and have fun.

In some cases, the installer may complain about not having the required packages. If it does this, make sure that the following packages are installed:

setarch-1.3-1.i386.rpm
openmotif-2.2.2-16.i386.rpm
compat-libstdc++-7.3-2.96.122.i386.rpm
compat-libstdc++-devel-7.3-2.96.122.i386.rpm
compat-db-4.0.14-5.i386.rpm
compat-gcc-7.3-2.96.122.i386.rpm
compat-gcc-c++-7.3-2.96.122.i386.rpm