RMAN 10G NFS Mount Options
We backup our Oracle databases using RMAN and then write the backup pieces out to an NFS share. This has always worked well, but RMAN started complaining that the NFS share was not mounted with the correct options when we upgraded to Oracle 10G. After some poking around in the docs I finally came up with a set of mount options that work.
Vfstab entry on a Solaria 8 box:
nfsserver.domain.com:/path/to/remote/mountpoint /local-mountpoint nfs 0 yes rw,bg,intr,hard,timeo=600,wsize=32768,rsize=32768
Manual mount on a Solaris 8 box:
mount -o rw,bg,intr,hard,timeo=600,wsize=32768,rsize=32768 nfsserver.domain.com:/path/to/remote/mountpoint /local-mountpoint
According to the docs, the options on a Linux box are pretty much the same, except you would add the following:
nfsver=3,tcp
RHEL System Configuration Changes for Oracle 10G
Below is a list of RHEL system configuration changes that Oracle 10G requires before it is installed.
First, check the following kernel parameters using the commands below:
/sbin/sysctl -a | grep kernel.shmall
/sbin/sysctl -a | grep kernel.shmmax
/sbin/sysctl -a | grep kernel.shmmni
/sbin/sysctl -a | grep kernel.sem
/sbin/sysctl -a | grep fs.file-max
/sbin/sysctl -a | grep net.ipv4.ip_local_port_range
/sbin/sysctl -a | grep net.core.rmem_default
/sbin/sysctl -a | grep net.core.rmem_max
/sbin/sysctl -a | grep net.core.wmem_default
/sbin/sysctl -a | grep net.core.wmem_max
If any parameters are lower than the examples below, you will have to increase them by editing “/etc/sysctl.conf” file, adding the appropriate lines as expressed below. If the current value is higher, leave it as is.
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
Next, edit your “/etc/security/limits.conf” file, adding the following lines:
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
If your current “/etc/pam.d/login” file does not already contain the following line, add it:
Finally, add the following lines to your “/etc/profile” file:
session required pam_limits.so
#Tweaks for Oracle
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
These are just the basic steps I take. See the “Oracle Database Installation Guide” for more complete instructions.
Why Modern RAID 5 is Ideal for Oracle Databases
There is a convention of thought amongst Oracle DBA’s that databases should never be installed on disks that are configured into a RAID 5 array. The argument goes, that since Oracle accesses and writes to random points within relatively large files, the overhead of constantly calculating block-level parity on these files is substantial, resulting in serious performance degradation. They suggest that RAID 1 (mirroring) is the ideal disk configuration since no parity needs to be calculated, and Oracle is more than happy to divide up its database over many smaller mount points.
This way of thinking has largely been correct over the years because most systems have traditionally used software RAID. This means that the CPU of the server itself had the job of doing all those parity calculations, and it really did slow down both the server and the disk when RAID 5 configurations were used. Oracle, in particular, had a hard time with these configurations for the exact reasons the DBA’s point to.
In many cases, software RAID is still used, and to be sure, it is wholly inappropriate to deploy RAID 5 in these environments. However, it is increasingly common to find IT departments using a SAN-type architecture where the RAID type and configuration are invisible to the host operating system. In these environments, the the disk array has a dedicated controller that is singly tasked with handling all read, write, and parity operations. The RAID controller is no longer software running on a generic CPU, but rather firmware that is optimized to handle parity calculations. This results in a system where parity is calculated so quickly by the dedicated controller that differences in speed between RAID 1 and Raid 5 should be virtually nonexistent.
To prove this, I carved up our new InfoTrend EonStor A12F-G2221 into three arrays - a RAID 5, a RAID 1, and a RAID 10. I then set out to run some benchmarks on these different arrays to see what, if any, the differences would be.
The hardware used was as follows:
- Dell OptiPlex GX260, 2.2 GHz Processor, 256 MB Ram
- RHEL 4 Linux
- QLogic QLA2340 HBA
- InfoTrend EonStor A12F-G2221 with 1GB cache
- The RAID 5 LUN consisted of 4 drives
- The RAID 1 LUN consisted of 2 drives
- The RAID 10 LUN consisted of 4 drives
I then identified the iozone tests that most accurately simulated Oracle disk activity. What I really wanted to do was to simulate select and update queries on various sized files and see how the different RAID types held up under the load. To do this, I ran iozone, a well-respected benchmark utility, with the following arguments:
/opt/iozone/bin/iozone -Ra -g 2G -b /home/sysop/new/raid5-2G-1.wks
This put the disk through its paces, as it ran the iozone tests in automatic mode on a 2 Gb file, but in the end, I was interested in analyzing the following tests because they were the ones our DBA team suggested would most closely represent database activity.
Random Read (select queries)
This test measures the performance of reading a file with accesses being made to random locations within the file. The performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.
Random Write (update queries)
This test measures the performance of writing a file with accesses being made to random locations within the file. Again the performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.
Strided Read (more complex select queries)
This test measures the performance of reading a file with a strided access behavior. An example would be: Read at offset zero for a length of 4 Kbytes, then seek 200 Kbytes, and then read for a length of 4 Kbytes, then seek 200 Kbytes and so on. Here the pattern is to read 4 Kbytes and then %%[Page: 3]%%
I ran several instances of the same tests using the same command line to ensure that there were no anomalies, and the machine was doing nothing else during the tests besides running the host OS. The results were pretty much as I expected, and I found little to no variation between the raid types on this disk subsystem.
Random Read Tests:
In this test, there seems to be the slightest advantage to the mirror-type RAID arrays when it comes to very small files. This, I suspect can be attributed to actual drive head latency as, in RAID 5 volumes, the correct block needs to be found on a larger number of disks. This advantage quickly falls off, however as the file size grows, meaning that this slight advantage would not be seen in an Oracle database.
Random Write Tests:
In this test, both RAID 5 and RAID 10 seem to hold a slight advantage over the direct mirror. This, I would imagine can be attributed to the fact that the writes are happening over a larger number of spindles. This indicates that the controller is calculating the parity faster than the 2Gb connection speed to the disk subsystem. Again, the variation is incredibly small, so there is no arguable performance advantage to using one type of RAID over another when using a hardware controller.
Stride Read Tests:
Here again we see no real advantage to one RAID type over any other. It could be said that the RAID 10 volume held up ever-so-slightly better on this test, but any edge is so slight that it would be hard to imagine how this could translate into a noticeable performance gain in an Oracle database.
In the end, these tests proved my suspicion that hardware RAID controllers have become so efficient and fast that it no longer makes any real difference what type of RAID you decide to use for your Oracle database. Largely gone are the days when your disk space and RAID volumes were inexorably tied to the server itself. So long as you are using hardware RAID, and the LUNS are abstracted from your operating system, you can largely feel free to make the most of your storage dollar by using RAID 5 in your production database environments.
Picking a Multiuser Blogging System
I’m a blogger. I’m also a systems administrator at a University. Thus, when the University decided to charter a project to offer each member of the institution a blog, I was selected to sit on the committee. We boiled all of the software choices down to Drupal, Movable Type, and Wordpress MU.
In my evaluation of these solutions, the software was ranked on a five-point scale against the following requirements, which we had decided were important to us. I’ve also included my personal rankings. In many cases the software received a lower ranking on a feature because it was not customizable by the individual user.
| Intuitive Drupal:2 Movable Type:5 Wrodpress MU:5 |
Skinnable Drupal:2 Movable Type:5 Wrodpress MU:5 |
Pingbacks Drupal:3 Movable Type:5 Wrodpress MU:5 |
| Trackbacks Drupal:2 Movable Type:5 Wrodpress MU:5 |
Supports XMLRPC Drupal:5 Movable Type:5 Wrodpress MU:5 |
Comment SPAM blocking Drupal:2 Movable Type:4 Wrodpress MU:5 |
| Image support Drupal:5 Movable Type:5 Wrodpress MU:5 |
WYSIWYG Drupal:5 Movable Type:5 Wrodpress MU:5 |
RSS feed Support Drupal:3 Movable Type:4 Wrodpress MU:5 |
| TAGS Drupal:5 Movable Type:5 Wrodpress MU:5 |
URL Rewriting Drupal:5 Movable Type:5 Wrodpress MU:5 |
Hosted here Drupal:5 Movable Type:5 Wrodpress MU:5 |
| Single installation Drupal:5 Movable Type:5 Wrodpress MU:5 |
Single Database Drupal:5 Movable Type:5 Wrodpress MU:5 |
Single Table structure Drupal:5 Movable Type:4 Wrodpress MU:0 |
| Mysql or Oracle Drupal:5 Movable Type:5 Wrodpress MU:5 |
Integrate with IDM (ex. CAS) Drupal:0 Movable Type:0 Wrodpress MU:0 |
Quota on file uploads Drupal:5 Movable Type:0 Wrodpress MU:5 |
| Prefer PHP: Drupal:5 Movable Type:1 Wrodpress MU:5 |
Ease of Upgrade Drupal:4 Movable Type:4 Wrodpress MU:5 |
Cost Drupal:5 Movable Type:3 Wrodpress MU:5 |
Totals:
Drupal: 88
Movable Type: 90
Wordpress MU: 100
Drupal came in last not because it is unable to meet the requirements we had listed, but because it is not as intuitive as the other solutions, and there are many areas where the user can not be granted any granularity of control over the blog. There is, for example, no way to allow the individual user to manage comment spam settings. The administrator chooses the settings that are then applied globally to each user on the system.
Movable Type evaluated quite well. The user interface is very nice, and the entire software package is very feature rich from a user experience standpoint. It meets most of our pre-determined criteria, but there are no built in “file upload quotas”, and the application is written in Perl. Also of possible concern is the fact that, while content exists within the database, it is delivered via static pages. This means that a site (all static pages) must be rebuilt whenever significant changes are made, such as changing a theme. It is anyone’s guess how this static / dynamic model will scale to a very large number of users. Also of note is the fact the MT is a pay for product. Like any proprietary software solution, this can be both a benefit and a drawback.
Wordpress MU mets or exceeds all of our pre-determined criteria with one major exception. The database does not have a unified table structure, meaning each blog requires its own set of database tables. It is largely unknown how this model will scale to a large number of users, however wordpress.com has this software successfully rolled out to a large and growing number of users. Should we decided on WPMU, it may be worth discussing the implementation with the WPMU team. The user interface is both elegant and intuitive, and the user retains maximum control over most of the unique blog settings. It is, however, worth mentioning that WPMU has adopted the Web 2.0 standard of not versioning their releases. Upgrades are downloaded via a nightly builds directory, which means that version releases may not be as well tested as with a more traditional release strategy.
For my part, even though I scored it slightly higher, I have no real preference for Wordpress MU over Movable Type. Both are great blogging solutions, and I feel that we could find a way around MT’s lack of file upload quota. I do, however, prefer either of these two solutions over Drupal because of issues surrounding usability, and customization.
In the end, the committee agreed that Wordpress MU was the winner. We will now go on to write the CAS authentication module, and see if we can get around the requirement that usernames have greater than three letters. We are also hoping to get in touch with the wordpress.com people to discuss how the non-unified table structure will scale to 30.000+ users.
libnnz9.so: symbol nzdsi_initialize: referenced symbol not found
So you’re running the Oracle Application Server on Solaris 8, and you’ve decided to be a good systems administrator and install the latest 8_Recommended patch cluster from SunSolve. Well, I’m very sorry to inform you that you have just broken your application server. After bouncing the system, you will most likely get the following error as OAS tries to start:
ld.so.1: opmn: fatal: relocation error: file /u01/app/oracle/product/oas_midtier_9.0.4/lib/libnnz9.so: symbol nzdsi_initialize: referenced symbol not found
opmnctl: opmn start failed
If you’re anything like me, you found this during your scheduled downtime, it’s insanely early in the morning, and you really just want to go back to bed. Well, hopefully this will help.
The first thing you want to do is (optionally) make sure you are preloading your libraries:
LD_PRELOAD=$ORACLE_HOME/lib/libclntsh.so; export LD_PRELOAD
It won’t fix the problem, but it is a good idea, and it has the added advantage of making the error go away. The server still won’t start, but the error will go away and that has to be progress, right?
To really resolve the problem, however, you have to backout SunOS 5.8 linker patch 109147-39
patchrm 109147-39
Sun Document ID: 101995 identifies the problem.
On systems with certain patches installed (109147-39 on Solaris 8), applications that reference large numbers of “delayed” loadable shared objects (where the shared objects load slowly and do not explicitly define their required dependencies), may experience start-up failure or may terminate due to an error of the runtime linker.
Sun has recognized the problem, but according to them, a final resolution is pending completion.
How To Install Oracle 10g on RedHat Enterprise 3
So you’ve got Oracle 10G and you want to install it on your RedHat Enterprise 3 server. Well, since Oracle can’t manage to create tar files like everyone else in the world, you have to find a way of dealing with the .cpio they send you. Here’s how to get it extracted:
cpio -idmv < /path/to/ship-version.cpio
This extracts everything nicely into a Disk1 directory.
Now, before flying off and running the installer, you have a couple of things to do first. To start, you have to tweak your kernel a bit. There are a number of ways to do this, but I like to use the /etc/sysctl.conf file.
Edit /etc/sysctl.conf and add the following lines:
kernel.shmall = 134217728
kernel.shmmax = 2147483648
kernel.semopn = 100
semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
Next you have to add an oracle user and a dba group. Run the following commands as root:
groupadd dba
useradd -d /path/to/oracle/user/directory -g dba -c ‘Oracle User’ -s /path/to/fovorite/shell oracle
chown oracle:dba /path/to/oracle/user/directory
passwd oracle (set new password)
Add the following environmental settings to your oracle user’s .bashrc file. Feel free to change them if you are using a C-Type shell.
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.1.0/Db_1; export ORACLE_HOME
ORACLE_SID=YOUR_SID; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
That should just about do it. Restart the system, log in as the oracle user and run the oracle installer (/path/to/Disk1/runInstaller). Check to make sure that all the settings from your .bashrc file are picked up by the oracle installer and have fun.
In some cases, the installer may complain about not having the required packages. If it does this, make sure that the following packages are installed:
setarch-1.3-1.i386.rpm
openmotif-2.2.2-16.i386.rpm
compat-libstdc++-7.3-2.96.122.i386.rpm
compat-libstdc++-devel-7.3-2.96.122.i386.rpm
compat-db-4.0.14-5.i386.rpm
compat-gcc-7.3-2.96.122.i386.rpm
compat-gcc-c++-7.3-2.96.122.i386.rpm
Oracle required changes to the Solaris /etc/system file
Ever needed to install Oracle on Solaris, but can’t seem to find all that good stuff you have to add to the /etc/system file? I have, so here they are!
set maxusers 32
set shmsys:shminfo_shmmax=4294967295
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=512
set shmsys:shminfo_shmseg=128
set semsys:seminfo_semmni=512
set semsys:seminfo_semmsl=512
set semsys:seminfo_semmns=1024
set semsys:seminfo_semopm=256
set semsys:seminfo_semvmx=65534
