Synopsis.
Oracle Database 11g Release 2 makes it much simpler to configure and
incorporate many of the grid computing features that were only available in a
Real Application Clusters (RAC) clustered database environment in previous
releases for a single-instance Oracle database. This article – the first in
this series – will demonstrate how to install and configure a new Oracle 11g
Release 2 (11gR2) Grid Infrastructure home as the basis for the majority of
these grid computing features.
It’s
been a few months since I summarized the incredible array of new features that
Oracle has introduced as part of Oracle Database Release 11gR2, and in that
span of time, I’ve been experimenting with those features as I’ve built a new
infrastructure for experimentation. Among the most intriguing new features is
the consolidation of Automatic Storage
Management (ASM) with Oracle
Clusterware (OC) into a pragmatic and sensible arrangement called
the Oracle Grid Infrastructure (GI). As
I’ll demonstrate in this article, the venerable Oracle Universal Installer (OUI) utility gets a welcome
update in this release, but first I’ll need to perform quite a bit of system
administration work before we can invoke it and explore its new features.
First … A Word About The (Computing) Environment.
I’ve made some long-desired changes to my home office’s personal computing
infrastructure so that I can manage my workload effectively and efficiently
with my favorite virtualization environment, VMWare:
-
I’ve upgraded to Oracle Enterprise Linux (OEL) 5
Update 2 (kernel 2.6.18-92.el5) for my base computing platform, a home-grown
gaming server with 4GB of memory running an AMD Opteron dual-core processor. -
I’ve also finally moved up to VMWare Workstation Version
7.0.0 for all my VMWare endeavors, and though I still occasionally long for the
freedom of VMWare Server 2.0a (as in free!), I’ve found that Workstation is
just as stable and that it works extremely well with OEL as both its host and
guest OS.
Setting Up For Oracle 11gR2 Grid Infrastructure
I’m
going to implement my 11gR2 Grid Infrastructure via a series of Oracle best
practices that I’ve encountered over the years and have gleaned through a
thorough reading of Oracle’s technical documentation. I’ll be using raw disk
partitions for configuring all of the ASM disks that will eventually comprise
the various ASM disk groups needed for my demonstrations.
Creating the Required Raw Partitions. The
Oracle 11gR2 Grid Infrastructure
leverages ASM to store multiple copies of the Oracle
Clusterware Registry (OCR) file, multiple Voting Disks, and of course the ASM disk groups disks
themselves. Since the maximum number of logical partitions that can be created
within any one extended partition is 12, I’ve created two VMWare virtual disks
sized at 18.5 GB and 11.0 GB, respectively. Here’s the output from the terminal
session during which I used the Linux fdisk command to create the
remaining logical partitions:
[root@11gR2Base ~]# fdisk -l /dev/sdeDisk /dev/sde: 19.3 GB, 19327352832 bytes
255 heads, 63 sectors/track, 2349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System
/dev/sde1 1 2349 18868311 5 Extended
/dev/sde5 1 281 2257069+ 83 Linux
/dev/sde6 282 562 2257101 83 Linux
/dev/sde7 563 843 2257101 83 Linux
/dev/sde8 844 1124 2257101 83 Linux
/dev/sde9 1125 1405 2257101 83 Linux
/dev/sde10 1406 1686 2257101 83 Linux
/dev/sde11 1687 1967 2257101 83 Linux
/dev/sde12 1968 2248 2257101 83 Linux[root@11gR2Base ~]# fdisk -l /dev/sdf
Disk /dev/sdf: 12.0 GB, 12079595520 bytes
255 heads, 63 sectors/track, 1468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System
/dev/sdf1 1 1468 11791678+ 5 Extended
/dev/sdf5 1 281 2257069+ 83 Linux
/dev/sdf6 282 562 2257101 83 Linux
/dev/sdf7 563 843 2257101 83 Linux
/dev/sdf8 844 1124 2257101 83 Linux
/dev/sdf9 1125 1405 2257101 83 Linux
Assigning Raw Partitions to Block Device Endpoints.
Oracle has recommended for some time that block devices are a much better
choice for the ASM file system, especially since I’ve occasionally heard rumors
that support for traditional raw devices allocated through the /etc/sysconfig/rawdevices
configuration file may be reduced or disappear in the future.
For
this and all future Oracle 11gR2 features demonstrations, I’ve configured a
special service, losetup, that will construct,
configure and allocate virtual block devices during server startup. For the losetup
script to work properly, however, note that I also needed to increase the
default number of loopback devices from eight to 16; I did this by adding the
following line to the /etc/modprobe.conf system configuration file, and
then rebooting the server to make sure it took effect:
options loop max_loop=16
Listing
1.1 shows the losetup script I used to complete
the assignment of raw partitions to virtual block devices. After I copied the
script to file /etc/init.d/losetup,
I then registered the new service (as the root user) via chkconfig:
#> chmod 775 /etc/init.d/losetup
#> chkconfig losetup –add
#> chkconfig losetup on
#> chkconfig losetup –list
After
rebooting the server, here’s the result of implementing the losetup
script – the successful allocation of block devices as shown below:
[root@11gR2Base ~]# ls -la /dev/xv*
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdb -> /dev/loop1
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdc -> /dev/loop2
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdd -> /dev/loop3
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvde -> /dev/loop4
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdf -> /dev/loop5
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdg -> /dev/loop6
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdh -> /dev/loop7
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdi -> /dev/loop8
lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdj -> /dev/loop9
lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdk -> /dev/loop10
lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdl -> /dev/loop11
lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdm -> /dev/loop12
lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdn -> /dev/loop13
Configuring and Implementing ASMLIB
To
keep my ASM configuration simple to manage, I’ll also use the Oracle ASM disk
management drivers that ASMLIB provides to “stamp” each
target mount point before actually creating ASM disks and disk groups. First,
I’ll confirm that the oracleasm drivers appropriate to my OS kernel
version have indeed been installed:
[root@11gR2Base ~]# rpm -qa | grep oracleasm
oracleasm-2.6.18-92.el5xen-2.0.4-1.el5
oracleasm-2.6.18-92.el5-2.0.4-1.el5
oracleasm-2.6.18-92.el5debug-2.0.4-1.el5
oracleasm-support-2.0.4-1.el5
Excellent!
My system administrator took care of this when she installed Oracle Enterprise
Linux 5 Update2; otherwise, I’d be forced to remind her to download the
appropriate ORACLEASM drivers and then install them on my server. However, it was necessary to make sure that
the connection to the appropriate oracleasm RPMs was available, and
that took a little extra manipulation as shown in the output below:
[root@11gR2Base ~]# /usr/lib/oracleasm/oracleasm_debug_link 2.6.18-92.el5 $(uname -r)
oracleasm_debug_link: Target exists
[root@11gR2Base ~]# ls -l /lib/modules/$(uname -r)/kernel/drivers/addon/oracleasm
total 576
-rw-r–r– 1 root root 579514 May 23 2008 oracleasm.ko[root@11gR2Base ~]# /etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets (‘[]’). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Loading module “oracleasm”: [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ][root@11gR2Base ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: [ OK ]
Checking if /dev/oracleasm is mounted: [ OK ]
“Stamping” Candidate Disks With ASMLIB.
Now that ASMLIB is configured properly, it’s time to apply ASMLIB “stamps” to
each virtual device via the createdisk command as shown below. This makes it
much simpler to configure and manage ASM disks without having to use complex
mount point naming conventions:
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK1 /dev/xvdb
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK2 /dev/xvdc
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK3 /dev/xvdd
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK4 /dev/xvde
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK5 /dev/xvdf
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK6 /dev/xvdg
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK7 /dev/xvdh
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK8 /dev/xvdi
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK1 /dev/xvdj
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK2 /dev/xvdk
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK3 /dev/xvdl
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK4 /dev/xvdm
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK5 /dev/xvdn
Finally,
I’ll invoke ASMLIB’s listdisks command to confirm that
all disks have been correctly “stamped” and are now ready for use in concert
with my upcoming Grid Infrastructure installation:
[root@11gR2Base ~]# /etc/init.d/oracleasm listdisks
ACFDISK1
ACFDISK2
ACFDISK3
ACFDISK4
ACFDISK5
ASMDISK1
ASMDISK2
ASMDISK3
ASMDISK4
ASMDISK6
ASMDISK7
ASMDISK8
