March 25, 2010 Synopsis. As its integration of grid computing features that had only been available in a Real Application Clusters (RAC) clustered database environment, Oracle Database 11g Release 2 (11gR2) integrates a new ASM-based file system – the ASM Clustered File System – that offers the ability to store files other than database specific files like online redo logs, control files, and datafiles. This article – the next in this series - will demonstrate how to install and configure a new Oracle 11g Release 2 (11gR2) Grid Infrastructure home as the basis for the majority of these grid computing features. Oracle Database 11gR2: Installing Grid Infrastructure explained how to:
Continuing along this same course, I’ll next illustrate how to:
Installing ACFS: PreparationNow that I’ve got an Automatic Storage Management (ASM) instance configured, I’ll turn my attention to setting up a new ASM disk group that uses 11gR2’s new ASM Cluster File System (ACFS) as its underlying file system. First, I’ll verify whether my ASM instance has been successfully restarted after I rebooted my host server: [oracle@11gR2Base ~]$ ps -ef | grep +ASM oracle 13568 1 0 11:30 ? 00:00:00 asm_pmon_+ASM oracle 13570 1 0 11:30 ? 00:00:00 asm_vktm_+ASM oracle 13574 1 0 11:30 ? 00:00:00 asm_gen0_+ASM oracle 13576 1 0 11:30 ? 00:00:00 asm_diag_+ASM oracle 13578 1 0 11:30 ? 00:00:00 asm_psp0_+ASM oracle 13580 1 0 11:30 ? 00:00:06 asm_dia0_+ASM oracle 13582 1 0 11:30 ? 00:00:00 asm_mman_+ASM oracle 13584 1 0 11:30 ? 00:00:00 asm_dbw0_+ASM oracle 13586 1 0 11:30 ? 00:00:00 asm_lgwr_+ASM oracle 13588 1 0 11:30 ? 00:00:00 asm_ckpt_+ASM oracle 13590 1 0 11:30 ? 00:00:00 asm_smon_+ASM oracle 13592 1 0 11:30 ? 00:00:00 asm_rbal_+ASM oracle 13594 1 0 11:30 ? 00:00:02 asm_gmon_+ASM oracle 13596 1 0 11:30 ? 00:00:00 asm_mmon_+ASM oracle 13598 1 0 11:30 ? 00:00:00 asm_mmnl_+ASM oracle 14318 14217 0 17:09 pts/2 00:00:00 grep +ASM [oracle@11gR2Base ~]$ . oraenv ORACLE_SID = [orcl] ? +ASM The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid is /u01/app/oracle [oracle@11gR2Base ~]$ srvctl status asm ASM is running on 11gr2base Excellent news! My ASM instance is running after the server was rebooted, and this means that the setup for Oracle 11gR2 High Availability Services (HAS) is still intact. If you’ve worked with Oracle Clusterware (OC) and Real Application Cluster (RAC) databases in prior Oracle releases, by the way, you’ll notice that I used the venerable srvctl OC command to confirm the status of my ASM instance. This illustrates that Oracle Clusterware is indeed imbedded within the Grid Infrastructure components of Oracle 11gR2. (I’ll be covering a range of new srvctl commands in my next article as we explore how an Oracle single-instance database is managed within the 11gR2 Grid Infrastructure.) Configuring ACFS For Server Reboots. Before I can proceed with the creation of an ACFS-based Oracle Home, I need to verify that the corresponding ACFS module and its service, oracleacfs, is also running. I’ll connect as the root user and then issue the lsmod command to search for that service:
#> lsmod | grep oracle
oracleasm 81576 1
Unfortunately, neither oracleacfs nor its counterpart oracleadvm service that Oracle uses to manage ASM Dynamic Volume Management (ADVM), is loaded right now because I haven’t yet implemented the proper acfsload startup script. Much like the losetup script in the prior article, I’ve built a new script, /etc/init.d/acfsload, as shown below, and I’ll activate this script as a service for automatic reboot using the Linux chkconfig command: #> view /etc/init.d/acfsload #!/bin/sh # chkconfig: 2345 30 31 # description: Automatically load Oracle 11gR2 ACFS drivers during system reboot /u01/app/oracle/product/11.2.0/grid/bin/acfsload start -s #> chmod 775 acfsload #> chkconfig --add acfsload #> chkconfig --list acfsload acfsload 0:off 1:off 2:on 3:on 4:on 5:on 6:off Once I constructed and then executed this script manually, I can confirm that the proper ACFS and ADVM modules are now loaded: #> lsmod | grep oracle oracleacfs 877320 0 oracleadvm 221760 0 oracleoks 276880 2 oracleacfs,oracleadvm oracleasm 81576 1 Installing ACFS: Creating a New ASM Disk GroupMy next step is to configure ACFS, and to do that I’ll first build a new ASM disk group just to contain this new file system. Instead of using the asmca utility to build this disk group in interactive mode as I did in my prior article, however, this time I’ll invoke it in command line mode. I can view a brief listing of all the things that asmca can perform in this mode by invoking the asmca –help command from the same terminal window, as shown in Listing 2.1. To build a new disk group named ACFS, I’ll invoke asmca with the –creatediskGroup directive. $> asmca -silent -createDiskGroup -diskString '/dev/oracleasm/disks/*' -diskGroupName ACFS -diskList /dev/oracleasm/disks/ACFDISK1 /dev/oracleasm/disks/ACFDISK2, /dev/oracleasm/disks/ACFDISK3 -redundancy EXTERNAL -au_size 1 -sysAsmPassword oracle And here’s the resulting entries from the +ASM instance’s alert log: Sun Mar 07 22:46:59 2010 NOTE: updated gpnp profile ASM diskstring: /dev/oracleasm/disks/* ALTER SYSTEM SET asm_diskstring='/dev/oracleasm/disks/*' SCOPE=BOTH SID='*'; SQL> CREATE DISKGROUP ACFS EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/ACFDISK1', '/dev/oracleasm/disks/ACFDISK2', '/dev/oracleasm/disks/ACFDISK3' ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */ NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/ACFDISK1) NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/ACFDISK2) NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/ACFDISK3) NOTE: initializing header on grp 2 disk ACFS_0000 NOTE: initializing header on grp 2 disk ACFS_0001 NOTE: initializing header on grp 2 disk ACFS_0002 NOTE: initiating PST update: grp = 2 kfdp_update(): 13 Sun Mar 07 22:47:00 2010 kfdp_updateBg(): 13 NOTE: group ACFS: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 2 completed successfully NOTE: cache registered group ACFS number=2 incarn=0x17288ec8 NOTE: cache began mount (first) of group ACFS number=2 incarn=0x17288ec8 NOTE: cache opening disk 0 of grp 2: ACFS_0000 path:/dev/oracleasm/disks/ACFDISK1 NOTE: cache opening disk 1 of grp 2: ACFS_0001 path:/dev/oracleasm/disks/ACFDISK2 NOTE: cache opening disk 2 of grp 2: ACFS_0002 path:/dev/oracleasm/disks/ACFDISK3 < . . . some lines removed for sake of brevity . . .> NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2 SUCCESS: diskgroup ACFS was mounted SUCCESS: CREATE DISKGROUP ACFS EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/ACFDISK1', '/dev/oracleasm/disks/ACFDISK2', '/dev/oracleasm/disks/ACFDISK3' ATTRIBUTE 'compatible.asm'='11.2.0.0.0', 'au_size'='1M' /* ASMCA */ Sun Mar 07 22:47:09 2010 NOTE: diskgroup resource ora.ACFS.dg is online I’ll use asmca to confirm that creation of the new ACFS ASM disk group has been successful and to modify any necessary ASM initialization parameters. As Figure 2.1 shows, ASM disk group ACFS was successfully created:
However, I also need to verify that the proper compatibility settings for the ACFS disk group are in place:
Figure 2.2. Verifying Compatibility Settings Excellent! These settings are correct. So now, I’ll create a new mount point within ACFS that will eventually host my new Oracle 11gR2 database home, as shown below:
Figure 2.3. Creating a New Oracle Database Home Within an ACFS Volume For the location of the new ACFS mount point, I’ll choose the new ACFS ASM disk group as the target for the ACFS volume created above …
... and the end result is the successful volume assignment as shown in Figure 2.5 below.
Managing ACFS: The ACFSUTIL UtilityI can verify the status of any ACFS-controlled mount point with the acfsutil command-line utility. For example, the acfsutil info fs command shows me the current state of the existing ACFS file system, including where and how it’s been mounted, its current availability, how much free disk space is available, whether any software redundancy is in place, and even the stripe size used: #> acfsutil info fs /u01/app/oracle/acsfmounts/acfs_db1 ACFS Version: 11.2.0.1.0.0 flags: MountPoint,Available mount time: Sun Mar 7 22:56:43 2010 volumes: 1 total size: 6442450944 total free: 6391668736 primary volume: /dev/asm/acfs_db1-152 label: flags: Primary,Available,ADVM on-disk version: 39.0 allocation unit: 4096 major, minor: 252, 77825 size: 6442450944 free: 6391668736 ADVM diskgroup ACFS ADVM resize increment: 268435456 ADVM redundancy: unprotected ADVM stripe columns: 4 ADVM stripe width: 131072 number of snapshots: 0 snapshot space usage: 0 Listing 2.2 shows a detailed list of acfsutil commands that are currently available. I’m going to dive much deeper into acfsutil in the next article in this series when I demonstrate how to create an ACFS file system snapshot (FSS) for faster patching of Oracle homes with virtually zero downtime. Mounting an ACFS File System Automatically After Server Reboot. As you can see from my invocation of the Linux ls command below, the new ACFS file system is actually not yet mounted:
#> ls -la /u01/app/oracle/acsfmounts/*
total 8
drwxr-x--- 2 oracle dba 4096 Mar 7 22:54 .
drwxr-x--- 3 oracle dba 4096 Mar 7 22:54 ..
To mount this new ACFS file system at the appropriate mount point, I’ll simply invoke the mount command as the root user, but note that for the first time, I’ll specify a new file system type of acfs: #> mount -t acfs /dev/asm/acfs_db1-152 /u01/app/oracle/acsfmounts/acfs_db1/ #> mount /dev/sda1 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda3 on /tmp type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/sdb1 on /u01/app/oracle/product type ext3 (rw) /dev/sdc1 on /stage type ext3 (rw) /dev/sdd1 on /u01/app/oracle/oradata type ext3 (rw) /dev/sdh1 on /u01/app/oracle/oraFRA type ext3 (rw) /dev/sdi2 on /DONTTOUCH type ext3 (rw) /dev/mapper/vg1-lvfra on /u01/app/flash_recovery_area type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) .host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1) none on /proc/fs/vmblock/mountPoint type vmblock (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) oracleasmfs on /dev/oracleasm type oracleasmfs (rw) /dev/asm/acfs_db1-152 on /u01/app/oracle/acsfmounts/acfs_db1 type acfs (rw) And one other thing that I must not forget to do before I reboot my server: I’ll need to add an entry to my server’s /etc/fstab configuration file to insure that my eventual new Oracle 11gR2 database home is actually mounted before I attempt to start the database within that home:
LABEL=/ / ext3 defaults 1 1
LABEL=/tmp /tmp ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/sdb1 /u01/app/oracle/product ext3 defaults 0 0
/dev/sdc1 /stage ext3 defaults 0 0
/dev/sdd1 /u01/app/oracle/oradata ext3 defaults 0 0
/dev/sdh1 /u01/app/oracle/oraFRA ext3 defaults 0 0
/dev/sdi2 /DONTTOUCH ext3 defaults 0 0
/dev/vg1/lvfra /u01/app/flash_recovery_area ext3 defaults 0 0
/dev/asm/acfs_db1-152 /u01/app/oracle/acfsmounts/acfs_db1 acfs defaults 0 0
Installing a New Oracle 11g Release 2 Database HomeJust as with the installation of a new Oracle 11gR2 Grid Infrastructure home, I’ll once again use the Oracle Universal Installer (OUI) utility to install a new Oracle 11gR2 database home. I simply invoked the standard runInstaller OUI script, this time from the /stage/11.2.0/database/ directory from my staging disk. I won’t bore you with my responses to every single prompt on every OUI panel here; however, I’ve included all of the panels and my responses in Listing 2.3 starting with Figure 2.6. Once all of the installation panels have been completed appropriately, the new Oracle database home has been successfully propagated, and I’ve run the standard root.sh script from within the new Oracle 11gR2 database home directory, I’m at last ready to create a new Oracle 11gR2 database, or even upgrade an existing 11gR1 database to release 11gR2 … but that’s a topic for my next article. Next StepsWe’re making excellent progress … but it’s going to get a lot more interesting (and a lot less tedious) in the next article in this series, as I’ll demonstrate how to:
References and Additional ReadingBefore you proceed to experiment with any of these new features, I strongly suggest that you first look over the corresponding detailed Oracle documentation before trying them out for the first time. I’ve drawn upon the following Oracle Database 11g Release 2 documents for this article’s technical details: E10881-02 Oracle Database 11gR2 New Features E10592-03 Oracle Database 11gR2 SQL Language Reference E10700-01 Oracle Database 11gR2 DataGuard Concepts and Administration E10595-05 Oracle Database 11gR2 Administrator’s Guide E10713-03 Oracle Database 11gR2 Concepts E10820-02 Oracle Database 11gR2 Reference E10500-02 Oracle Database 11gR2 Storage Administrator’s Guide E10837-02 Oracle Database 11gR2 VLDB and Partitioning Guide |
Figure 2.4. Selecting a Destination for New ACFS Volume