Oracle RAC: How shared storage works on VMware - Part 2
July 27, 2007
In this article, we will discuss creating clustered VMs on the VMwares ESX 3 Server. There are several options for doing so. You can create a cluster on a single ESX host. This is generally the easiest way of doing things. However, you might want to test and spread your nodes out across ESX hosts. That is possible as well. Or, if you do have a test or development RAC on physical hardware and are willing to move your RAC to a Virtual Infrastructure, you can go on and add another node; but this time just on the VI3 stack. By gradually deleting nodes from the physical infrastructure and adding nodes to the VI3 infrastructure, you will migrate your RAC to VI3!
Clustering: Application Clustering vs. VirtualCenter Clustering
We won't delve too much into clustering concepts here; you basically use cluster technology in order to achieve high availability or/and scalability. We can cluster nodes by the means of software and hardware clustering solutions. Typical applications like Oracle RAC come to mind when thinking of cluster-aware applications. A typical clustering setup would be a couple of nodes, sharing the same disk, much like our Oracle RAC, mail servers, etc. The shared disks (LUNs: Logical Unit Number) must reside on a SAN using Fiber Channel (FC). Usually you have an extra network connection for cluster heartbeat traffic, much the same as Cache Fusion for Oracle RAC.
Also, you have traditional clustering against the Virtual Center Clustering. VMwares HA (High Availability) option is a cold way of clustering nodes within the Virtual Center clusters. Traditional clustering uses a hot standby node while the Virtual Center HA is a cold stand by approach.
Clustering Options in VMware ESX 3 Server
Testing Oracle RACs installation and even doing a demo to the audience at a client site is fine, but if you want to try migrations, upgrades or patches, it is advisable to use ESX Server. In fact, most of my readers are ESX users, and DBAs who want to learn their RAC are increasingly deploying Oracle RAC in their test, development and even staging environments. Production environments are not yet running RAC on ESX but I wont be surprised if there are environments that have Oracle RAC on VMwares ESX. I have seen a reasonably intensive single node Oracle on ESX. Moreover, the OS was Windows!
Coming back to the types of clustering for your Oracle RAC options:
1. Clustering Oracle RAC on single ESX host
2. Clustering Oracle RAC across several ESX hosts
3. Clustering Oracle RAC across ESX and physical nodes
Clustering RAC on single ESX host
The steps are simple here. We first create our first node.
1. Start your VI client and log on to the Virtual Center or the ESX Host itself.
2. Create new machine and select custom.
3. RAC Resource pool: If you want your RAC to run within a specific pool that has high CPU shares then you will use this to create your VMs.
4. Datastore: It could be on your SAN (Storage Area Network) or DAS (Direct Attached Storage). For Linux/Unix, I usually create two disks. One 10G and the other with 4G for swap.
5. Guest OS: Choose your OS.
6. CPU: You can choose up to 4 vCPUs.
7. Memory: You can allocate up to 16G per VM.
8. Network: We create two NICs. One for public and the other for private.
9. Install your OS (Operating System)
10. Add additional nodes for shared storage.
Eventually depending on your clustering preferences it will look like this after the first node creation:
The final picture might look like this; do note that we created virtual switches in the ESX that allow for high speed interconnects while staying totally disconnected from the physical NIC of the ESX server. The public virtual switch connects the nodes to the application via the hosts NICs :
As you see, this is not hard to achieve. Following simple tasks you can have your virtual machines up and running in no time. You create and customize your first node, create the second node, add storage (including shared storage) and configure the IP addresses.
How to create shared disks on the ESX host?
This isnt that hard either. Just execute the following steps on the ESX host:
1. Logged into ESX as root on the remote secure shell, you could use putty on windows or simple ssh command on your Linux OS.
Create and Zero out the disks using the following command:
vmkfstools -c <size> -d eagerzeroedthick /vmfs/volumes/<mydir>/<myDisk>.vmdk. Example: vmkfstools -c 10Gb -d eagerzeroedthick /vmfs/volumes/RACShare/asm01.vmdk
2. Zeroing out an existing disk can be done : vmkfstools [-w |--writezeroes] /vmfs/volumes/<mydir>/<myDisk>.vmdk
3. Open edit settings for node one and select add existing disk; browse to the datastore and create a new SCSI device by choosing SCSI(1:0)
4. Upon adding your first shared disk you will notice that a new SCSI controller is created.
5. Add all the vmdks you created: ocr.vmdk, votingdisk.vmdk, spfileasm.vmdk, asm01.vmdk, asm02.vmdk (for Oradata), asm03.vmdk, asm04.vmdk (for Flash Recovery Area)
6. Repeat these steps on all of the nodes.
7. Select the SCSI controller , in the edit settings pane, check if the controller type is set to LsiLogic
8. In the same panel set the SCSI bus sharing to Virtual (as illustrated)
* : In the Solaris 10 installation for the Oracle RAC 10g R2 testbed, I encountered a funny problem. After having installed the OS and configuring all of the memory limits, ssh equivalency, I then moved on to cloning my VM (which was fully prepared) on the Virtual Center. Then I added shared storage and upon restarting the OS, I lost my vmxnet0 NIC. This happened because I added another SCSI controller for the shared disks and the PCI slots shifted, making the OS to assume that the NIC was removed. So in Solaris RAC on ESX, make sure that you add all of the disks while creating the VM and then proceed to install the OS.
1. Strangely enough, even after checking the type LsiLogic, you get a warning (per shared disk on that controller) regarding the type. Click yes to all of them.
2. Use Virtual Center to clone the fully ready node and save configurations, which you can always fall back on should things go wrong.
3. If you dont have Virtual Center, then use VMware Converter to make a backup copy and also create online clones on the ESX machine.
In the next installment, we will explore the Shared Storage considerations across the ESX hosts and also Oracle RAC clustering between VM on ESX and on an Oracle node on a physical host.