Oracle Enterprise Manager 12c Cloud Control – Incident Manager – Part Four

Monitoring target systems is a critical responsibility of today’s database and system administrators. DBAs and SAs are responsible to ensure our systems are available, that their performance is within acceptable parameters, and watching out for any kind of error conditions and/or unusual behavior patterns.

Oracle has made great strides in providing tools and techniques for administrators to be able to take steps to proactively monitor their systems for all of the above conditions, and then some.

This trend definitely continues with the new feature in Oracle Enterprise Manger 12c Cloud Control called Incident Manager.

Oracle Enterprise Manager 12c Cloud Control – Incident Manager – Part Four

This is the final article in my series on the great new feature in Oracle Enterprise Manger 12c Cloud Control – Incident Manager. In this last article we’re going to look at how to create rules and rule sets and how they work together with using notifications.

Creating Rules and Rule Sets

Rules are the starting point for really controlling what and how Incident Manager monitors and responds to any kind of issues or errors that arise. To create a rule, these are the basic steps:

     Navigate to Setup> Incidents> Rules

From here you can edit an existing rule set or create a new one. Rules are added to rule sets. To edit an existing rule set, highlight the rule set and select Edit. To create a new one, click Create.

You’ll have to decide what type of target the rule set applies to, and optionally you can select “All,” which means it could be associated with any applicable target.

The first step is to choose the type of rule set, and you have three choices – Event, Incident, Problem. Click on continue, and then it’s a matter of simply following the wizard and saving the rule into the rule set.

Using a Rule to Generate an Incident

Let’s say you want to create a rule that will generate an incident to be tracked in Incident Manager – and it’s going to be based on an event such as a Transactions Per Second metric alert.

Edit or create the rule set. Once in the rule set, select the “Rules” tab and click Create.

Because this would most likely be the first action to take, choose the option “Incoming Events and Updates to Events”, and then choose Continue.

For the Event Type, choose Metric Alert. Then choose the specific metric you are basing the rule on. Click on the +Add button to see a list of available metrics by target. On this page, you will also see the options for defining a severity level and corrective action option. Once you have selected the metric, click Next.

Your next step is to add the action that should be taken, and Create Incident is one of the available actions. You can assign the Incident to a user, set the priority and even log a ticket as you set the action.

Then click Next to move onto the final step in the wizard, which is to add a name and description for the rule.

Using Rules to Escalate an Incident

Once an Incident is generated, another rule that we may want in place would be to escalate the incident if it is not addressed quickly, for example, we may expect that an incident is at least acknowledged within 24 hours, in particular if it is a severity of fatal. If this does not happen, the incident is going to be escalated to the IT Manager. If it goes 48 (total) hours it will be further escalated to the IT Director and if it’s 72 (total) hours it goes to the VP. We’ll use escalation levels to help control this, Escalation Level 1 for IT Manager, Escalation Level 2 for IT Director and 3 for VP.

In this case, when you click Create on the Rules Tab (after editing an existing rule set or creating a new one as appropriate), we are going to be using an incident as the basis for our rule, we need to make a different initial selection, so rather than “Incoming Events…” be sure to choose “Newly Created Incidents or Updates to Incidents”.

In the next wizard step you can either select specific incidents, or select all incidents where the severity = fatal if you want more of a blanket rule.

We want the action to happen based on a specific condition. In the Conditions for Actions region select “Execute the actions on the conditions specified” then select “How long the incident is open and in a particular state”.

  • Set the time to 24 hours
  • Set the Status to be equal to Open
  • In the Basic Notification Region, select the administrator to be notified and the method such as email.
  • Under Update Incident, choose the option to Escalate To Level 1

To create the next escalation – add another rule, similar to above with the following options:

  • Set the time to 48 hours
  • Set the Status to be equal to Open
  • Set the Escalation Level equal to Level 1
  • Add the administrator email information for the notification
  • Choose the option to Escalate to Level 2

The final escalation would be

  • Set the time to72 hours
  • Set the Status to be equal to Open
  • Set the Escalation Level to 2
  • Add the appropriate administrator account to be notified
  • Escalate to Level 3

By the way, more than one account can be listed as the recipient, which means, the rules could be set up to copy the DBA for any incidents that do get escalated (or any other administrator that might need/want to see the notification)

Once all of the rules are created, review the rule set to ensure that they have all been added in the correct order.

Rule sets and rules can be created in the same manner to work on problems, track job status changes, and other notifications in addition to working with events and incidents making them very robust. For example, a DBA can create a private rule set that would send them a notification when a scheduled backup job gets a status of failed (or completed if desired) by creating an Event rule on the event “Job Status Change”.

Advanced Options

There are also many advanced options that are available for rules and rule sets including:

  • Separate Notifications Based on Severity States – for example, send a page to a DBA for a critical metric alert and an e-mail for a warning level alert on the same metric. These could go to the same DBA or different DBA’s
  • Rule Sets To Generate Tickets – rules can be configured to automatically send a ticket to a help desk tool such as Remedy (naturally this would have to be fully configured to Cloud Control first)
  • Rules to Notify Different DBAs Based On Event Type – in an organization that has application DBAs, database DBAs, rules can be defined that events tied to general system settings go to the database DBA while events that are application specific can be directed to the appropriate application DBA.

For more details on the advanced options, Oracle has provided a very detailed documentation guide that can be found here.

In conclusion

The new Incident Manager feature of Oracle Enterprise Manger 12c Cloud Control is a feature rich tool that will enable DBAs to track, manage, research and escalate (as necessary) virtually any sort of hiccup that their database(s) and related targets might throw at them.

Definitely a feature worth knowing.

See all articles by Karen Reliford

Karen Reliford
Karen Reliford
Karen Reliford is an IT professional who has been in the industry for over 25 years. Karen's experience ranges from programming, to database administration, to Information Systems Auditing, to consulting and now primarily to sharing her knowledge as an Oracle Certified Instructor in the Oracle University Partner Network. Karen currently works for TransAmerica Training Management, one of the foremost Oracle Authorized Education Centers (OAEC) in the Oracle University North America region. TransAmerica Training Management offers official Oracle and Peoplesoft Training in Coral Gables FL, Fayetteville AR, Albuquerque NM, Providence RI and San Juan PR. Karen has now been teaching Oracle for Oracle University for more than 15 years. Karen has attained her Certified Technical Trainer designation along with several Oracle certifications including OCP-DBA, OCP-Internet Developer, Oracle Expert - Oracle 10g RAC and Oracle Expert - Oracle Application Express (3.2). Additionally, Karen achieved her Oracle 10g Oracle Certified Master (OCM) in 2008. Karen was raised in Canada, and in November 2009 became a US Citizen. Karen resides in Columbus OH with her husband, Ron along with their 20 pets, affectionately referred to as the "Reliford Zoo".

Get the Free Newsletter!

Subscribe to Cloud Insider for top news, trends & analysis

Latest Articles