DW/BI Administration Without Process or Procedures Can be Catastrophic
January 25, 2011
DW/BI applications are a huge investment. So much time, effort and cost goes into the implementation of the ETL solution, the staging area, the actual data warehouse, the data marts and the BI solution, so why do organizations not follow or have procedures in place to manage communication, enhancements and /or troubleshooting?
There is a huge investment in a DW/BI application, so much time, effort and cost has gone into the implementation of the ETL solution, the staging area, the actual data warehouse, the data marts and the BI solution. Why do organizations simple not follow or have processes or procedures in place to manage communication, enhancements and /or troubleshooting?
This is what happens when there is no process….chaos!
I was assigned to an account primarily to work on streamlining a DW/BI application. I was struck by the fact that not only was there minimal documentation (that should have been my first sign) but also that there really were no processes or procedures to manage the application and its related components. I am speaking of change management, release management or communication management. So what do you think happened during the first release under my watch? Well here's what happened. As this was a data change at the source application, it meant that the entire set of processes to populate the data marts and generate reports had to be tested through and through. This battery of tests included the ETL cycle to ensure that there would be no issues in populating the data mart and the cube with the new source data. There also had to be testing of the reports impacted, as well as regression testing.
In this case, the method of communication of this change came in the form of our business users not being able to access data in the production on Monday morning. After some troubleshooting, we discovered that the source application had made some data changes but neglected to inform us! So after the dust settled. We developed a plan to somehow deploy into production.
The first test of the ETL cycle went well. As the initial testing was beginning on the report generation process, the business analysts from the source application stated that the data was entered in error and the test is invalid. Therefore, the source data had to be corrected and the ETL cycle had to be rerun. This went on for several cycles, which meant that the window for testing for the users of the DW/BI application was drastically reduced. There was such a limited time for testing and validating the ETL cycle and the report generating processes, that the production implementation was postponed twice! The impact of this postponement was that the business users did not have the reports they needed to complete a number of their business processes. Having business users adversely impacted because of miscues, mistakes and missed deadlines is unacceptable and can be avoided.
This is what happens when there is process……success!
How could this disaster have been avoided? Well, for starters, as the source application began its development work related to this data change, the DW/BI team should have been included in the communication loop. The DW/BI team or designate should be a part of the extended project team of the source application. This speaks to having a communication management plan in place. Working with the source application, a plan should be in place that manages the development, testing and deployment. A release management plan does just that! Once a modification or enhancement has been approved, the release management plan is executed. It would oversee the integration and flow of development, testing, deployment, and support of both the source and downstream BI applications. The release management plan and communication management plan would ensure that the development, testing and issues resolution would be managed such that both applications would be implemented into production within the budgeted cost and time with minimal disruptions or outages (if any) to the business users.
However, here's the thing, whereas a communications management plan and a release management plan provides a framework and structure for the effective deployment of changes into production, agile development methods iterates software development. What does agile development provide? Well, it stresses close collaboration between the IT and the business teams. This means constant communication between organized teams working to deploy business valued solutions into production, and, of course, reducing the number of surprises or outages in production!
Successful DW/BI Deployment requires a blend of project methodologies
In order for an organization to have a competitive advantage in the market place, its DW/BI application must deliver timely and innovative products to a widening set of business users. Agile methodology offers organizations the ability to deliver innovative solutions in an iterative and timely way. It is especially effective during software development where tasks are broken into small increments, with each iteration going through a full development lifecycle to deliver a new product or new version of an existing product. However, as agile methodology works best for software development, a more traditional project management approach provides tremendous value in areas such as communication management, release management and program management. It is with this combination of project management methodologies that a DW/BI application will be able to successfully implement innovative, high quality products without breaking its components, incurring outages and increasing costs!