DW/BI applications are a huge investment. So much time, effort and cost goes into the implementation of the ETL solution, the staging area, the actual data warehouse, the data marts and the BI solution, so why do organizations not follow or have procedures in place to manage communication, enhancements and /or troubleshooting?
There is a huge investment in a DW/BI application, so much time, effort and
cost has gone into the implementation of the ETL solution, the staging area,
the actual data warehouse, the data marts and the BI solution. Why do
organizations simple not follow or have processes or procedures in place to
manage communication, enhancements and /or troubleshooting?
This is what happens when there is no process….chaos!
I was assigned to an account primarily to work on streamlining a DW/BI
application. I was struck by the fact that not only was there minimal
documentation (that should have been my first sign) but also that there really
were no processes or procedures to manage the application and its related
components. I am speaking of change management, release management or
communication management. So what do you think happened during the first
release under my watch? Well here’s what happened. As this was a data change at
the source application, it meant that the entire set of processes to populate
the data marts and generate reports had to be tested through and through. This battery
of tests included the ETL cycle to ensure that there would be no issues in
populating the data mart and the cube with the new source data. There also had
to be testing of the reports impacted, as well as regression testing.
In this case, the method of communication of this change came in the form of
our business users not being able to access data in the production on Monday
morning. After some troubleshooting, we discovered that the source application
had made some data changes but neglected to inform us! So after the dust
settled. We developed a plan to somehow deploy into production.
The first test of the ETL cycle went well. As the initial testing was
beginning on the report generation process, the business analysts from the
source application stated that the data was entered in error and the test is
invalid. Therefore, the source data had to be corrected and the ETL cycle had
to be rerun. This went on for several cycles, which meant that the window for
testing for the users of the DW/BI application was drastically reduced. There
was such a limited time for testing and validating the ETL cycle and the report
generating processes, that the production implementation was postponed twice!
The impact of this postponement was that the business users did not have the
reports they needed to complete a number of their business processes. Having
business users adversely impacted because of miscues, mistakes and missed
deadlines is unacceptable and can be avoided.
This is what happens when there is process……success!
How could this disaster have been avoided? Well, for starters, as the source
application began its development work related to this data change, the DW/BI
team should have been included in the communication loop. The DW/BI team or
designate should be a part of the extended project team of the source
application. This speaks to having a communication management plan in place.
Working with the source application, a plan should be in place that manages the
development, testing and deployment. A release management plan does just that!
Once a modification or enhancement has been approved, the release management
plan is executed. It would oversee the integration and flow of development,
testing, deployment, and support of both the source and downstream BI
applications. The release management plan and communication management plan
would ensure that the development, testing and issues resolution would be
managed such that both applications would be implemented into production within
the budgeted cost and time with minimal disruptions or outages (if any) to the
business users.
However, here’s the thing, whereas a communications management plan and a
release management plan provides a framework and structure for the effective
deployment of changes into production, agile development methods iterates
software development. What does agile development provide? Well, it stresses
close collaboration between the IT and the business teams. This means constant
communication between organized teams working to deploy business valued
solutions into production, and, of course, reducing the number of surprises or
outages in production!
Successful DW/BI Deployment requires a blend of project methodologies
In order for an organization to have a competitive advantage in the market
place, its DW/BI application must deliver timely and innovative products to a
widening set of business users. Agile methodology offers organizations the
ability to deliver innovative solutions in an iterative and timely way. It is
especially effective during software development where tasks are broken into
small increments, with each iteration going through a full development
lifecycle to deliver a new product or new version of an existing product.
However, as agile methodology works best for software development, a more
traditional project management approach provides tremendous value in areas such
as communication management, release management and program management. It is
with this combination of project management methodologies that a DW/BI
application will be able to successfully implement innovative, high quality
products without breaking its components, incurring outages and increasing
costs!
