Building a Data Warehouse Blueprint for Success
July 28, 2010
One of the most integral components and critical success factors of any enterprise data warehousing initiative is the Solutions Architecture document, a high-level conceptual model of a data warehousing solution. Learn why this collaborative effort that addresses the needs of all major stakeholders, including both the business units and Information Technology (IT), is essential.
One of the most integral components and critical success factors of any Data Warehousing initiative is the Solutions Architecture document.
It is the high-level conceptual model of a data warehousing solution. The solutions architecture creates the roadmap of the possible solution, providing a context for the planning and construction of a comprehensive future state environment. It essentially sets the framework for the development of the data warehousing solution. It is a collaborative document that addresses the needs of all major stakeholders including both the business units and the Information Technology (IT). It transfers all of the ideas, concerns, issues and pain points associated with the current infrastructure and project from business questions to possible technologically based answers. It reinforces the scope of the project.
Therefore, the Solutions Architecture document should not be an academic exercise. It must be a high-level design of the data warehousing solution specific to the project and should include the following items.
Using a complete set of business requirements, the ETL framework will be used to build solutions to cleanse, standardize and integrate data from disparate sources into the data warehouse and/or operational data store.
The staging area is the information hub primarily used in support of the ETL framework were source data goes through a series of enrichment processes for the purpose of populating the data warehouse and/or the operational data store.
Operational Data Store
The operational data store (ODS) is a set of logically related data structures within a database that primarily includes integrated, subject oriented, volatile, non-historical atomic data used specifically in support of transactional processing and timely operational reporting.
The data warehouse is the opposite of the ODS in that it is a database that contains integrated, historical, non-volatile, organizational level, atomic data used primarily in downstream decision support initiatives such as business intelligence.
A data mart is a subject oriented set of organizational level data created to support analytical reporting requirements for a specific business unit.
Business Intelligence is a logical grouping of applications and repositories that are specifically architected for information delivery to the business community.
Steps in building a data warehouse solutions architecture document
As the saying goes, "a picture is worth a thousand words", this is a great way to start the process. Once a diagram has been developed, building the outline and related detail become almost organic. This is especially true with the use of industry standard enterprise architectural frameworks.
However, the architects on the project team must choose a format for the solutions architecture document. In my years of doing data warehousing, two formats are quite common. It can be a high-level "how-to" guide with the requisite definitions of each of the data warehouse components. Alternatively, it can be a "roadmap" with definitions of each data warehouse component along with the descriptions of related standards and guidelines that need to be adhered to within the Software Development Life Cycle (SDLC).
The format selection is usually based on the culture and the maturity of the organization and the team must fully understand this aspect of the project in order to make the right selection.
The main thing is that independent of the format chosen, the building of the solutions architecture is a collaborative effort where the major project stakeholders from the executive team, business units and IT must provide input into the document to create a meaningful and usable asset to the organization.
In Today's World
However, the audience for this document has dramatically changed over the years. In the mid to late nineties, this document was created only for the executive and the in-house project teams. Since the 21st century, IT has gone through a technological and cultural evolution of sorts and the ways things are done are quite different now. The solutions architecture document now has a different audience that it speaks to. This audience is a project team that is not quite co-located, meaning that the team now includes the executive team, the in-house project team, the outsourced and the offshore project teams.
Since the architects are the primary authors of this document, it is therefore their responsibility to complete a number of exercises specifically created to understand the organizational communication plans, rules of engagement and project management procedures. It may seem like this is too much of an extra effort, however if the whole idea behind the document is to produce a workable solution that the entire project team thinks is realistic and the executive team has bought into, it's worth the extra hours of time spent!
Don't ever forget that the solutions architecture document must be able to stand with other project assets within the organization and also provide a comprehensive guide to the build out of the data warehousing environment and its related components.
And yes! Of course, I have a couple of stories that that I would like to share that illustrate these points!
The one that did not work
A few years ago, I was assigned to a data warehousing project that would completely modernize an aging infrastructure. My role was that of the data architect, I was responsible for the design of the entire data layer. The project charter and scope had been socialized and accepted. So, my thinking is that the easy part is done now the real work begins. I just did not anticipate how real and tough it would be!
The approach was to have the project manager author all documents produced from the project with the team (including the architects) contribute as required in the project plan.
Well what do you think happened?
The project manager completed the entire document and asked each of the architects to contribute only a diagram and set of definitions for each component within the diagram. During the review with the major stakeholders, I just remembered noticing lots of disinterested looks, few questions being asked except from the IT department who were very angry that they were not included in the creation of the document. The sections of the document related to the data layer (including the data warehouse, staging area, ODS and data marts) was criticized as not being based in reality, not taking the existing standards into consideration. The whole document was dismissed as an academic exercise by the IT review team. Needless to say, that review did not go well and after the meeting was over, I basically walked out of the conference room feeling like I was blindsided by a train that I did not see coming!
So what finally happened to that document? It succeeded in being an item that was checked off on a project plan as being completed. Last I recall, it is now a very expensive doorstop or bookshelf filler in somebody's office! Not a good ending for such an important asset to the project and the organization.
The one that did work
Recently, I have been working on assignment as a data architect within a financial company. This project was scoped to modernize and centralize an IT infrastructure that was extended over the IT and business landscape. Meaning, IT supported some of the database applications and the business units supported others.
My role was to complete sections of the Solutions Architecture document related to the data layer. The approach used here was very different to my prior assignment. The project management team assigned the completion of the Solutions Architecture document to the architecture team!
We agreed that with the completion of each component of the solutions architecture, we as architects would socialize, review, address issues, solicit input and provide clarifications with all major stakeholders on the IT and business teams fully aware that this document would be used by a project team that was not co-located and included offshore members as well.
As we stepped through the process and with every review, it became quite clear that this document had "legs" to stand up with all other assets of the project as well as a common belief being shared that the information being produced will be used extensively as the data warehousing environment is rolled out over time! That's a much better ending than the last one! I can assure you this document will NOT be the dusty pile of papers on any bookshelf anywhere at any time!