We are in the midst of a data architecture revolution!
Whether it’s data warehousing, MDM, business intelligence or data modernization,
there is some sort of related project, program or initiative.
However, as we all sit around the planning table, in work
session after work session, the theme is always the same. That is, metadata is
added to the project plan as an integral component but is downgraded and
eventually dropped off the implementation part of the project plan. The reasons
given in many instances are always related to time constraints, re-definition
of scope or the metadata component is too large and should be treated as a
separate project, which almost always never happens.
The impacts of not including metadata and metadata
management as part of the project have far-reaching repercussions throughout
the organization. These impacts are costly and affect the organization in a
variety of ways. In my experience on many of these projects, not having
metadata and/or metadata management rears its ugly head in the following ways:
Extended discovery phases of major data warehousing or MDM initiatives
During the early planning stages of any DW or MDM
initiatives, it is almost a certainty that the time allocated to complete the
discovery phase is not enough. Why? Because you almost always have to do the
type of digging into the documentation and SME reviews/interviews that can
easily be compared to an archeological dig at one of the pyramids in Egypt, in
the desert heat! Let me walk you through one of my recent projects. This IT
modernization project included building a centralized ODS from many silo-ed
applications that were written back in the seventies, that is the 1970s! I was
told that all applications were documented, kept in binders and were up to
date. Well folks, not only was the documentation limited (only the name of the
application and a description), but the binders had an inch of dust (remember
Egypt and desert heat) and the “last updated date” was usually in the nineties
that is the 1990s! I ended up identifying the SMEs, did some extensive
interviews, recorded the information and cataloged the information gathered. I
then built a series of data models and DB tables that cataloged the metadata
and constructed queries to generate a series of reports. The original estimate
for this discovery phase was three months, the actually time spent was SIX
MONTHS! Now I know that this is an unusual case, however, I think the point is
made that had the metadata been a part of application maintenance, we would not
be having this conversation. I am just now recovering from the emotional scars!
Difficulty in completing business processes
In any organization, there is always the introduction of new
products and services. These new offerings require the underlying IT
infrastructure to make a number of modifications to the environment, including
the data layer, application layer, etc. However, in the rush to get these
products and services implemented (and I mean RUSH!), there is always the
“cutting corners” to get it done sooner. And what do you think is cut first?
DING! DING! That’s right! Tasks related to completing the metadata layer. So
the products and services are deployed into production without the metadata.
Everyone is happy! YEA! However, some time passes and management or an external
agency makes a request for information related to the new products and services.
The team that typically performs these functions does not have any of the
metadata information to complete the request because it was a “corner” that was
cut! Its almost like going on a road trip but the GPS was not updated with the
new landmarks (that you want to visit) that were recently created. Therefore,
you drive right past, get lost and spend hours trying to find it! How did I
resolve this issue? I reversed engineered the database into a data model and
worked with the business analysts to create definitions of the new entities and
attributes and updated the metadata layer. I also documented the process for
reference. The total time taken to complete the request is A LOT LONGER than
the original estimate! All I am saying is that metadata is like the GPS that
needs to be synchronized with new information on a very regular basis. It is NOT
a “corner” to cut. Moreover, who do you know likes to go on a road trip with
map that looks like a papyrus from ancient Egypt! Not me!
Cross training new members of the team
In the wonderful world of IT, we all know that there is a
constant turnaround of staff on project and support teams. It’s a fact of life.
Why does it have to be so difficult to get new members of a team ramped up with
information needed to make an immediate positive impact and make valuable
contributions sooner? Maybe because there is not enough of a library of
information that is readily available as a reference to incoming teammates. How
many folks out there can relate to that? I mean you are the person in the shoes
of a new member of a team taking on massive projects related to data
warehousing, business intelligence or MDM and you have like a paragraph of
information to use as a reference to fulfill the role of the data architect,
database administration or ETL architect! I see a lot of hands being raised. I
have been in those shoes way too many times to count. However, I have also been
on the other side where I have had to introduce new teammates to the project.
The reaction I always have in this scenario is WOW! And a smile and look of
relief. Why? Because I took the time to add the required metadata documentation
and artifacts to the underlying architecture that can easily be generated into
reports. I don’t know about you folks out there but I would rather have tons of
information to reference as a new team member to add value to the project as
quickly as possible.
Breaking the Cycle
A way to break from this painful cycle is perhaps placing
a value on metadata and metadata management. I think as
experienced professionals in IT that some responsibility lies with us to market
the concept of metadata and metadata management. We should use our experiences
and that of business areas that we support to dispel the notion that metadata
is a “frill” and not a mandatory requirement to complete a project or
initiative. Maybe we can market this idea by capturing “pain points” that
business units have experienced or better yet not just putting a value on
metadata but also actually placing a cost of NOT including metadata efforts as
part of a project. It’s just something to think about as we build scalable and
sophisticated data architectures that includes SOA, HPC/cloud computing, data
governance, and dashboards. There are many software solutions out there that allow
us to break this painful cycle. Let’s put it in motion!
