Denise Rogers discusses the essential tasks in conducting effective software evaluations revolving around data warehousing and business intellegence. Each step has a dependency on the previous one, starting with establishing the framework of the evaluation and adding progressively elaborate data that facilitates a decision making process that is resolute.
So the buzz is that there
is a huge and I mean HUGE initiative about a data warehousing and business
intelligence program and related projects that will put the company in a great
competitive position in the market place.
After the kickoff
meetings, the balloons, confetti and catered breakfasts have been popped,
tossed and consumed, reality sets in! The reality is that your unit has
primarily provided support for transactional, retail type operations that
"keep the lights on" for a number of business areas and now you are
being asked to expand your support infrastructure to provide services for data
warehousing and business intelligence! I don’t know about my data architect
family out there, but that’s enough of a smack of reality to make me put down
the toasted sprouted wheat bagel with smoked salmon and freshly squeezed orange
juice at the kickoff breakfast!
The reason for this wave
of nausea that has overcome us is that these types of projects mean building a
complete infrastructure, including a service management component in support of
something where nothing exists!
So we do what all data
architects do. We work with the project management team and IT Support to
conduct a number of extremely important software evaluations. These are career-defining
moments! It’s like being charged to choose the new car for the family that is
not only moving from an apartment in the city to the suburbs but is also
expanding to include the addition of new members to the family including a new
puppy.
If you have conducted the
evaluations well, you have successfully created a solution set that enables the
program to take giant leaps forward to successful delivery and added some sheen
and luster to your reputation as super hero! If the evaluations were poorly
executed, you have placed the project at extreme risk and added to your
reputation as a super zero!
So what are the essential
tasks in conducting effective and impactful software evaluations? In many ways,
it’s very similar to purchasing a new car. Think about it! When you make the
decision to buy a new car for whatever reason, it usually includes figuring out
what type of vehicle, reading tons of reviews and blogs, visiting lots of car
dealerships, taking endless test drives and speaking with various friends and
family members in order to have enough information to make the right decision
about what vehicle fits you and your family best.
Of all the evaluations I
have done so far, it’s not just the software itself that is being evaluated but
it’s the vendor and the company that is being evaluated too! There are a number
of mandatory tasks and/or steps that guarantee success every single time. It’s
all about doing the groundwork to ensure that due diligence was done. Each step
has a dependency on the previous one, starting with establishing the framework
of the evaluation and adding progressively elaborate data that facilitates a
decision making process that is resolute.
Understand the problem
What is the scope of the
project? Will this project have ripple effects across the enterprise? There
must be a complete understanding of the scope and the business requirements.
What is the problem that this project is being asked to solve? The answers to
these questions create the framework for the evaluation process.
Research the industry leaders and trailblazers
What software vendors are
leaders in this space? What are the research groups saying? What is the market
place’s view of these vendors? Are these companies financially solvent? Do they
have a robust organization? What is the size of their customer base? Are they perceived
as a forward thinking/leading edge company? In this economic climate, these
questions could not be more pertinent. This information provides the evaluation
team a perspective on the fiscal strength and reputation of the leaders and
trailblazers, as well as the company’s image as a technological leader in their
domain.
Select a list of vendors
Based on the research done
in the previous step, the list of vendors is an easy one to generate. However,
the list should include vendors that are leaders, up and coming companies,
niche players, even open source companies.
Create and Publish the
Request for Proposal
The RFP should include
information related to the project scope, functions and features that the
software solution must have, approaches and methodologies used as well as support
and operating costs.
The vendor’s responses to
the RFP provides the team with enough information to refine the vendor list and
initiate the process of scoring the vendor responses based on weighting factors
establish by the team.
Score the selected vendors
After the vendor scores
have been tabulated, ranked and rated, a picture is beginning to form that
gives the evaluation team a good idea of potential software solutions that
would be a good fit for the project and strategically for the company. (Whew!
That was intense!)
Conduct the technical
demos (aka dog and pony)
These meetings, apart from
the vendor demonstrations of how many customer logos and screen shots can be
included in a presentation deck and the marketing team’s ability to say as many
buzz words as humanly possible, are really the first chance that the evaluation
team gets to see the software. When I say "SEE THE SOFTWARE", I mean
paying close attention to navigation throughout the toolset, the number of
steps that are required to complete common functions, wizard-like assistance,
and "out-of-the-box" packages to enable solutions to be built
quicker.
Conduct a number of
customer reference feedback sessions
Schedule conference calls
with customers that have used the software solution(s) that your company is
considering. All members of the evaluation team should participate and gain a
deeper understanding of how the software works within a real life situation.
Also, ask the customer reference if they had a chance to do it over, what would
they do differently.
Research and Review Social
Media Outlets and Tech Blogs
Research the blogs and all
the social media outlets. In my experience, the opinions expressed here are
real and at times brutally honest. This task enriches the previous one in that
real customer experiences are captured.
Review session to reduce
and refine vendor list
Okay, so you are almost
there and you and your time have survived! After going through the first eight steps,
it should be pretty easy to know which of the vendors are serious contenders
and which ones are truly not a good fit for the project and the company
direction. The list should be no more than three of the top vendors.
Conduct the Proof of
Concept and review the results
This is the most critical
task in the effort! A successful proof of concept must start with a carefully
structured test where you have the evaluation team and the vendor together on
site to achieve a specific result. At the end of this task, the evaluation team
should have answers to the questions, "Did the software achieve the
results expected? How difficult was the software to work with? How did the
software perform on the server? If there were errors, how intuitive was it to
resolve the issues? Will there be a significant learning curve for the project
team?"
Compile findings and
create summary
Document everything! The
evaluation team spent a tremendous amount of time and effort. A lot of data was
recorded. This is where the team catalogs and packages information from all
previous steps in the process and creates a complete summary to be presented to
IT management on the software solutions evaluated. It should be crystal clear
which vendor came out on top!
Make final selection and
recommendation
The work is done and
management decision is a pretty easy one to make because the evaluation was
comprehensive and complete and would withstand any audit.
Now here are some stories
of doing these steps and not doing these steps!
Should I go with the semi-tractor trailer or the golf cart? The red sports coupe is really cool!
Okay, so I am working on
assignment at a client site, on a new project to completely overhaul and
modernize their entire IT infrastructure. This includes silo-ed applications
that had data replicated everywhere in all forms, shapes and sizes. Part of the
future state environment was to consolidate and integrate all common data in
one book of record for CUSTOMER! Sounds like a pretty good project to work on
as a data architect (me). Well, for whatever reason between the project manager
and the project sponsor the selection process took the form of listening to a
vendor presentation that has sold products to this company prior to this project.
That was enough for them to purchase an ETL solution (include data profiling
and data cleansings components). What do you think happened? Well for starters,
the installation and configuration took several weeks due to pre-requisites
that were never discussed, several components would generate errors and flag
the databases as corrupt, not to mention countless hours with the vendor
support to troubleshoot slow responsiveness of the servers and compatibility
issues. Whew! I just broke out in a cold sweat thinking of the countless traces
I had to send to the vendor support group to create patches for! Needless to
say, the project did not meet its project delivery dates and there were many
cost overruns. And what do you think the usage of that ETL software includes
now? Well the data profiling and cleansing components are either shut down or
used sparingly and the ETL toolset is used at about 50% of its functionality!
The project spent over $1,000,000 on a software solution that returned a fraction
on its investment! Ouch! And to think if a proper evaluation had been done, the
amount of pain, suffering and cost that could have been avoided! YIKES!
I want mud flaps on the pickup truck but I don’t want the bed-liner!
While working for a
healthcare company in the IT division, I had an opportunity to work on a DW/BI
project that included ETL and BI components. This company had not had done any
real data warehousing or business intelligence at the corporate level so this
was a new phenomenon for everyone. We worked as a team conducting evaluations
as defined in this article all the way to going through a complicated amount of
red tape to obtain servers on loan to conduct the proofs of concept! In the POC
however, we took it a step further and also engaged the vendors to do the
installation as part of the structured tests! We really, really, really got a
strong sense on how each vendor’s solution worked and as a result selected the
products that were a perfect fit for the project and the company! The project
was completed on time and within budget. This was done a few years ago and
these products have survived and are an integral part of the infrastructure
today!
As you can clearly see, it
really pays (literally and figuratively) to take the time to do software
evaluations properly!
