Performing Data Warehouse Software Evaluations

Here are some ideas that may make the process of evaluating data warehousing software more effective. This is not a comprehensive list of tasks to follow in a technology evaluation. Rather, these are points that seem to be rarely discussed or followed in this wave of interest in data warehousing. An excellent paper to read along with this essay is Nigel Pendse's How not buy an OLAP product - which has advice that, for the most part, is applicable to buying any sort of data warehousing/decision support technology.


Do the evaluation yourself
That is, do not rely solely (or even in large part) on the ideas of someone outside your organization. There is no "metaphysically" best technology out there. All technologies have to be evaluated in the context of your organization's needs, expectations, limitations, and resources - which you know better than any outsider. Also, you can never be sure of the outsider's biases. Outsiders's main worth really comes from their knowledge of criteria you can use in the evaluation - though you have to decide the weight of each criterion.


Always first ask whether technology already in-house can do the job
Successful data warehousing/decision support systems can often be built without the specialized tools you see listed in this site. Taking on additional technology in you organization always imposes some burdens that should always be recognized before you hand over your organization's money.


Get references
Talking to reference sites is one of the most effective means of getting practical information. You would be surprised how important operational issues surface while doing evaluations. Some hints on reference gathering practices that have worked for me are:

Ask the software vendor for a complete list of referenceable sites - Try to have options as to which organizations you will call.

If this is a major decision for your company, call 5-6 sites - You need a minimum number of sites to help you detect patterns.

Make a telephone appointment to talk with the reference - The reference will appreciate this.

Plan on 20 minutes with the reference - Again the reference will appreciate this.

Ask open-ended questions - You will find some interesting information with skillful questions.

Send your questions to the reference in advance - Some of the references will be more comfortable if they know what you'll be asking.

Send a thank you note to your references asking if it would be okay to make a quick follow-up call if necessary - This will lay the groundwork if you have to call about another issue.


If you are going to see multiple vendor demos, build a test case that each vendor will follow
This will allow you to compare apples to apples and peaches to peaches. Leave some open time at the end of the demo so the vendors can show features that were not covered well in the test case. One more point. Because departing from the standard vendor dog and pony show takes time on part of the vendor, many will be unwilling to do this unless you are talking about a major purchase.


Be skeptical of data warehousing pundits' endorsements or reviews of technology
Often these pundits get compensated handsomely for these objective appearing endorsements or reviews.


Read stock analyst reports on publicly held vendors and the industry outlook
Though these reports are intended mainly to get people to buy stocks, many times these reports can be an excellent source of background information on a vendor. Many libraries will have a large collection of these reports stored on CD.


Check how well the software handles maintenance
Most of the time spent with a software tool will be with maintenance. See how well the tool handles changes. For instance, most tools work with something like a data dictionary. See what are the consequences of changing the name of a field in the data dictionary. See how the dictionary helps you locate and change queries, reports, forms, macros, etc. that may be affected by the name change.


Understand the tradeoffs the software makes
Usually there is not a free lunch! Designers of tools trade off speed, capacity, computer resource consumption, ease of development, ease of use, and ease of maintenance. For example, several report and query tools can be made quite accessible to end users if you are willing to maintain extensive data dictionaries. Several OLAP tools attain quick retrieval times by requiring the storage of huge amounts of pre-calculated numbers. To prevent some nasty surprises once the tool has been purchased, make sure the persons making the buying decision understand these tradeoffs.


Go to the vendor road shows to talk with other attendees
Sometimes I think that the audience at the vendor road shows is the best source of information. If you'll make a point of talking with several other attendees, chances are you will come across a person who is in at the same stage in evaluating warehousing tools. You will find that you and that person can exchange information that is mutually beneficial.


Check the financial stability of the vendor
If you for work for an organization with an accounts receivable department, the people in that department can help you with this. A simple check could save you some major potential grief.


Have a representative team perform the evaluation
Often technology acquisitions fail or go awry because a group within an organization felt it did not get its views heard during the evaluation. One of the first steps in a technology evaluation is to identify all 'interested parties' in the acquisition. Make sure these parties are asked how they want to be represented in the evaluation. If parties that are in conflict with each other will actively participate, if you do not have the skills and/or patience to be a mediator, seek the services of an outside facilitator. Facilitation skills can be especially helpful if you have sessions dedicated to setting criteria, making your short list, and making the final decision.


If you're evaluating an end user tool, let an end user lead the evaluation effort
It seems odd but some organizations buy end user tools with little input from the end users of these tools.