Project teams constantly face a barrage of new products and technologies, and
have a difficult time differentiating marketing slides and grand promises from
deliverable products when making strategic IT acquisitions. Flat IS budgets
and limited internal IS resources further exacerbate the selection problem.
Project teams state that they struggle with a number of critical issues when
selecting enterprise technologies:
Project teams have no effective way to identify the critical
vendor and product questions necessary to successfully initiate the evaluation
Project teams have no ability to effectively prioritize
the different criteria, once identified, relative to one another. As a result,
final priorities are often more the result of internal political agendas
than true needs and requirements.
Project teams have no ability to gather objective, validated,
updated data on the available vendor alternatives. It is a well-known problem
that vendors have a tendency to exaggerate product, service and corporate
capabilities if it enables them to move to the next phase of the deal. Unfortunately,
most project teams have no true ability to separate fact from hype, especially
because its high-end, strategic technology selections are often either the
first of its kind or the first in an extended period within a specific organization.
to TEC research, the net result has been that over 80% of enterprise technology
evaluations run over time and budget, and that once selected, over 50%
of the implementations fail to meet functional and total cost expectations.
The solution is to create a structured, repeatable process for evaluating
technology solutions and the vendors that provide them. Best practices
drawn from TEC client organizations that have completed internal technology
selections suggest that project teams should examine six key criteria
groupings. The first three criteria sets should examine product specific
capabilities, while the second three should investigate the software vendor's
overall corporate capabilities. TEC has found that organizations that
experience the most successful selections have placed almost an equal
(approximately 50 percent each) overall priority on the tactical and strategic
criteria documented below.
Product Evaluation Criteria:
This is the most obvious product evaluation criterion and the favorite of
technophiles worldwide. Simply put, this evaluates the features and functions
delivered by the product as it exists today, and together with product architecture,
often incorrectly makes up over 90 percent of the overall selection importance
within IT product selections. Product functionality sub-components are usually
defined in one of two ways:
Functional groupings: within enterprise management
technologies, these categories could include enterprise management console
functionality, user administration, asset and inventory tracking, electronic
software distribution, server management and monitoring, job scheduling,
backup and recovery, enterprise service desk and enterprise security.
Process groupings: within user administration
technologies, this could include profile development, profile administration,
profile distribution and profile manipulation.
Relative to the other five evaluation criteria, best practice
selections place approximately 25 percent of the overall selection importance
on the product functionality criterion.
This criterion defines the technical architecture of the product, and the
technological environment in which the product can run successfully. Subcriteria
include product and application architecture, software usability and administration,
platform and database support, application standards support, communications
and protocol support and integration capabilities. Relative to the other
five evaluation criteria, best practice selections place a lower relative
importance, approximately 15 percent, on the product technology criterion.
However, this apparently lower importance is deceptive, because the product
technology criterion usually houses the majority of the selecting organization's
mandatory criteria, which usually include server, client, protocol and database
support, application scalability and other architectural capabilities. The
definition of mandatory criteria within this set often allows the client
to quickly narrow the long list of potential vendors to a short list of
applicable solutions that pass muster relative to the most basic mandatory
This section, often the criterion most closely scrutinized by project teams
that make sub-optimal product selections, should examine the initial product
acquisition cost relative to its peers in a series of specific, predefined
acquisition scenarios. In addition, the IS organization should understand
longer term costs, including maintenance fees, upgrade costs, training and
implementation costs, and service and support fees. Traditionally, the most
successful project teams eliminate cost from the initial evaluation steps,
focusing first on mandatory functional and technical criteria, as well as
the strategic vendor evaluation criteria documented below. Only after a
short list has been defined do the most successful organizations reinsert
the cost criteria into the decision, using relative cost differences between
products for negotiating leverage during the final selection phases. Relative
to the other five evaluation criteria, best practice selections place approximately
10 percent of the final selection importance on the product cost criterion.
Vendor Evaluation Criteria
Corporate Service and Support
This criterion defines the capability of the vendor to provide a high level
of implementation services and ongoing support. Repeated industry surveys
have identified this parent node as the single largest differentiating factor
among potential selection options, as well as the greatest indicator of
ultimate user implementation success and long term vendor viability. It
therefore represents a critical evaluation category for strategic IT selections.
A proper professional services and support evaluation should include both
subjective, quality measures validated by current product users, and objective,
quantitative criteria within both the professional services and product
support categories. Within the professional services category, project teams
should examine vendor consulting services, systems integration capabilities
and project management skills. Product support should evaluate characteristics
such as the geographic, language and time coverage of the vendor help desk,
help desk service quality, and number, quantity and scope of vendor training
classes. Relative to the other five evaluation criteria, best practice selections
place approximately 20 percent of the overall selection importance on the
service and support criterion.
This is a critical yet often overlooked category that should examine the
financial and management strength of the vendor. Given the huge dollars
spent upon and strategic importance applied to most major IT procurements,
the financial stability of the vendor supplying the product cannot be overlooked.
The vendor viability evaluation section should combine quantitative Wall
Street ratio and metric analysis with qualitative management and corporate
evaluations. Only by combining the two components can executives accurately
assess the risk and benefit of corporate investment in a specific product
and vendor. At a minimum, the corporate viability criterion should evaluate
the overall financial viability of the vendor, its macro and micro market
viability, its sales and marketing viability, its management viability,
and its research and development viability. Relative to the other five evaluation
criteria, best practice selections place approximately 20 percent of the
overall selection importance on the corporate viability criterion.
This evaluates the corporate road map and strategy of the software vendor
with specific timelines regarding how the product will be developed, sold,
and supported within the specific market. This is the most subjective and
long term evaluation category, and should compare at a micro level the stated
vendor direction across the five categories documented above to the stated
goals and directions of the specific project teams making the selection
decision. At a macro level, any dissonance between stated vendor direction
and overall market direction should be a cause for great concern, and should
be quickly rectified by the vendor through either a shift in corporate policy
or a detailed and market validated explanation for the discord. Missionaries
rarely win the strategic battle, and continued discontinuity between a specific
vendor and the larger market direction will inevitably lead to a vendor
viability issue. Relative to the other five evaluation criteria, best practice
selections place between 5 and 10 percent of the overall selection importance
on the corporate strategy criterion.
of a logical, objective, structured process for technology selection has been
proven to reduce political agendas, enhance internal credibility, reduce the
time and cost of a technology selection, and increase the accuracy and overall
satisfaction of the final decision. The model documented above serves as the
basic foundation for our evaluation of specific technology vendor and products.
We recommend that any organization initiating a strategic technology evaluation
utilize a similar model to ensure proper coverage of the critical evaluation
categories necessary to ensure the ideal selection.