An Overview of the Knowledge Based Selection Process
Written By: B. Spencer
Published On: March 15 2001
An Overview of the Knowledge Based Selection Process
Spencer, J. Diezemann & J. Dowling
The TechnologyEvaluation.Com's (TEC) Knowledge Based Selection Process
is a multi-part comprehensive analysis of a vendor's tactical, strategic
and qualitative measures as compared to the specific business and technology
needs of our clients. The selection process encompasses nine process steps:
Assessment/Request for Information
first step in the selection process is to identify the capabilities required
to support our clients' business and translate those required capabilities
into a Request for Information. The RFI is then transmitted to a select
list of vendors who have been matched by TEC as a best fit to the specific
business needs of our clients.
goal of the internal capabilities assessment is to establish a set of
requirements or criteria that supports the business objectives. An average
technology evaluation takes into consideration only a small portion of
the necessary criteria needed to adequately evaluate a potential vendor.
By and large these criteria focus on functionality and technology and
are vital to a potential acquisition, but they are primarily tactical
this stage of the process, a TEC specialist walks the project team through
an overview of strategic market criteria such as the vendors' ongoing
service and support capabilities, their financial viability, and strategy
for long term growth and success. Additional qualitative measures are
identified to support market feedback from vendor client references, the
ease of use and navigation capability of the software, the level to which
the vendors proposed solution matches our client's business processes,
the vendor diligence level demonstrated throughout the process and the
product flexibility as perceived by the client.
process step allows the project team to concentrate on the totality of
the selection factors, not simply those with respect to product features
and functions. Vendor RFI responses are validated against market data
and entered into the decision-support tool for the initial results.
Conference / Vendor Discovery
After the vendor responses are received and analyzed, the top (typically
two or three) scoring vendors are established as the finalists and offered
the opportunity to participate in the remaining competitive processes.
Each finalist is invited to attend a Bidder's Conference where the client's
selection project team members present the unique operational environment
and challenges faced to achieve the results of the Capabilities Assessment.
plant tour is also arranged which provides the first real discovery opportunity
for the finalist vendors to learn the specific operating parameters and
challenges of the client. Vendors are also offered additional opportunities
for discovery to prepare for the Technical Review session and Scripted
During the scripted scenario preparation period, vendors are required
to present their technical solution to a predetermined list of criteria
developed collaboratively between TEC and the client. This process step
serves to validate specific elements contained within the Product Technology
section of the RFI and provides a detailed real time verification of our
client's architecture preferences. The areas addressed include:
of Integration among Modules
to integrate with other applications
Protection and Restoration
Further sources of information available to the project team are client
references. References are contacts in organizations that have previously
installed the software and can answer questions about the vendors' implementation
services, ongoing maintenance support, and quality of training. In addition,
though no two organizations have the same corporate strategy or business
requirements, client references can provide useful insights in terms of
strengths and weaknesses of the software, challenges they faced during
the implementation, organizational impact, and other issues. The project
team may visit the references in person or via a teleconference.
The Scripted Scenario process step is an opportunity to validate the Product
Functionality portion of the RFI through real business activities that
need to be supported by the technology. TEC collaborates with our client
to develop the scripted scenarios, which are a realization of business
processes, current or anticipated, resulting from the Capabilities Assessment.
Vendors present these business scenarios on their live products - tailored
to the way the organization does business as defined in the scenarios.
scenarios allow the organization to see how the live product operates
in its specific environment, according to the business processes outlined
by the project team. In addition, the project team gains an understanding
of the extent to which the vendor is willing to adjust the technology
for their needs.
completed scenarios are explained to the finalists' teams during discovery
sessions. The vendors are then given a period of time in which to prepare
for the scenario demonstrations. TEC develops and provides score sheets
to all project team members in attendance for use in rating each vendor's
performance. Typically the scenario demonstrations span three days with
the scores tabulated daily and entered into the decision support model.
for Quotation - Support & Analysis
TEC provides guidance and support in collaboration with our clients to
develop a detailed Request for Quotation (RFQ) for the software and services
desired. The RFQ is provided to the finalists (typically at the Bidder's
Conference) to allow them to prepare their cost proposal in parallel with
the other process activities. This timing provides the opportunity to
dovetail all the process elements together to permit a comprehensive analysis
of all factors that affect a recommendation without impacting the schedule.
responses to the RFQ are entered into the decision support model and analyzed.
Each finalist is compared relative to one another against the requirements
established for our client.
Analysis & Recommendation
The results collected throughout the process are entered into the decision
support model. TEC assists the client project team with the analysis,
and facilitation of changes to criteria weights. TEC leads the project
team through a sensitivity analysis to understand the impact of weight
changes to the criteria on the overall vendor scores. This process is
conducted exclusively using the decision-support tool, the trade-off analysis
capabilities and additional model variations developed by TEC.
Weighting Rating Differences Chart is used to show the relative value
amongst the factors for one or more vendors. Large values can indicate
a likelihood of rank change as the weight is modified in sensitivity analysis.
For the expert or consultant, it enables a scale consistency check. In
other words, one can check whether the rating given for one vendor against
the other for each factor makes sense. Drill down to one or more levels
in the model hierarchy is provided to perform more detailed rating comparison
and differences evaluations.
the collaborative analysis exercise, TEC has sufficient data as well as
a clear audit trail of due diligence to provide the recommended selection
decision which meets the Capabilities Assessment of the client.
Final Report captures and presents all data compiled at each process step
and serves as the audit document for due diligence. Each data element,
RFI, Scripted Scenario etc., are included as appendices to the report.
Each appendix item will include vendor responses and scores as appropriate.
recommendation is made, TEC provides the client with guidance and support
to put into action any pre-determined counter-proposal or negotiation
tactics. These steps may include a combination of the following: a request
to lower initial software costs, inclusion of free consulting or training
resources, reducing the scope of the services offered, or a decrease in
maintenance fees. At this point in the process, TEC provides advice regarding
negotiation tactics, possible areas of vendor weakness, and, if desired,
can personally negotiate with the vendors. This stage ensures that the
organization will receive the highest quality and quantity of vendor products
and services at the lowest price.
several distinct advantages in the way that they deliver Knowledge Based
selection managers. TEC analysts have extensive experience within
their given research area, and have lead numerous selection engagements.
These individuals have participated in several full cycle implementations,
giving them an invaluable hands-on understanding of the marketplace
as a whole.
TEC does not participate in the systems integration business or the
installation business. This provides a major advantage over other firms
such as the Big 5 (Accenture, PWC, D&T, CAPGemini E&Y, KPMG) or boutique-consulting
firms who, since they perform integration work, may have a vested interest
in which vendor is chosen. Typically, these organizations immediately
have available a short list of vendors who happen to comprise their
strategic partners. Thus, the decision is often made according to which
implementation resources the consulting firm has idle, and not based
on what is best for the client.
decision support tool. Developed in-house, TEC's decision support
tool breaks down a decision into its component pieces, and then uses
multi-attribute decision science to support a selection decision. This
stands in marked contrast to traditional selection decisions that rely
solely on gut feel, executive mandate, spreadsheet compilations, etc.
list of criteria. TEC's repositories contain a superset of criteria
our clients would want to use in the selection process. Some models
contain 10,000 to 11,000 detailed criteria that evaluate not only the
quality of the vendors functionality and technology, but also address
strategic criteria that indicate whether a vendor will be a reliable
partner for the life of the implemented software. These important and
often overlooked criteria include the vendor's quality of service and
support, financial viability, and corporate strategy. This wealth of
criteria ensures that clients do not have to "reinvent the wheel" when
drawing up an RFI, RFP or RFQ. In addition, it enables the project team
to complete the internal needs assessment in a period of weeks as opposed
to months, once again cutting down the time and cost of the selection