Most potential customers understand that detailed scripted scenarios should
be performed with software vendors who are on the short list of candidates,
in order to evaluate the functional abilities of products being considered.
Unfortunately, many companies neglect to investigate the technological
underpinnings of these products during the evaluation phase. This can
lead to the purchase of a product which is functionally excellent, but
difficult (or impossible) to support in the customer's environment.
document is designed to help craft a series of "Technical Architecture"
meetings with vendors that can begin to explain the "how" of a vendor's
solution, not just the final functional result. If the customer follows
TEC's methodology for functional scripted scenarios and a vendor is still
in the running, the next step must be taken to ensure a smooth product
implementation and effective on-going support in the customer's environment.
Be Prepared to Discuss the Right Subject:
must be stressed to the vendor that this phase of the software selection
process is designed to evaluate technical (i.e., does the server run on
Windows NT, is it a 2-tier or a 3-tier architecture) issues, not functional
ones (i.e., how is payroll processed). The vendor should be provided with
a list of questions in advance and given adequate time to prepare their
responses. Someone in the organization should be designated to get clarifications
for the vendor where required, before the on-site meeting.
Evaluation Criteria (with examples; detailed criteria are provided
in TEC's selection model):
processing through application tiers, middleware and messaging components,
Protection & Restoration
and restore methodologies, archiving.
authentication and authorization, transaction and database security,
directory services support.
session control, print spooling.
such as performance degradation when multiple application windows
are opened on the same workstation, thread control between tiers
of the application, network bandwidth and latency issues.
for ad hoc reporting, third party integration, and custom report
for Client-developed Applications
and control mechanisms, configuration management, methodologies.
demonstration of selected features
vendor should be tasked with demonstrating the most important technical
features (i.e., demonstrate assignment of security to a user, monitoring
application performance, etc.) and notified in advance and in detail
as to what must be shown. For tips on how to make this process successful,
Post-Mortem: Why Vendors Lose Deals". This document discusses
functional demonstrations, but the concepts remain the same.
Bring the right people:
most important criteria for the success of a technical architecture software
evaluation is the presence and participation of individuals qualified
to hold the discussion. Once the proper subject areas have been determined,
it is imperative that the customer provide a "jury" of individuals in
the proper disciplines to ensure that subjects are properly covered. These
individuals must be tasked with doing the background work to understand
what the vendor is proposing, and fit that proposal into the "template"
of their unique environment.
need not necessarily be a single individual for each of these areas. Some
employees will be able to speak to multiple subjects.
both the customer and the vendor do not send qualified individuals
to the meetings, this effort will be severely hampered.
Participants: (with examples of some roles they may fulfill)
individual responsible for the overall coordination of the "jury".
This individual tallies ratings from the other participants and
collates a collective score for each function point. This individual
is usually also responsible for reporting progress to, and getting
clarifications from, higher levels of management within the company.
address issues such as bandwidth requirements, network protocols.
discuss needs for specific databases (e.g., product only runs on
Microsoft SQL Server).
of platforms, disk space usage, CPU requirements.
implications of LDAP (Lightweight Directory Access Protocol) support,
single sign-on, role-based access.
availability of vendor published API to enable custom enhancements/extensions.
web server and browser requirements, need for active server pages
or other software dependencies, etc.
individual may not understand much of the technical discussion,
but is an invaluable aid in helping the group to understand the
If everyone involved is engaged and the proper preparation has been done,
technical architecture discussions can be extremely fruitful. Various
architectural features of the products can be compared to each other and
contrasted with those from other vendors. TEC has a Technical Architecture
Practice with experience in addressing these issues and following up with
vendors on unresolved questions. Some vendors have been surprised to see
that customers are interested in "how things work", but soon realize that
it is to everyone's advantage and that a full understanding by all parties
will help ensure a smooth rollout, implementation, and on-going support
of the product. A good "out of box" experience is best for all concerned.