Do You Know How to Evaluate Your Strategic Technology Provider?

Introduction

Issue: Project teams constantly face a barrage of new products and technologies, and have a difficult time differentiating marketing slides and grand promises from deliverable products when making strategic IT acquisitions. Flat IS budgets and limited internal IS resources further exacerbate the selection problem.

Problem: Project teams state that they struggle with a number of critical issues when selecting enterprise technologies:

1. Project teams have no effective way to identify the critical vendor and product questions necessary to successfully initiate the evaluation process.

2. Project teams have no ability to effectively prioritize the different criteria, once identified, relative to one another. As a result, final priorities are often more the result of internal political agendas than true needs and requirements.

3. Project teams have no ability to gather objective, validated, updated data on the available vendor alternatives. It is a well-known problem that vendors have a tendency to exaggerate product, service and corporate capabilities if it enables them to move to the next phase of the deal. Unfortunately, most project teams have no true ability to separate fact from hype, especially because its high-end, strategic technology selections are often either the first of its kind or the first in an extended period within a specific organization.

According to TEC research, the net result has been that over 80% of enterprise technology evaluations run over time and budget, and that once selected, over 50% of the implementations fail to meet functional and total cost expectations.

Solution: The solution is to create a structured, repeatable process for evaluating technology solutions and the vendors that provide them. Best practices drawn from TEC client organizations that have completed internal technology selections suggest that project teams should examine six key criteria groupings. The first three criteria sets should examine product specific capabilities, while the second three should investigate the software vendor's overall corporate capabilities. TEC has found that organizations that experience the most successful selections have placed almost an equal (approximately 50 percent each) overall priority on the tactical and strategic criteria documented below.

Tactical Product Evaluation Criteria:

1. Product Functionality
   
   This is the most obvious product evaluation criterion and the favorite of technophiles worldwide. Simply put, this evaluates the features and functions delivered by the product as it exists today, and together with product architecture, often incorrectly makes up over 90 percent of the overall selection importance within IT product selections. Product functionality sub-components are usually defined in one of two ways:

   1. Functional groupings: within enterprise management technologies, these categories could include enterprise management console functionality, user administration, asset and inventory tracking, electronic software distribution, server management and monitoring, job scheduling, backup and recovery, enterprise service desk and enterprise security.

   2. Process groupings: within user administration technologies, this could include profile development, profile administration, profile distribution and profile manipulation.

   Relative to the other five evaluation criteria, best practice selections place approximately 25 percent of the overall selection importance on the product functionality criterion.

2. Product Technology
   
   This criterion defines the technical architecture of the product, and the technological environment in which the product can run successfully. Subcriteria include product and application architecture, software usability and administration, platform and database support, application standards support, communications and protocol support and integration capabilities. Relative to the other five evaluation criteria, best practice selections place a lower relative importance, approximately 15 percent, on the product technology criterion. However, this apparently lower importance is deceptive, because the product technology criterion usually houses the majority of the selecting organization's mandatory criteria, which usually include server, client, protocol and database support, application scalability and other architectural capabilities. The definition of mandatory criteria within this set often allows the client to quickly narrow the long list of potential vendors to a short list of applicable solutions that pass muster relative to the most basic mandatory selection criteria.

3. Product Cost
   
   This section, often the criterion most closely scrutinized by project teams that make sub-optimal product selections, should examine the initial product acquisition cost relative to its peers in a series of specific, predefined acquisition scenarios. In addition, the IS organization should understand longer term costs, including maintenance fees, upgrade costs, training and implementation costs, and service and support fees. Traditionally, the most successful project teams eliminate cost from the initial evaluation steps, focusing first on mandatory functional and technical criteria, as well as the strategic vendor evaluation criteria documented below. Only after a short list has been defined do the most successful organizations reinsert the cost criteria into the decision, using relative cost differences between products for negotiating leverage during the final selection phases. Relative to the other five evaluation criteria, best practice selections place approximately 10 percent of the final selection importance on the product cost criterion.

Strategic Vendor Evaluation Criteria

4. Corporate Service and Support
   
   This criterion defines the capability of the vendor to provide a high level of implementation services and ongoing support. Repeated industry surveys have identified this parent node as the single largest differentiating factor among potential selection options, as well as the greatest indicator of ultimate user implementation success and long term vendor viability. It therefore represents a critical evaluation category for strategic IT selections. A proper professional services and support evaluation should include both subjective, quality measures validated by current product users, and objective, quantitative criteria within both the professional services and product support categories. Within the professional services category, project teams should examine vendor consulting services, systems integration capabilities and project management skills. Product support should evaluate characteristics such as the geographic, language and time coverage of the vendor help desk, help desk service quality, and number, quantity and scope of vendor training classes. Relative to the other five evaluation criteria, best practice selections place approximately 20 percent of the overall selection importance on the service and support criterion.

5. Corporate Viability
   
   This is a critical yet often overlooked category that should examine the financial and management strength of the vendor. Given the huge dollars spent upon and strategic importance applied to most major IT procurements, the financial stability of the vendor supplying the product cannot be overlooked. The vendor viability evaluation section should combine quantitative Wall Street ratio and metric analysis with qualitative management and corporate evaluations. Only by combining the two components can executives accurately assess the risk and benefit of corporate investment in a specific product and vendor. At a minimum, the corporate viability criterion should evaluate the overall financial viability of the vendor, its macro and micro market viability, its sales and marketing viability, its management viability, and its research and development viability. Relative to the other five evaluation criteria, best practice selections place approximately 20 percent of the overall selection importance on the corporate viability criterion.

6. Corporate Strategy
   
   This evaluates the corporate road map and strategy of the software vendor with specific timelines regarding how the product will be developed, sold, and supported within the specific market. This is the most subjective and long term evaluation category, and should compare at a micro level the stated vendor direction across the five categories documented above to the stated goals and directions of the specific project teams making the selection decision. At a macro level, any dissonance between stated vendor direction and overall market direction should be a cause for great concern, and should be quickly rectified by the vendor through either a shift in corporate policy or a detailed and market validated explanation for the discord. Missionaries rarely win the strategic battle, and continued discontinuity between a specific vendor and the larger market direction will inevitably lead to a vendor viability issue. Relative to the other five evaluation criteria, best practice selections place between 5 and 10 percent of the overall selection importance on the corporate strategy criterion.

Value

Utilization of a logical, objective, structured process for technology selection has been proven to reduce political agendas, enhance internal credibility, reduce the time and cost of a technology selection, and increase the accuracy and overall satisfaction of the final decision. The model documented above serves as the basic foundation for our evaluation of specific technology vendor and products. We recommend that any organization initiating a strategic technology evaluation utilize a similar model to ensure proper coverage of the critical evaluation categories necessary to ensure the ideal selection.