<
Mitigating Drawbacks Through The Use Of A Knowledge Base
Using
a knowledge base in the selection process can reduce the time, risk and cost
of procuring technology. With regard to technology selections, TEC describes
a knowledge base as having three components:
-
First, a comprehensive set of functional and technical criteria
covering every functional area of an enterprise application (in this case,
CRM).
-
Second, a set of vendor ratings for the criteria on the vast
majority of products on the market.
- Third,
a decision support tool to compare vendor offerings and quantify the gaps
between vendor capabilities and company requirements.
TEC
provides web-enabled versions of its knowledge bases available online.
Visit www.crmevaluation.com and www.erpevaluation.com
for more information.
Using the criteria in the knowledge base as the requirements for the RFI can significantly reduce the time associated with creating, issuing, and receiving RFIs back from vendors. This entire process can be reduced from months to weeks because the RFI responses are available immediately once the criteria in the knowledge base have been prioritized to match the company's requirements. A comprehensive knowledge base will eliminate the need to create, issue and receive RFIs back from vendors. Depending on the depth and breadth of the company's requirements, a comprehensive knowledge base will not only include all of the functionality and technology required, but can actually help identify requirements that would have otherwise been overlooked. Nonetheless, as stated above it is important to document the CRM vision, determine the business processes required to enable the vision and begin to determine the functionality and technology required to enable each of the business processes before creating an RFI or comparing vendors in a knowledge base.
A good knowledge base can also diminish data quality issues. Companies that focus on developing and maintaining knowledge bases typically validate the data prior to making it available. For example, the vendor responses in TEC's knowledge bases are vetted by independent third party analysts prior to being uploaded. This allows selection teams to make technology decisions without being concerned about the validity of the data. A typical selection data validation is usually cost or time prohibitive.
Using a knowledge base equipped with a decision support tool enables a sophisticated level of analysis that cannot easily be achieved with common office productivity software. Ranking, scoring and quantifying differences between vendors requires decision support software specifically designed analyze rated data.
To illustrate the value of knowledge base assisted selections we will compare the results of three vendors in TEC's knowledge base.
This
is Part Two of a two-part tutorial on CRM selection.
Part
One detailed the challenges faced by companies when selecting a CRM solution.
TEC's CRM Knowledge Base
The
criteria in the knowledge base are organized in a hierarchy. The CRM knowledge
base (found at www.crmevaluation.com)
contains over 2,000 functional and technical criteria. Figure 1 illustrates
the high level modules in the knowledge base. The Marketing Automation and Product
Technology modules have been expanded to view the sub-modules. Expanding the
sub-modules will reveal the end-level criteria that have vendor ratings.
Figure
1
Each level in the knowledge base, including the end-level criteria, can be prioritized to allow the selection team to put their organization's individual footprint on the model. Figure 2 illustrates priority settings for the high level modules. The pie chart above the priorities indicates the percent distribution of the decision. For example, given the current set of priorities Marketing Automation is responsible for 11% of the total decision. The knowledge base criteria are prioritized on a six point verbal scale ranging from Must Have to Not Important. The relative priorities dictate the percent distribution of the decision.
Figure
2
Figure 2 represents a baseline set of priorities where every criterion is given an equal weight. The results appear in Figure 3. The two metrics that appear in Figure 3 are calculated by TEC's patented BestMatch statistical decision support engine. The BestMatch engine converts the verbal priorities set in Figure 2 to a numerical weight. The sum of the numerical weights on the end level criteria total 100%. A vendor rating for each product in the knowledge base exists for each end level criterion. Figure 4 details the vendor ratings and numerical equivalents. The Weighted Average is the sum of the weight (x) the numerical equivalent of the rating for each end level criterion. The BestMatch Factor is a metric that uses patented algorithms to identify the risk associated with inconsistent vendor ratings across the model. The BestMatch Factor reduces the Weighted Average by an amount that is relative to the variability of the vendor's ratings across the model. For example, if two vendors have the same Weighted Average but Vendor A's ratings vary more than Vendor B, Vendor B will have a higher BestMatch Factor.
Figure
3
Figure
4
TEC's
CRM Knowledge Base (continued)
The
results in figure three illustrate the rank and score of the vendors for the
overall model. Figure 5 is a radar diagram indicating the rank and score of
the vendors for each of the sub modules. The radar diagram allows the selection
team to identify the relative strengths and weaknesses of the vendors. The graph
indicates that although Microsoft had the highest BestMatch Factor overall,
its scores were not the highest in four out of the nine sub modules in the model.
Exact Software took first place in CRM Analytics and e-CRM,
while Genesis Global Technologies took first place in Call
Center and Customer Service and Industry Vertical Module Availability. The CRM
Evaluation Center allows selection teams to drill into any area of the model
to compare vendor strengths and weaknesses.
Figure
5
Although
Microsoft CRM took fist place with baseline priorities, a few
changes to the priorities can change the rank and scores. Figure 6 indicates
a new set of priorities.
Figure
6
In this scenario Exact Software took first place in the overall model, as indicated in Figure 7. Selection teams can easily see how changes in their priorities impact the rank and scores of the vendors. This information is particularly useful when a selection team is trying to determine which vendors to include on the short list. A selection team's priorities will customize the knowledge base, resulting in a unique rank based on the priorities.
Figure
7
TEC's CRM Knowledge Base (continued)
Yet
another set of priorities will generate a different rank among the three vendors.
Figure 8 illustrates a new set of priorities. Figure 9 indicates that with the
new priorities Genesis Global Technologies takes first place while Microsoft,
which was first in the baseline model, is in third place.
Figure
8
Figure
9
The above comparisons involved changing the priorities at the high level, but priorities can be changed at any level in the model. Figure 10 illustrates the capability to modify detail priorities. The cookie crumbs in the upper left section of Figure 10 indicate the position of the criteria in the knowledge base.
Figure 10 also displays the capability to set thresholds. A threshold is a user-defined minimum requirement for an end-level criterion in the knowledge base. In this example a threshold of Supported is set to the criterion Multi-user scheduling. Note from Figure 4 that a rating of Supported indicates that the functionality is provided "out of the box." The capability to set thresholds is useful because it allows the selection team to set minimum requirements for important end-level criteria.
One of the challenges of working with a large knowledge base is that a selection team may find a specific end-level criterion to be extremely important, but because it lies four or five levels deep in the hierarchy it will only receive a fraction of a percent of the total decision even though it is given the highest priority. Thresholds are a useful way to identify each vendor's ability to meet the threshold regardless of the priority on the criterion.
Figure
10
Figures 11 and 12 indicate the outcome of the threshold. The results screen clearly indicates that within the section of the model that contains the criterion Multi-user scheduling, Microsoft is in first place, but a FAILED button appears next to the product name. Clicking on this button brings up a screen similar to Figure 12. This indicates that Microsoft failed because a threshold of Supported was set on the criterion and Microsoft's rating was Third Party Support.
Figure
11
Figure
12
Sensitivity Analysis
Other key features of the knowledge base include the ability to compare the strengths and weaknesses of the vendors, quantify gaps between the vendors' capabilities and the selection team's requirements and identify the sensitivity of the rank and scores of the vendors to changes in priorities. Figure 13 illustrates a sensitivity analysis for the module CRM Analytics. The graph answers the question, "How would a change in the relative importance of CRM Analytics impact the rank and scores of the three vendors?" The X-axis is Relative Importance, which is the percent of the total decision assigned to the selected criterion (in this case CRM Analytics). The current Relative Importance of CRM Analytics is 11.11%, as stated in parentheses below the X-axis. The Y-axis is the Weighted Average. The graph indicates that as the Relative Importance of CRM Analytics increases, the scores for Microsoft and Genesis Global Technologies decrease while the score for Exact Software increases. The graph also indicates that the rank of the vendors would change if the Relative Importance of CRM Analytics was increased to approximately 30%.
This graphing capability is useful for two reasons. First, it can distinguish criteria that significantly impact the decision by identifying areas of the knowledge base where either the lines cross and/or there are steep slopes. Second, in cases where the lines do not cross or there are shallow slopes, it can build confidence in the decision by indicating that even drastic changes in Relative Importance will not impact significantly impact the rank or scores of the vendors.
Figure
13
Conclusion
CRM projects can fail for many reasons, including a flawed selection procedure. Business initiatives should not only drive the CRM project, but should form the justification for each requirement listed on the RFI.
Evaluating vendors to generate the short list is one step in the selection process that if mishandled, can cause the CRM project to fail. Tools such as the CRM Evaluation Center can reduce the time risk and cost of many aspects of the technology procurement process and should be considered during appropriate stages of any large scale technology implementation. Procurement assistance can be especially valuable for companies that do not have the appropriate resources to dedicate to the selection project or the time to administer a traditional full-scale RFI with the appropriate vendors.
Please
send comments to ltalarico@technologyevaluation.com.