A Case Study and Tutorial in Using IT Knowledge Based Tools Part 2: A Tutorial

Part 2: A Tutorial
E. Robins - May 31, 2001

Executive Summary   

Most business managers, whether vendors, vendor clients or implementers, are unaware of the fundamental capabilities that knowledge based decision support can provide to minimize project risk for all sides of technology utilization. Given that over 90% of IT projects fail on first attempt, according to the Standish Group, more thought and research on the evaluation and selection process is needed. Many of these failures - some 30% - are the direct result of poor selections, and represent upward of $30 billion in wasted investment annually.

On the vendor side, the challenge of educating the potential client of their offerings results in long sales cycles, meticulous and numerous RFI responses, and potential for a mismatch, result in projects that go awry. These failed projects do not bode well for the vendor, since the sales cycle costs can only rise, and their reputation can suffer. Consequences can be more severe for the client where it can, in extreme cases, lead to business failure. Implementers (which can be internal IT departments as well as consultants) can also find that decision-maker indecision leads to lengthened sales cycles, missed opportunities, and risk of competitive intrusion. The root cause of this indecision is an inability of the implementer to give confidence to the stakeholders of their choice of solution.

All sides need thought and research to build data and process information in a meaningful context, which takes time and costs money for all participants going through the selection process. But without spending time, thought, research and money there is increased business risk to all.

To cut away from this devil-and-the-deep-blue-sea conundrum means looking under the hood of evaluation and selection practices, to determine if there are better ways of moving through them. There is certainly room to ask the fundamental question of whether the current practice of RFI / RFP processes, among other internal organizational procedures, are adequate to the task of selecting complex systems? The record indicates there is much room for improvement.

In essence, for complex selections, the human-machine combination has to work together to drive the solution. Both have to be understood and complement each other in the process. It is easy for the human to be overwhelmed, or simply run out of time, and the machine interface and engine to be inadequate to the task. However, the results must benefit the process if human and machine can function effectively together to process information and avoid the pitfalls of past selection processes.

In this, the second part of this article, we shall follow a simplified process as an illustration. The method was used by the author to conduct a selection on a Personal Digital Assistant (PDA). In this case, the end result was the purchase of the suggested item! The first part, is an overview of decision support systems and knowledge based selection.

About This Note: This is a two part note where Part 1 is a discussion of the use of an IT Knowledge Based selection tool as part of a Decision Support System selection process. This second part of our two-part article is a tutorial with which illustrates using such a system to select a Personal Digital Assistant (PDA).

Though a PDA is far less complex than, for example, an ERP system, processes and procedures enabling narrowing down of solutions, and avoiding dissatisfaction, while taking on assessed risks, are part of process in Knowledge Based Selection methods.

A Selection as a Tutorial   

In this article, we shall follow a simplified Knowledge Based selection process as an illustration.

To follow this process as a tutorial, you should go to the WebTESS 2.0 website, and bring up the PDA knowledge base on a separate browser window.

Click here to launch webTESS

Note, however, that the author also invoked TESS to explore more sophisticated analysis and tradeoffs. TESS is TEC's full-featured Knowledge Based system designed for major complex selections involving hundreds and even thousands of selection criteria of many different levels of criticality.

Value Trees   

Value trees, simply put, provide a measure of value of a solution against your business requirements. They generally consist of criteria arranged into hierarchical trees, much like directories on a hard disk - and not too differently related. Files in a folder are related by implication to the name of the folder, for example. The criteria are prioritized according to the main goal, and alternatives are rated against the criteria at the lowest level of the hierarchy. WebTESS 2.0 Knowledge Bases (KBs) are arranged in this way.

(Author note: The criteria can or should represent variously business objectives, business capabilities, processes, and end criteria which can represent features and functions, or measurable items related to other issues of the vendor-client relationship, including project management and vendor viability issues(for example). In our case the relationship is limited in the KB to availability and warranty concerns only, partly to simplify the issue, and partly because the selection of a single PDA did not depend on how long the company is likely to be around - at least, the significance of this to the decision was not a significant or a major cause for differentiation)

As a noted in part 1, most IT selection tools on the web contain only "features and functions" criteria. For many complex decisions, this can be inadequate.

After selecting the PDA KB, you will be placed in a window entitled "Summary". There are four sections to this window. On the left side is the decision hierarchy - in fact the value tree of criteria. The top three criteria are "Features", "PDA Configuration", and "Other Product and Manufacturing Considerations", and these criteria are what you first see in the tree. You can view deeper levels in the tree by clicking on the [+] boxes on the left of each criterion. Note the initial focus is on the top criterion (or root) of the tree - "Selection of a PDA". The root name is repeated on the right hand side, above the Criteria label. To the right of the tree is a list of criteria, and input boxes under a title "Relative Priorities". A pie graph gives feedback to the relative priority distribution further to the right. This constitutes the second area of the Summary window.

Relative Priorities   

On the simplest level, Relative Priorities are a way of inputting ones sense of the relative "level of importance" of the criteria in the decision, capturing your business perspective of priorities. These relative priorities can be changed to reflect your sense of relative need for each of the criteria. We shall discuss this more below.

Generally, the first round in our battle is to narrow the number of alternative solutions. Further down in the mid-right of the Summary window is the list of alternative PDAs in the KB, an area that is the third part of this window. (Note: the full TEC KB has well over 80 PDA models). You can add or remove each of the candidate PDAs by clicking on the ADD/REMOVE button, and get information about them by clicking on the hyperlinked names in the Summary window. There are hyperlinks to one or two sites below the descriptive text of the pop-up information window - the vendor site and/or epinions.com. Epionions.com lets product users (the "experts") provide product reviews.

This 'winnowing' or short-listing process is using typical descriptive information to remove from the mix the choices that obviously will not match. In order for this to be adequate and reliable, information needs to be present, presented in a way that enables a quick elimination. Side-by-side table constructs can help, but can also confuse, and are not necessary for this first step. To filter a few out of a large number of choices, however, can require better tools, as we shall see below.

From this process, I have narrowed down the choice to the iPAQ, Jornada, Palm IIIc, and Casiopeia EM-115. I have a penchant for a color screen, and hence rejected the Aero 1550 as the cheaper monochrome alternative. However, let us add the Aero to our shortlist for a demonstration below.

The fourth area of the window - at the bottom - shows the results of the evaluation as a bar graph.

Once the candidates have been narrowed down, you need to insert business requirements. There are two steps generally related to this part of the process, exhibited in the Prioritize tab, one related to performance requirements, and a second to the priorities you assign to the capabilities you need. This will become clearer as we follow the example in the Prioritize window.

In the Prioritize window, from the tree on the left, drill down under "PDA Configuration" and select "Display". You will note in the sub-criteria table on the right hand side of the window the Threshold "Set" button for each criterion. In the criterion "Color" row, click the Threshold Set button to bring up the Threshold form. You can now set the minimum performance requirement for Color as "Yes" using the drop down. Pressing "Submit" causes the decision engine to validate all the alternative solutions against this requirement. If you now go to the Results window, you will find a "FAILED" button for the Aero 1550, and indeed pressing on it reveals the color issue with the Aero. In this case this quickly eliminates the Aero. In other cases, it is an issue to be raised with the vendor if other considerations lead one to believe it is a viable solution. These issues can be prioritized on the basis of priorities you assign.

Note that thresholds are intended to ensure minimum performance standards are met. This is different from setting priorities. Priorities enable you to express the relative significance of your criteria: i.e. some are more significant than others, and represent choices of what you prefer to see in the solution compared to another criterion. For example, I would prefer my basic functionalities to be met, more than worrying about manufacturer warranties or replacement programs. Priorities do not eliminate on the basis of performance as thresholds enable you to do.

You can continue to define your own priorities and thresholds in the Prioritize form. You may note that unless price is taken into consideration, the iPAQ is likely to win every time. This is a case of general product passing over niche players like the Casiopea EM-500, which is intended for a more multimedia and different audience.

Differentiate the Lead Solutions   

I am going to cheat a little and invoke some of TESS's graphics. Figure 1 shows the picture of my own priorities, taken to the second level of the decision hierarchy tree (you can also sort criteria by order of priority, but it is shown in the order of the hierarchy here).

Figure 1. Criteria Priorities for my requirements.

Assuming I am satisfied by this priority layout, I can ask further questions such as would the decision change if I excluded low priority criteria? Figure 2 illustrates the TESS response to this question. In the spider diagram, criteria decline in priority going clockwise from the top. The light blue dashed line indicates the cumulative priorities of the criteria, going from about 15%+ each for "Basic features" and "PC Synchronization" (note these two account for 30%+ of the full decision - see Figure 1), down to the lowest impacting criterion of Product Manufacturer (they're all good brands and this factor did not seem a strong differentiator for me as I indicated earlier). Given the priority setup, the iPAQ has the highest weighted average value all the way round, and the Jornada only achieves near-parity when the lower priority criteria are included. I can interpret this as the iPAQ having more of what is important to me.

Figure 2. How the decision appears when
lower priority criteria are neglected.

At 11 o'clock in the spider, all the criteria are included in the decision. Going counterclockwise, more criteria are excluded. At twelve o'clock, we're basing our decision solely on "Basic Supported Features". At two o'clock, the decision is based on three criteria - "Basic Supported Features", "PC Synchronization", and "Advanced Features".

I can ask the question whether a change in priority of any of the criteria would lead to a different solution, and by how much would I need to change the priority. To answer this question I have to focus on the criteria with which there is an issue in the iPAQ solution (i.e. where it is weak, and where other solutions offer an advantage). Another spider diagram helps me - it looks at the value (score) for each of the second level criteria (Figure 3). The major iPAQ concern to me is "Product Availability". This indeed proved to be my biggest problem. In WebTESS you can go the Summary window, and change the priorities (with a little effort) to see the impact on results. TESS gives a neater solution, so I cheat here again. In Figure 4 is the Sensitivity graph from TESS.

The plot is Percent Match (a special algorithm TEC has developed to give better measure of selection risk) against the priority weight. The numbers above each line indicate the rank of the PDA solutions. Where the lines cross, the ranking of the PDAs change.

I have selected "Product Availability" as the variable for priority, and can see that it would not take too much to swing the decision to the Jornada. This poses a problem to me. However, as I've gone without a PDA for most of the years of my life, a little more waiting is not likely to kill me, in part reflected by the low priority I have given availability. I also reason if I am to get one, it had better be good. I decide to take the risk, as the rest of the evidence favors the iPAQ. Note the Cassiopeia E-115 has not been a contestant. My priorities are not set up for multimedia stuff, and there are things lacking in it for me, but not critical things. Perhaps this would be my final fall back position.

Figure 3. Value of each factor (iPAQ is in red).
Note the low value for "Product Availability"

Figure 4. The priority sensitivity of the decision to
"Product Availability"

click here to view larger version

What About ROI?   

What about the ROI? From TESS I can do a trick and examine the value returned per dollar spent. More than that, I can include all costs (my "TCO"), and compare the ROI's. In Figure 5, the total expected cost is measured for each PDA in my shortlist on the y-axis. Interestingly, the Jornada has the best ROI, and would "win" in this scenario. However, the chart also tells me from the table at the bottom of the figure that a cost reduction of $44 - some 8% - would give me the same ROI for the iPAQ. Unfortunately, my sway with Priceline.com - where I eventually bought the product and accessories for $499.95 - was inadequate on this basis since the price would have had to be down to $470. However, as the price was only $29.95 more, this was not that enormous to cause me to rethink my decision, so I kept to my guns about the iPAQ.

Figure 5. Value equivalency of the iPAQ solution
to the best ROI PDA (Jornada).

The shaded area at the top of the bars indicates the cost reduction
to make it the same in ROI as the Jornada.

Conclusion   

From a business perspective, the use of Knowledge Based Selection highlights the fact that technology and human decision procedures should be integrated as a single system in the process of complex technology or other selections. The technology should be able to identify issues, create better fits for vendors and clients, and speed decisions. It should not take businesses eighteen months to reach good decisions in technology selections, particularly as information becomes quickly outdated.

Necessity is the mother of invention. I created the PDA KB for purely selfish reasons - I wanted a PDA after seeing how useful it was for my wife (my "competition") to have one (she is a senior manager in a leading Telecommunications company which gave it to her as a personal organizer). The trouble was that I knew nothing about them, and shopping around the web merely overloaded me with data. The net time in collecting data and building the KB was about three days, and the best fit became immediately apparent, though it upped the ticket I was willing to pay. The rest, as indicated above, is history.

On a general level, the process I invoked from literally zero knowledge to a decision went through a number of steps - most of which can be accomplished in WebTESS, though more sophisticated analysis and tradeoffs were done with TESS (my advantage!). However, I could remove unacceptable alternatives rapidly, and drive to a decision, aware of the risks and costs. Figure 6 illustrates the steps in the decision process.

Figure 6. Steps in the PDA selection process.

Additional use can be made of expert knowledge. For example, in the WebTESS PDA KB, under the criteria describing warranty, details of the warranty contract can be found in WebTESS PDA ratings (check it out under the criterion "Other Product and Manufacturer Considerations"). These small details can make significant differences to the expectations of users. The length of warranties, and what is included in them, as well as how service is delivered, are other vital pieces of information - particularly for the traveling businessman - that can make decisions less rocky than otherwise. Or at least the risks have been weighed, and you have no one to blame but yourself

Further, the direct verbal statement of the warranty and associated conditions can be customized in terms of the value proposition to each stakeholder (in other words, a number must somehow be associated with each warranty option for calculation purposes). Many systems miss the point and assume 'one size fits all' which can be a mistake, potentially leading to poor matches. There are practical considerations of course, since one cannot do this for large numbers of criteria, and a judgment call is required of where to draw the line between time, complexity and accuracy. Having an effective evaluation tool, and a process with an experienced analyst capable of vetting data and value provides better protection against methods that can miss the mark.

The PDA KB can be made more complex, of course, particularly if it is intended for corporate wide selection. Wireless support, availability of applications, and communication compatibility may play a major part, particularly in data exchange issues. For example, I discovered Windows NT 4.0 does not support USB and infra-red synchronization, which meant having to pay for an additional serial link cable. As knowledge builds, this kind of information can be easily captured in a knowledge base for future users. Hopefully they will then avoid the pitfall, though it would not have changed my decision - simply let me understand the additional cost involved in supporting both USB and serial synchronization, or maybe identify the need to change my operating system. As a final note, I got my iPAQ at the limit of my patience. The risks of waiting paid off, and I am happy to report I am pleased with my purchase. I found some opinions from Epinions.com as not particularly valid - such as a gentleman who described the painful experience he had when the pen would jam frequently in the slot in the casing: after a week of use, I have had no such problem, except if I were silly enough to put it in the wrong way! I admit I was near the end of my patience in waiting - some two months - and almost was at the point of canceling and going for the Jornada. However, I admit to not having checked the availability of the Jornada lately

As a note, so far, I'm happy. My iPAQ and I are doing well, particularly as my wife is envious when she looks at her Palm Vx next to mine.