A Case Study and Tutorial in Using IT Knowledge Based Tools
2: A Tutorial
Most business managers, whether vendors, vendor clients or implementers,
are unaware of the fundamental capabilities that knowledge based decision
support can provide to minimize project risk for all sides of technology
utilization. Given that over 90% of IT projects fail on first attempt,
according to the Standish Group, more thought and research on the evaluation
and selection process is needed. Many of these failures - some 30% - are
the direct result of poor selections, and represent upward of $30 billion
in wasted investment annually.
the vendor side, the challenge of educating the potential client of their
offerings results in long sales cycles, meticulous and numerous RFI responses,
and potential for a mismatch, result in projects that go awry. These failed
projects do not bode well for the vendor, since the sales cycle costs
can only rise, and their reputation can suffer. Consequences can be more
severe for the client where it can, in extreme cases, lead to business
failure. Implementers (which can be internal IT departments as well as
consultants) can also find that decision-maker indecision leads to lengthened
sales cycles, missed opportunities, and risk of competitive intrusion.
The root cause of this indecision is an inability of the implementer to
give confidence to the stakeholders of their choice of solution.
sides need thought and research to build data and process information
in a meaningful context, which takes time and costs money for all participants
going through the selection process. But without spending time, thought,
research and money there is increased business risk to all.
cut away from this devil-and-the-deep-blue-sea conundrum means looking
under the hood of evaluation and selection practices, to determine if
there are better ways of moving through them. There is certainly room
to ask the fundamental question of whether the current practice of RFI
/ RFP processes, among other internal organizational procedures, are adequate
to the task of selecting complex systems? The record indicates there is
much room for improvement.
essence, for complex selections, the human-machine combination has to
work together to drive the solution. Both have to be understood and complement
each other in the process. It is easy for the human to be overwhelmed,
or simply run out of time, and the machine interface and engine to be
inadequate to the task. However, the results must benefit the process
if human and machine can function effectively together to process information
and avoid the pitfalls of past selection processes.
this, the second part of this article, we shall follow a simplified process
as an illustration. The method was used by the author to conduct a selection
on a Personal Digital Assistant (PDA). In this case, the end result was
the purchase of the suggested item! The first part, is an overview of
decision support systems and knowledge based selection.
This Note: This is a two part note where Part
1 is a discussion of the use of an IT Knowledge Based selection tool
as part of a Decision Support System selection process. This second part
of our two-part article is a tutorial with which illustrates using such
a system to select a Personal Digital Assistant (PDA).
a PDA is far less complex than, for example, an ERP system, processes
and procedures enabling narrowing down of solutions, and avoiding dissatisfaction,
while taking on assessed risks, are part of process in Knowledge Based
Selection as a Tutorial
In this article, we shall follow a simplified Knowledge Based selection
process as an illustration.
follow this process as a tutorial, you should go to the WebTESS 2.0 website,
and bring up the PDA knowledge base on a separate browser window.
here to launch webTESS
however, that the author also invoked TESS to explore more sophisticated
analysis and tradeoffs. TESS is TEC's full-featured Knowledge Based system
designed for major complex selections involving hundreds and even thousands
of selection criteria of many different levels of criticality.
Value trees, simply put, provide a measure of value of a solution against
your business requirements. They generally consist of criteria arranged
into hierarchical trees, much like directories on a hard disk - and not
too differently related. Files in a folder are related by implication
to the name of the folder, for example. The criteria are prioritized according
to the main goal, and alternatives are rated against the criteria at the
lowest level of the hierarchy. WebTESS 2.0 Knowledge Bases (KBs) are arranged
in this way.
note: The criteria can or should represent variously business objectives,
business capabilities, processes, and end criteria which can represent
features and functions, or measurable items related to other issues of
the vendor-client relationship, including project management and vendor
viability issues(for example). In our case the relationship is limited
in the KB to availability and warranty concerns only, partly to simplify
the issue, and partly because the selection of a single PDA did not depend
on how long the company is likely to be around - at least, the significance
of this to the decision was not a significant or a major cause for differentiation)
a noted in part 1, most IT selection tools on the web contain only "features
and functions" criteria. For many complex decisions, this can be inadequate.
selecting the PDA KB, you will be placed in a window entitled "Summary".
There are four sections to this window. On the left side is the decision
hierarchy - in fact the value tree of criteria. The top three criteria
are "Features", "PDA Configuration", and "Other Product and Manufacturing
Considerations", and these criteria are what you first see in the tree.
You can view deeper levels in the tree by clicking on the [+] boxes on
the left of each criterion. Note the initial focus is on the top criterion
(or root) of the tree - "Selection of a PDA". The root name is repeated
on the right hand side, above the Criteria label. To the right of the
tree is a list of criteria, and input boxes under a title "Relative Priorities".
A pie graph gives feedback to the relative priority distribution further
to the right. This constitutes the second area of the Summary window.
On the simplest level, Relative Priorities are a way of inputting ones
sense of the relative "level of importance" of the criteria in the decision,
capturing your business perspective of priorities. These relative priorities
can be changed to reflect your sense of relative need for each of the
criteria. We shall discuss this more below.
the first round in our battle is to narrow the number of alternative solutions.
Further down in the mid-right of the Summary window is the list of alternative
PDAs in the KB, an area that is the third part of this window. (Note:
the full TEC KB has well over 80 PDA models). You can add or remove each
of the candidate PDAs by clicking on the ADD/REMOVE button, and get information
about them by clicking on the hyperlinked names in the Summary window.
There are hyperlinks to one or two sites below the descriptive text of
the pop-up information window - the vendor site and/or epinions.com. Epionions.com
lets product users (the "experts") provide product reviews.
'winnowing' or short-listing process is using typical descriptive information
to remove from the mix the choices that obviously will not match. In order
for this to be adequate and reliable, information needs to be present,
presented in a way that enables a quick elimination. Side-by-side table
constructs can help, but can also confuse, and are not necessary for this
first step. To filter a few out of a large number of choices, however,
can require better tools, as we shall see below.
this process, I have narrowed down the choice to the iPAQ, Jornada, Palm
IIIc, and Casiopeia EM-115. I have a penchant for a color screen, and
hence rejected the Aero 1550 as the cheaper monochrome alternative. However,
let us add the Aero to our shortlist for a demonstration below.
fourth area of the window - at the bottom - shows the results of the evaluation
as a bar graph.
the candidates have been narrowed down, you need to insert business requirements.
There are two steps generally related to this part of the process, exhibited
in the Prioritize tab, one related to performance requirements, and a
second to the priorities you assign to the capabilities you need. This
will become clearer as we follow the example in the Prioritize window.
the Prioritize window, from the tree on the left, drill down under "PDA
Configuration" and select "Display". You will note in the sub-criteria
table on the right hand side of the window the Threshold "Set" button
for each criterion. In the criterion "Color" row, click the Threshold
Set button to bring up the Threshold form. You can now set the minimum
performance requirement for Color as "Yes" using the drop down. Pressing
"Submit" causes the decision engine to validate all the alternative solutions
against this requirement. If you now go to the Results window, you will
find a "FAILED" button for the Aero 1550, and indeed pressing on it reveals
the color issue with the Aero. In this case this quickly eliminates the
Aero. In other cases, it is an issue to be raised with the vendor if other
considerations lead one to believe it is a viable solution. These issues
can be prioritized on the basis of priorities you assign.
that thresholds are intended to ensure minimum performance standards are
met. This is different from setting priorities. Priorities enable you
to express the relative significance of your criteria: i.e. some are more
significant than others, and represent choices of what you prefer to see
in the solution compared to another criterion. For example, I would prefer
my basic functionalities to be met, more than worrying about manufacturer
warranties or replacement programs. Priorities do not eliminate on the
basis of performance as thresholds enable you to do.
can continue to define your own priorities and thresholds in the Prioritize
form. You may note that unless price is taken into consideration, the
iPAQ is likely to win every time. This is a case of general product passing
over niche players like the Casiopea EM-500, which is intended for a more
multimedia and different audience.
the Lead Solutions
I am going to cheat a little and invoke some of TESS's graphics. Figure
1 shows the picture of my own priorities, taken to the second level of
the decision hierarchy tree (you can also sort criteria by order of priority,
but it is shown in the order of the hierarchy here).
1. Criteria Priorities for my requirements.
I am satisfied by this priority layout, I can ask further questions such
as would the decision change if I excluded low priority criteria? Figure
2 illustrates the TESS response to this question. In the spider diagram,
criteria decline in priority going clockwise from the top. The light blue
dashed line indicates the cumulative priorities of the criteria, going
from about 15%+ each for "Basic features" and "PC Synchronization" (note
these two account for 30%+ of the full decision - see Figure 1), down
to the lowest impacting criterion of Product Manufacturer (they're all
good brands and this factor did not seem a strong differentiator for me
as I indicated earlier). Given the priority setup, the iPAQ has the highest
weighted average value all the way round, and the Jornada only achieves
near-parity when the lower priority criteria are included. I can interpret
this as the iPAQ having more of what is important to me.
2. How the decision appears when
lower priority criteria are neglected.
11 o'clock in the spider, all the criteria are included in the decision.
Going counterclockwise, more criteria are excluded. At twelve o'clock,
we're basing our decision solely on "Basic Supported Features". At two
o'clock, the decision is based on three criteria - "Basic Supported Features",
"PC Synchronization", and "Advanced Features".
can ask the question whether a change in priority of any of the criteria
would lead to a different solution, and by how much would I need to change
the priority. To answer this question I have to focus on the criteria
with which there is an issue in the iPAQ solution (i.e. where it is weak,
and where other solutions offer an advantage). Another spider diagram
helps me - it looks at the value (score) for each of the second level
criteria (Figure 3). The major iPAQ concern to me is "Product Availability".
This indeed proved to be my biggest problem. In WebTESS you can go the
Summary window, and change the priorities (with a little effort) to see
the impact on results. TESS gives a neater solution, so I cheat here again.
In Figure 4 is the Sensitivity graph from TESS.
plot is Percent Match (a special algorithm TEC has developed to give better
measure of selection risk) against the priority weight. The numbers above
each line indicate the rank of the PDA solutions. Where the lines cross,
the ranking of the PDAs change.
have selected "Product Availability" as the variable for priority, and
can see that it would not take too much to swing the decision to the Jornada.
This poses a problem to me. However, as I've gone without a PDA for most
of the years of my life, a little more waiting is not likely to kill me,
in part reflected by the low priority I have given availability. I also
reason if I am to get one, it had better be good. I decide to take the
risk, as the rest of the evidence favors the iPAQ. Note the Cassiopeia
E-115 has not been a contestant. My priorities are not set up for multimedia
stuff, and there are things lacking in it for me, but not critical things.
Perhaps this would be my final fall back position.
3. Value of each factor (iPAQ is in red).
Note the low value for "Product Availability"
4. The priority sensitivity of the decision to
here to view larger version
What about the ROI? From TESS I can do a trick and examine the value returned
per dollar spent. More than that, I can include all costs (my "TCO"),
and compare the ROI's. In Figure 5, the total expected cost is measured
for each PDA in my shortlist on the y-axis. Interestingly, the Jornada
has the best ROI, and would "win" in this scenario. However, the chart
also tells me from the table at the bottom of the figure that a cost reduction
of $44 - some 8% - would give me the same ROI for the iPAQ. Unfortunately,
my sway with Priceline.com - where I eventually bought the product and
accessories for $499.95 - was inadequate on this basis since the price
would have had to be down to $470. However, as the price was only $29.95
more, this was not that enormous to cause me to rethink my decision, so
I kept to my guns about the iPAQ.
5. Value equivalency of the iPAQ solution
to the best ROI PDA (Jornada).
shaded area at the top of the bars indicates the cost reduction
to make it the same in ROI as the Jornada.
From a business perspective, the use of Knowledge Based Selection highlights
the fact that technology and human decision procedures should be integrated
as a single system in the process of complex technology or other selections.
The technology should be able to identify issues, create better fits for
vendors and clients, and speed decisions. It should not take businesses
eighteen months to reach good decisions in technology selections, particularly
as information becomes quickly outdated.
is the mother of invention. I created the PDA KB for purely selfish reasons
- I wanted a PDA after seeing how useful it was for my wife (my "competition")
to have one (she is a senior manager in a leading Telecommunications company
which gave it to her as a personal organizer). The trouble was that I
knew nothing about them, and shopping around the web merely overloaded
me with data. The net time in collecting data and building the KB was
about three days, and the best fit became immediately apparent, though
it upped the ticket I was willing to pay. The rest, as indicated above,
a general level, the process I invoked from literally zero knowledge to
a decision went through a number of steps - most of which can be accomplished
in WebTESS, though more sophisticated analysis and tradeoffs were done
with TESS (my advantage!). However, I could remove unacceptable alternatives
rapidly, and drive to a decision, aware of the risks and costs. Figure
6 illustrates the steps in the decision process.
6. Steps in the PDA selection process.
use can be made of expert knowledge. For example, in the WebTESS PDA KB,
under the criteria describing warranty, details of the warranty contract
can be found in WebTESS PDA ratings (check it out under the criterion
"Other Product and Manufacturer Considerations"). These small details
can make significant differences to the expectations of users. The length
of warranties, and what is included in them, as well as how service is
delivered, are other vital pieces of information - particularly for the
traveling businessman - that can make decisions less rocky than otherwise.
Or at least the risks have been weighed, and you have no one to blame
the direct verbal statement of the warranty and associated conditions
can be customized in terms of the value proposition to each stakeholder
(in other words, a number must somehow be associated with each warranty
option for calculation purposes). Many systems miss the point and assume
'one size fits all' which can be a mistake, potentially leading to poor
matches. There are practical considerations of course, since one cannot
do this for large numbers of criteria, and a judgment call is required
of where to draw the line between time, complexity and accuracy. Having
an effective evaluation tool, and a process with an experienced analyst
capable of vetting data and value provides better protection against methods
that can miss the mark.
PDA KB can be made more complex, of course, particularly if it is intended
for corporate wide selection. Wireless support, availability of applications,
and communication compatibility may play a major part, particularly in
data exchange issues. For example, I discovered Windows NT 4.0 does not
support USB and infra-red synchronization, which meant having to pay for
an additional serial link cable. As knowledge builds, this kind of information
can be easily captured in a knowledge base for future users. Hopefully
they will then avoid the pitfall, though it would not have changed my
decision - simply let me understand the additional cost involved in supporting
both USB and serial synchronization, or maybe identify the need to change
my operating system. As a final note, I got my iPAQ at the limit of my
patience. The risks of waiting paid off, and I am happy to report I am
pleased with my purchase. I found some opinions from Epinions.com as not
particularly valid - such as a gentleman who described the painful experience
he had when the pen would jam frequently in the slot in the casing: after
a week of use, I have had no such problem, except if I were silly enough
to put it in the wrong way! I admit I was near the end of my patience
in waiting - some two months - and almost was at the point of canceling
and going for the Jornada. However, I admit to not having checked the
availability of the Jornada lately
a note, so far, I'm happy. My iPAQ and I are doing well, particularly
as my wife is envious when she looks at her Palm Vx next to mine.