How Some ERP Vendors Demonstrated - Warts And All
The subject of this case study is the synopsis of a crucial step within every software selection process - finalist vendors' scripted scenarios software demonstrations; this particular series of events took place in February/March 2001. The importance of this milestone in any software selection undertaking has been widely publicized (for more information, see An Overview of the Knowledge Based Selection Process). Demonstrations can and should be grueling for both vendors and users, as they are the only way to discern how a software application behaves under real world expectations before any firm commitment and point of no return happens.
The customer in case was a predominantly engineer-to-order (ETO) manufacturing division of a large global corporation that, inter alia, operates within high-tech and aerospace industries; it needed to replace an over a decade-old, highly customized and disparate, islands of information, systems. The company had made considerable strides towards a Lean Manufacturing environment by acting on a number of initiatives, the upgrade and maintenance of the IT infrastructure being one. A lack of functionality and integration in these scattered operational systems were recognized obstacles to the company effectively transforming its business process to enable strategic initiatives such as B2B e-Business. To that end, it embarked on the selection of an ERP/business applications package that would satisfy its needs and return maximum business value. TEC was retained as a neutral process facilitator with an intimate knowledge of the market/vendors and no vested interest (e.g., future system integrating opportunity) in the outcome of the selection.
The finalist vendors were determined based on their responses to the request for proposal (RFP) document sent to eleven prospective candidates that TEC aligned with the customer's business requirements. The responses provided by each vendor and validated by TEC were used to assess the vendors' Product Functionality, Technology and Market Standing (Corporate Viability, Corporate Strategy, Corporate Service & Support, and Client Site Reference) criteria.
Figure 1 shows the initial standings of the four best responses and indicates a virtual tie for second place. Since the scores were so close, TEC recommended that all four vendors - Oracle, SAP, J.D. Edwards and IFS - be included in the scripted scenarios demonstration phase. Although we were aware that four detailed demonstrations might 'numb' the selection team, we also felt strongly that eliminating one vendor would only have done a disservice to both the client and the vendor.
Figure 1. Vendor Results From The RFP Responses
About this Note:
This is a two part note with Part 1 covering the Scripted Scenario Demonstrations. Part 2 discusses the specific results of the demonstrations with recommendations for users facing a similar selection process.
The Scripted Scenarios Process
The Scripted Scenarios process step was an opportunity to validate the Product Functionality portion of the RFP and the overall proof of concept through real world imitated business activities that need to be supported by the software. TEC coached the client in developing the scripted scenarios, which were a realization of business processes, either current, anticipated or improved, resulting from the Required Capabilities Assessment phase.
The completed scenarios were explained to the finalist vendors' teams during discovery sessions. The vendors were then given an equal period of time in which to prepare for the scenario demonstrations. TEC developed and provided score sheets to all selection team members in attendance for use in rating each vendor's performance. The scenario demonstrations spanned three days with the agreed average scores tabulated daily and entered into the customer-specific decision support model. ERGO 2001, TEC's decision support tool, was used to record and analyze the results of the scenarios (For more information on ERGO 2001 see ERGO 2001 IT Evaluation Tool)
The vendors presented the business scenarios on their live products using mock-up data - tailored to the way the organization does business as defined in the scenarios. The scenarios allowed the organization to see how the live product operated in its specific environment, according to the business processes outlined by the team. In addition, the selection team gained an understanding of the extent to which the vendor was willing to adjust the software to accommodate the customer's needs.
The candidate vendors and packages in order of demonstrations' sequence included the following:
||Oracle Applications 11i|
||MySAP.com R/3 v4.6|
||IFS Applications 2000|
The sequence of vendors' demonstrations was decided by a lottery. While the final vendor is always in a somewhat advantageous position, the team made sure that it did not have the chance to be in possession of the scripted scenario document any longer than its predecessors. In other words, the documents were released to all vendors exactly four weeks before the date of their presentations.
Although all were large, well-known ERP software providers that seemed to offer the type of capabilities our client required, we were determined to delve well beneath the marketing brochures to perform a thorough, statistically valid comparative evaluation of the options. TEC ensured that every vendor was focused on addressing the customer's actual needs, rather than to market its generic strengths. We also made every effort to prevent any vendor from obfuscating weaknesses and to allow that only questions to the point were asked and answered (or marked for a follow-up session).
Determining the Outcome
The main pitfalls during software demonstration are generally due to a lack of sufficient preparation by the vendor exemplified in a need for more time than allotted, an inability to strictly follow the requested bullet points of the scripted scenario document, and/or too much "tell" vs. "show" approach (for more information, see Demonstration Post-Mortem: Why Vendors Lose Deals).
Lack of preparation is utterly obvious to clients, who downgrade script performance execution scores for the vendors accordingly. Few things frustrate clients more than listening to presales consultants babble on and on about a 'killer' functionality without ever showing how it works. This is particularly true when the functionality was not even requested in the scripts. As for the case in point, the overall level of preparation by the four vendors during the demonstrations was satisfactory, what we expected given each vendor had four weeks to prepare. Part 2 summarizes the demonstration preparation and performance of each vendor.
TEC facilitated a scoring wrap up session with the selection team members at the end of each demonstration day and compiled the results into the decision support model (Figure 2). Each member of the selection team was asked to score vendors' script execution performance for every individual bullet point within the scenario (denoting a basic functional requirement, e.g., 'setup quantity price breaks' or 'release engineering data to production'). The following two qualitative aspects of software demonstrations, which were evaluated only at the entire scenario level, were also evaluated:
- The Ease of Use/Navigation and
- The Process Fit.
The Ease of Use/Navigation aspect deals with intuitiveness of the demonstrated software processes as well as the cumbersomeness of screen navigation. These factors are a representation of how adaptable the software will be throughout the live operation and the associated cost of the learning curve to get all users trained and proficient.
The Process Fit aspect deals with how well the product matches with either how the customer currently runs or plans to run the organization in the future. Should the demonstrated solution not fit the client's process, a gap was identified and the question " Is it a process improvement with value or a change not in keeping with projected improvements?" would be asked. A high score in this section indicates the solution that best matches the company's operational and strategic requirements with the fewest identified gaps.
Actual Results by Vendor
IFS had the highest overall score as well as the highest Ease of Use and Process Fit score. IFS holds a slight lead over SAP overall, whereas the other vendors had a gap greater than 5% behind SAP. IFS held a lead by greater than 3% in both Ease of Use and Process Fit. This is consistent with the project team's consensus that IFS' applications are flexible and intuitive. However, IFS also impressed the selection team with the functionality footprint.
While not necessarily the most robust functionality on paper, IFS Applications suite features quite uniformly solid functionality across the board, without serious ups and downs per modules. IFS team's confidence without arrogance and/or glitzy marketing collaterals as well as astute preparation and following the scripts to the letter, with only two all-rounder pre-sales consultants covering the entire range of product modules, left no one indifferent.
While being the last to perform the demo likely bears the unfair advantage ("halo effect") on one hand, on the other hand, IFS had to overcome the aura of an unknown vendor and, subsequently, the client's skepticism. In addition to ease of use (a user has only to learn five different screen forms, e.g., query, graph, etc. that are pervasive throughout the entire suite) and agility, the IFS' product features very deep, vertically focused functionality that is not overwhelmingly complex. IFS Applications encompasses Front Office, finite scheduling, E-Commerce, product data management (PDM), and plant maintenance management functionality, in addition to conventional ERP modules. Therefore, IFS is very competitive in mid-market manufacturing industries that require strong engineering, asset management and complex project manufacturing functionality.
The client generally felt that both IFS and JD Edwards were flexible and easy to use and navigate. Besides the astute approach to conducting the demo, J.D. Edwards impressed with the product's agility, which is enabled in part by "category codes" - the soft coded database table fields that accommodate many of possible future changes (i.e., mergers & acquisitions or divestitures) within the customer's organization. Also impressive is OneWorld Xe release's wide interoperability options set that includes application access via COM, CORBA, and Java standards. J.D. Edwards' screens mimic both the windows and Internet Explorer (Java or HTML) metaphors.
On a downside, J.D. Edwards' functionality, while solid overall, was somewhat inferior in some important areas (e.g., product engineering/definition and sales order processing), whereas the product's native functionality was superior in areas that the customer did not consider as very important (e.g., warehousing & distribution, advanced planning & scheduling). Also, some functional bullet points were demonstrated through a number of 3rd-party products, which may have deterred some team members despite exceptional functionality and the company's support assurances through premier partnerships with these niche providers like Siebel and Microstrategy.
The client was impressed with the breadth and depth of SAP, which is still the most robust product in the market, and with the demo team's professional approach and preparation, but was dismayed by the cumbersome screen navigation and perceived steep learning curve. Some team members were overwhelmed with demonstrated functionality and commented that it would likely not be needed in their day-to-day operations ("Too much of a good thing").
The team expressed some high-level concerns with regards to R/3 intricacy and the need for a sophisticated end-user, which was also bolstered by a large number of SAP's presenters, each focusing on a relatively narrow area. The general feeling was that R/3 is a more rigid system than other products, which would demand a radical change of the client's current processes and people's mindset.
Further, while the answer to the client's question whether something could be done in a peculiar way in the shape of a condescending counter-question "Why would you want to do that this way?" may be valid, the client always prefers the answer "Sure, it can be done, but why would you want to do that this way, btw?"
Also, while SAP has done a notable job in improving its user interface looks with the release of mySAP.com, its hundreds of standard out-of-the-box reports, however to the point, often leave users unimpressed by their drab look, which has not improved much from older R/3 versions, and whose modification would require significant time and human resources.
The audience felt that Oracle could have scored much higher, but the demonstration was a cry far from flowing smoothly, which may have impacted client scores. It was interesting, though, to notice that Oracle took the 2nd place regarding the Process Fit criterion, which may indicate a strong horizontal functionality that can often accommodate most of common business processes, with only some tweaking through alerts and workflow.
Oracle seems to have rounded up a formidable portfolio for almost all aspects of business, which should give even SAP a pause. While in general, functionality in financials and manufacturing is very strong, quality management, flow manufacturing and service modules were exceptionally advanced. Further, the tight integration with the native Oracle database where all configuration data, user fields' modifications, etc. are stored should facilitate future product upgrades. Also impressive is search capability throughout the entire suite and the availability of native business intelligence and data warehousing tools.
Oracle, like SAP, however, was perceived as being rigid applications with myriad modules' interdependencies, possibly requiring more implementation and product configuration time. There is also the proverbial confinement to Oracle database. Nevertheless, with its "one-stop-shop" offering and a diligent preparation for a demo, Oracle should be a formidable competitor feared by everybody, based on what its 11i product is seemingly able to support. Oracle should also attempt to be more accommodating towards potential customers in terms of providing them with 3rd-party and/or legacy applications integration and with certain levels of customization.
Scenario Scoring Results
Figure 2. Scripted Scenario Scoring Results
Although IFS took the overall lead, it did not win in every category within the Business Scenarios section of the decision model. The Advantages and Disadvantages of each vendor for each major category within the Business Scenarios is the topic of a separate TEC's case study note (see Selecting An ERP System: A "Facts And Figures" Case Study).
For a discussion of the results and user recommendations, see Part 2 of this note.
The graphs shown have been generated using the TEC's patented decision analysis software ERGO (formerlyTESS), which uses the Multi-Attribute Utility theory (MAU), Analytic Hierarchy Process (AHP) and TEC's patented decision science to compare vendors and products relative to one another in a statistically valid model.