The Challenges of a Business Intelligence Implementation: A Case Study

Company Background

More than 70,000 students enroll each year at the University of Illinois, which offers more than 150 fields of study in 30 colleges, free-standing schools, and institutes across 3 campuses: Chicago, Springfield, and Urbana-Champaign (US). The university, one of the original land-grant colleges, opened its doors in 1867, and since then has awarded more than 500,000 degrees.

The Urbana-Champaign campus houses the largest public engineering library and third-largest academic library in the US, after Harvard and Yale. The university is recognized as a world-class center for medical, computing, engineering, and agricultural research. Faculty and staff, for example, built the first computer owned entirely by a university; developed a computer-based learning system; and created Mosaic, the first popular computer browser. Its Chicago-based Medical Center is well-known for organ transplantation and research into diabetes. Nearly twenty faculty and alumni have received Nobel Prizes, and sixteen have won Pulitzer Prizes. Alumni have also created leading companies such as Oracle Corp., Netscape Communications, and Siebel Systems. Its annual operating budget is more than $3.6 billion (USD), and sponsored research exceeds $600 million (USD).

Business Problem

As an extension of a university-wide initiative in 2001 to replace the university's course systems, the question became how to access data and report on it using the new enterprise resource planning (ERP) system. The university planned to implement the ERP system in phases, enabling it to develop a system incrementally, and to design in stages the output required to meet users' needs. Based on the depth and breadth of the university's core systems, the development of a centralized system incrementally allowed each department's and business unit's needs to be assessed and met.

The issue associated with the new implementation involved whether to use the ERP system for report generation, or to leverage the features and functionality of a system tailor-made to provide these features. With the help of Linda Bair, executive director of Illinois University, a decision support (DS) group was created. Bair had done the required research to determine which type of software would meet the needs of the university's reporting and analysis requirements. The decision to create a data warehousing architecture with a business intelligence (BI) layer on top seemed to best meet the organization's business requirements. The DS group's responsibility was to develop a data warehousing and reporting environment, which would provide users access to the information they needed for reporting and analysis.

The data warehousing group was directly aligned with the development of the ERP system. As each module was built, the DS group identified the reporting needs of the associated departments. The DS group developed the data warehousing, reporting, and BI structure as an application layer on top of the ERP system to identify the required data needed from the system. By identifying the reporting and analysis needs in parallel with the ERP system, the DS group was able to identify the requirements that would later be transferred to help in vendor selection. Although the data warehouse was being built simultaneously, application layer development required the tools of a third party vendor. With this structure in place, the choice of BI vendor became key.

Business Solution

To identify the appropriate BI tool, the University of Illinois took a user-centered approach. Focus groups were established to interview over 200 users from various user communities to identify requirements. The needs assessment was two-fold. Firstly, it would identify the requirements for static reports; and secondly, it would assess the functionality for ad hoc reports that would enable users to create their own reports. One of the key needs of user-enabled report building was a user-friendly environment. Due to the new ERP implementation, user focus would be concentrated on the new enterprise-wide system, as opposed to access to data for reporting. Additionally, the DS group identified several static reports, such as standardized budgets and student lists. Based on the focus group results, a team of core users was chosen to help with the software selection process. This selection extended beyond the simple features and functionality provided by vendors. In addition to the criteria collected, the vendor's scalability, growth, costs, support, and technical expertise were taken into account.

With the results captured, the DS group shortlisted five vendors and selected for vendor presentations using a sub-set of the university's data. After the demonstrations, the university chose Business Objects, for two main reasons. The first reason was scalability regarding the ability of its servers to accommodate large amounts of data and to leverage data from all required data sources. The second reason was its ease of use from the user perspective, due to its web-enabled environment.

The DS group wanted to keep the core focus on users, to keep the project business-focused as opposed to information technology (IT)-centered. The issue surrounding this strategy was the ability of the IT staff to leverage their expertise of technology and to anticipate the requirements based on their knowledge of the ERP system without overshadowing the needs of users. Additionally, the various business units would have a feeling of ownership over the process and implemented system, which would transfer into user buy-in.

Across all three campuses, there are over 4,500 standardized report users who source reports from the data warehouse, or directly from the ERP system. Also, 2,000 of those report users are people using ad hoc reporting functionality. Additionally, over 1,000 (this is the most current number) users are classified as active users, meaning they run a query at least once a month to access data and perform analyses. Although Business Objects is the vendor software being maintained, users can use any other program to access and run queries against the data warehouse.

Challenges

Some of the challenges encountered by the University of Illinois during the Business Objects implementation included user training, the reporting focus, and the use of BI tools, as well as user buy-in. Additionally, the scalability of the server environment and the amount of data being processed created severe issues. Although each of these areas had positive outcomes, much work was required on the part of the DS group and Business Objects to overcome the challenges associated with the implementation.

The two main challenges associated with training involved timing and selling the new system. For the actual deployment, training was conducted too early. Once the users had been trained by the DS group (themselves "super-users" trained by Business Objects), the system didn't go live for another three months. This meant that users were not able to utilize their newly acquired knowledge, diminishing the effects of the training. When the new tool was in place, a follow-up was conducted by the DS group to see how the users were doing with the new system. Subsequently, a tool was put in place to identify the correlation between users trained and the associated help desk inquiries. Labs were set up to help people create the departmental reports needed. Also, a technical support center was created by the university, and was set up to answer any subsequent questions and to solve outstanding issues.

The second issue regarding user training dealt with selling the new system. Due to the implementation of the ERP system, users were more concerned with how to enter a new student into the system, and with how to use the system in general to do their jobs (as opposed to how to create reports or analyze financials). The DS group created an internal marketing campaign to target users and appeal to their needs. The focus was on key players in various departments, and on users on a practical level. This included a focus on the fact that once users knew how to operate the system, their interest would shift to the required output—namely, reports, and analysis of the required information. This issue highlighted the debate between the "build and they will come" theory, versus building a new system once needs are defined by the user community. In this case, with the high level of usage, the first approach worked, but only because of the diligence of the DS group and its commitment to creating a successful BI environment for the university.

One pitfall identified by the DS group was the use of Business Objects as a general report generation application. Business Objects' strength lies in creating and deploying ad hoc reports and trending analyses. The tool was not designed to be used as an application for general report generation. For example, the University of Illinois generates a daily report that identifies each student and all of the information available about that student. The smallest of these reports is over 10,000 pages. Consequently, the reporting tool ran for forty-five minutes before beginning to generate the actual page views. To this day, the report is still run using Business Objects; however, a PDF file is created for users to view the report. This way, users do not have to wait for the report to generate, and can view the entire report immediately after the query has run.

Aside from the high involvement of users with the selection process, the procurement policies of the State of Illinois were followed by the university. From the beginning, management understood the reasoning behind the software selection choice. The procurement policies gave management the understanding they needed to see the immediate value in the decision. Aside from sending out the RFP, the technical and political processes were adhered to, and since the initiative was directly associated with the overall ERP implementation, a general budget had already been allocated to the project. From the user side, attaining buy-in took more work. On one level, user involvement in the project was massive. Whether through focus groups, the RFP creation and demonstration process, or training initiatives, users were aware of and had a say in the overall process. User input ranged from the needs assessment, to software evaluation choice and user friendliness. On another level, interest was low. Had there been significant interest from the user community (or as an alternative, had interest been developed at the time of the student systems overhaul), interest and general buy-in might have taken place more quickly.

Another important challenge was the actual technical design of the server environment. The university's initial assessment was that one server would be able to handle the data loads and report builds of the whole system. This assumption was based on five users using the system initially, which underestimated the extent of use of the tool and access to the information. Actual usage extended to over 400 regular users. This assumption actually spiraled into a 17-month process of developing the proper server platform to support the environment. Aside from the amount of traffic, with the version 5.1.7 upgrade of the Business Objects server, the servers kept crashing, and many bugs were reported. Users couldn't access the system, and blamed the data warehouse and subsequently left the environment (in other words, they stopped using the reporting tools provided, and reverted back to the previous system. Bug issues included the appearance of repeated login screens, system lock at user login, and the creation of logs overpowering the system. In terms of service, Business Objects worked closely with the university to create a stable environment with the upgrade to version 6.5.1.

Recommendations and Lessons Learned

The challenges of creating a server environment that encourages not just use of the BI tool but that does so based on its strengths are two areas to consider when implementing a BI solution. To develop a stable environment, organizations should overestimate the use of the system. Included in this assessment is the need for organizations to allocate an additional budget percentage for the selected solution. The impact of not doing so can be seen in the university's situation of underestimating the report generation frequency and data volumes required—a common occurrence in data warehousing implementations. Also, when evaluating software, organizations should fully understand the limitations, challenges, and weaknesses of the tool. This can be highlighted by Business Objects' generation of large static reports, in which it took forty-five minutes to generate the first page. Although the tool has its strengths in ad hoc reporting and analytics, extending that to application-based generation of reports can only highlight the weaknesses associated with transferring a BI tool to a report generation application.

User training and the development of in-house expertise is another important factor. Business Objects is a good example of a vendor that transfers knowledge to its users and allows them to become experts in the field. This provides user organizations the confidence and skill set to develop and to maintain their own BI environments. Accordingly, the DS group was able to develop and deliver tailor-made training to its users with the added business knowledge that is only available within an organization's own environment. Additionally, even though a help desk environment is essential to deal with issues as they arise, issues related to training can be minimized by providing system training as close to system deployment as possible, giving users the ability to practice their new skills and take full advantage of the new environment.