Pilot Project Report -- Impact Analysis

Benefit-Cost Analysis and Impact Analysis

Metrics were defined early in the pilot project to measure some of the quantifiable effects of ODIN, such as increased productivity and increased office space available for other needs. The measurements were performed before and after implementation. In addition, users were surveyed to assess important intangible impacts, such as faster customer service and improved job satisfaction.

Project Costs

The pilot system was designed to be a simple, low cost solution. A variety of issues led to extending the project duration from 6 months to 14 months, but user involvement assured that everyone understood and accepted the reasons for the delay. Ironically, decreasing hardware costs led to substantial savings.

The core development tasks (design, programming, and unit testing) were completed very nearly within the estimates. The amount of time spent converting old files turned out to be much higher than expected, largely due to the time required to prepare a file for scanning. The total cost which can be directly attributed to ODIN is $126,221.

The part time demands on management, users, and technical support were deliberately omitted from the plan. The people contributing this effort were already working full time in related areas, and their work on ODIN fell within their existing responsibilities. Nonetheless, to facilitate future planning and get an indication of the total institutional cost of ODIN, their contribution was measured. It amounted to just over $40,000.

ODIN has been up and running for over a year. The ongoing support and administration costs have been low, averaging less than 10 hours per month.

Measurable Benefits and Impacts

Prior to ODIN, incoming documents were hole-punched and stored in folders. The folders were stored alphabetically in a file room, and they had to be pulled and refiled for each document. Time was also required to create and label each new folder. On average, 27 incoming documents could be filed per hour. With ODIN, filing efficiency has increased 53% to 42 documents per hour, saving 264 hours per year.

Once the documents are online, there are more time savings. ODIN files can be viewed as often as needed, by any number of people, without alphabetizing and refiling them after each use. Retrieval itself is also faster, although not as fast as people would like. It takes 10 seconds in ODIN to enter a name or number and bring up the list of documents in a borrower's file, and up to 10 more seconds to retrieve a specific page of a document. The electronic table of contents, however, makes finding the desired page much faster. Users report that reviewing files in ODIN takes an average of 4.7 minutes - one-third of the 14.5 minutes reported for paper files.

The actual impact on productivity is even greater than this suggests because ODIN has slashed the amount of time spent on missing files and other exceptions.

Before ODIN, it took 75 seconds to retrieve and return a paper file. Approximately 10% of the time, however, the files were not found on the first try. One-third of the missing files were found by looking nearby, which took 15 more seconds. Another third were found after checking name and status changes on the computer, a process that averaged 40 seconds. The final third required an extensive hunt, including walking around the office, looking for files on desks, talking to other people, making phone calls, sending email, putting the problem aside, coming back to it later, asking someone else to try, and doing it all over again the next day. This often took several hours spread over several days, and there were times when the file was never found. On average, each of these tough cases cost the office 2.5 hours. Putting this together, it took 3,042 seconds (approximately 51 minutes) to track down a file which was misplaced or in use. That happened 10% of the time, so the average retrieval time was 75 + (10% x 3042) = 379 seconds (more than 6 minutes).

ODIN has radically changed document retrieval. The time spent walking to the file room, pulling files, filling out "out cards", alphabetizing and refiling has been eliminated. Time is now spent starting the web browser (if it isn't already running), signing on, and entering the student number or name. This takes only 15 seconds per file, compared to 75 seconds to retrieve and return a paper file.

The biggest change is the dramatic reduction in time spent handling exceptions. Since multiple people can view a file simultaneously, it doesn't matter if a file is in use. Since files can be found by student number or social security number, name changes are not a problem. Surveys indicate that some small percentage of the documents that should be in ODIN are still not found, perhaps due to indexing errors or loss prior to scanning. Using 1% (the real number is probably much lower) and the same 51 minute penalty that applied to the paper files, then the average for all searches is 15 + (1% x 3042) = 45 seconds. This is more than 8 times faster than the 379 second average for the paper files.

One downside of imaging is that reading images is more difficult than reading paper. The limited resolution of the monitors and the need for scrolling adds 22% to the time required to read a document.

Putting all this together at the current rate of use (90 files per day), the office is saving 2,398 hours per year reviewing files online, corresponding to a 660% increase in productivity when reviewing files.

Additional savings come from freeing up space that was used for files. Office space is difficult to find on a confined campus. The going rate is approximately $16 per square foot in the University District. The pilot implementation of ODIN released 139 square feet of file space, valued at $2,224 per year. If future legal changes allow the original promissory notes to be shredded, another 121 square feet of space will be available, bringing the total annual office space benefit to $4,160.

The savings were not applied to staff reductions and they have not been realized through decreased expenditures. Instead, the offices have taken this opportunity to change the way they do business to improve customer service. The files are more accessible and they have a convenient table of contents for the first time, so they can be used in new ways. Surveys and usage logs show the average number of files reviewed each day has roughly doubled. The paper files were only referenced for 13% of the office inquiries; now ODIN is used to answer 21% of the questions. At the same time, the number of pages read per file reviewed has decreased by a factor of two, largely because the table of contents can be used to answer some questions and it makes it easier to find the correct document.

The potential benefits for other sites will vary depending on how they store and use files currently. For example, departments that spend a lot of money on microfilm and film readers may see greater savings, while others that use their files less often may save less.

Calculation of Payback Time

Ignoring discount rates and intangibles, it would take 2.7 years for the pilot site to recover the equivalent of the $126,221 direct cost of ODIN. Including all the indirect costs would extend the payback period to 3.6 years. Applying a discount rate of 5% results in a positive return on investment after 3 or 4 years of use, depending on which cost figure is used. All this comes before the intangible benefits are even considered.

Intangible Benefits and Impacts

The benefits of ODIN go beyond the measurable savings. The ability to view any document without going to the file room allows questions to be answered faster and more accurately. Surveys using a scale of 1 to 10 showed the convenience of retrieving files increased from 5.3 to 9.8, convenience of filing documents leapt from 3.0 to 9.2, customer service when the files are involved rose from 5.5 to 9.3, job satisfaction related to file retrieval increased from 5.0 to 8.4, and job satisfaction related to filing documents increased from 2.5 to 8.2 for filing.

On the downside, the readability of online images is lower. Survey responses said readability decreased from 9.6 to 7.6 on the scale of 1 to 10. When asked if they would like to return to paper files, however, the response was a unanimous "No!"

Benefits to Other Projects

In addition to the impact on the pilot site, the entire University is benefiting from the skills developed integrating web applications with mainframe and Unix systems. These new skills are being applied in several other projects that will bring even greater benefits.