Report 2010-102 Summary - February 2011

Administrative Office of the Courts: The Statewide Case Management Project Faces Significant Challenges Due to Poor Project Management

HIGHLIGHTS

Our review of the Administrative Office of the Courts' (AOC) oversight of the development of the statewide case management project revealed that the AOC:

RESULTS IN BRIEF

Proposition 220, approved in 1998 by California voters, began the process of unifying California's superior and municipal courts. The Trial Court Funding Act of 1997 established a funding scheme where these courts receive state, rather than local, funding. With administrative functions provided by the Administrative Office of the Courts (AOC), these superior courts receive funding through allocations from the Judicial Council of California (Judicial Council). As part of an effort to address technology problems facing the many case management systems used by the superior courts, the AOC, at the direction of the Judicial Council in 2003, continued the development of a single court case management system, referred to as the statewide case management project. Two interim systems—the criminal and traffic system (referred to in this report as the criminal system) and the civil system—are presently in use at seven superior courts. Currently, the AOC is responsible for managing the development of the most recent version of the statewide case management project—the California Court Case Management System (CCMS)—which covers all court case types. The AOC asserts that once this system is deployed statewide, CCMS will improve the access, quality, and timeliness of justice; promote public safety; and enable court accountability. CCMS is also designed to include statewide reporting; court interpreter and court reporter scheduling; and the capacity to interact electronically with other justice partner systems, such as those of local sheriffs and district attorneys. Further, the AOC stated that the system will replace a myriad of disparate commercial and custom-built case management systems that the 58 superior courts currently use. The AOC's records show that as of fiscal year 2015-16—the year in which the AOC estimates that CCMS will be deployed statewide—the full cost of the project is likely to reach nearly $1.9 billion. However, this amount does not include costs that superior courts will incur to implement CCMS. By June 2010 the AOC and several superior courts had spent $407 million on the project.

The work undertaken by the AOC on the statewide case management project has lacked sufficient planning and analysis. In implementing the project, the AOC, acting at the policy direction of the Judicial Council, should have more fully defined how it would implement the objectives and scope of the project before it began the development of the interim systems. In addition, the AOC should have analyzed whether the project would be a cost-beneficial solution to the superior courts' technology needs. The AOC had a consultant prepare a business case in December 2007 (2007 consultant study), four years after the project's inception. However, the AOC had already made a significant commitment to the statewide case management project as it had spent a total of $217 million as of June 2007, developed two interim systems, and deployed or was deploying these systems at seven superior courts. Despite significant investment in the interim systems, by the time the 2007 consultant study was completed the AOC had decided to develop CCMS, which would use functionality from the interim systems and include all case types. The AOC maintains that it commissioned the study to quantify the benefits that would be realized from CCMS. However, it appears that rather than critically analyzing the propriety of the statewide case management project, the AOC commissioned the 2007 consultant study to justify its previous actions and decisions.

Furthermore, at key points during planning and development—specifically, the decisions to develop and deploy the interim systems and then later downsize implementation of the civil system and eventually to discontinue deployment of both the criminal and civil systems in favor of a comprehensive system—it is unclear on what information the AOC made these critical decisions. The AOC asserted that it developed the project by using an iterative approach that focused on building CCMS by developing two smaller systems. Although this is the AOC's explanation of the events that took place, it was unable to provide contemporaneous analysis or documentation supporting key decisions on the project's scope and direction. We expected analysis and documentation to demonstrate the reasons for the dramatic change in the AOC's approach to developing the statewide case management project, especially given that AOC records show that the total costs invested to develop and deploy the criminal and civil systems were approximately $109 million and that the added cost to develop CCMS amounts to $199 million.

Additionally, the AOC did not structure its contract with Deloitte Consulting LLP (development vendor), the firm that has assisted in developing CCMS, to ensure that the AOC could adequately control the total cost and size of the contract. Over the course of seven years, the AOC entered into 102 amendments to develop, deploy, and support the civil system; to deploy and support the criminal system; and to develop CCMS. As a result, the cost of the contract has increased significantly—growing from $33 million to $310 million—and the AOC has become increasingly dependent on the development vendor's knowledge and expertise. Further, the AOC did not ensure that it could benefit from the warranty for the civil system because no superior court had begun to use the civil system in a live operational environment before the warranty expired. The AOC is trying to avoid similar problems with the warranty for CCMS by working to ensure that the warranty will be in effect only after CCMS has met all acceptance criteria and at least one superior court has the system deployed in a live operational environment.

In addition to planning inadequately for the statewide case management project, the AOC has consistently failed to develop accurate cost estimates. Projected in 2004, the AOC's earliest available cost estimate for the system was $260 million, an amount that grew substantially to $1.9 billion based on the AOC's January 2010 estimate. Over the same period, complete deployment to the superior courts has been postponed by seven years, from fiscal year 2008-09 to fiscal year 2015-16. However, the $1.9 billion estimate fails to include costs that the superior courts have already incurred to implement the interim versions—which they reported to us as costing nearly $44 million—as well as the unknown but likely significant costs that superior courts will incur to implement CCMS. The latest estimate also does not reflect the nature of the costs that state and local government justice partners will incur to integrate their systems with CCMS.

Although the AOC has fulfilled its reporting requirements to the Legislature, it did not provide to the Legislature additional beneficial information about the projected increases in total project costs. Specifically, the four annual reports that the AOC submitted to the Legislature between 2005 and 2009 did not include comprehensive cost estimates for the project, and the 2010 report did not present the costs in an aggregate manner. As a result, these annual reports did not inform decision makers about the true cost of the statewide case management project. When asked by the Legislature in August 2010 what the true cost of the project will be upon its completion, AOC officials cited a figure of $1.3 billion, which excludes both the $557 million that has been spent or will be spent for the criminal system and the support costs for the civil system and CCMS until CCMS is fully deployed.

Moreover, the statewide case management project is at risk of not being able to obtain the funding needed for statewide deployment. The AOC believes the core portion of CCMS development will finish in April 2011, and it estimates that it will need funding of roughly $1 billion to deploy the system for use at the 400 court facilities located statewide. However, because future funding for this project is uncertain, it is unclear whether the AOC will be able to obtain the $1 billion deployment cost or the additional $391 million needed to support CCMS through fiscal year 2015-16 when the AOC has estimated that the CCMS will be fully deployed. The AOC is attempting to develop alternative plans to minimize project costs and to deploy CCMS based on the level of funding that may be available, but the AOC believes that without full deployment to all 58 superior courts, the value of CCMS to the judicial branch may diminish.

Further, although the Judicial Council has the authority to compel the superior courts to implement CCMS, the successful implementation of the system will require the AOC to foster support from the superior courts more effectively. We conducted a survey of the seven superior courts currently using an interim version of CCMS in part to determine their satisfaction with the interim systems.1 Two superior courts that implemented an interim system reported experiencing challenges and difficulties in doing so and indicated that as a result, they are reluctant to deploy CCMS. The largest of the superior courts, the Superior Court of Los Angeles County (Los Angeles), which implemented the civil system in only four courtrooms at one location, believes that the AOC's plan for CCMS has been overly ambitious. Further, Los Angeles stated that due to the lack of a mature underlying product, a program management strategy, a solid business case, and a resource model to ensure its achievement, the statewide case management project is extremely risky. In fact, both Los Angeles and the Superior Court of Sacramento County asserted that they will not adopt CCMS unless their concerns about the system are resolved. Without the willing participation of these larger superior courts, particularly Los Angeles, the AOC may encounter significant challenges in achieving its goals, which mostly depend on the successful statewide implementation of CCMS. Of the seven superior courts that deployed an interim system, the four larger courts reported encountering difficulties during and after implementation, including the need to hire additional staff, system performance issues, and problems with the process for fixing defects. Conversely, the three smaller superior courts, although also encountering challenges during and after implementation, reported generally positive perspectives about the interim systems.

Interestingly, in response to our survey of the 51 superior courts that do not use an interim system, 18 superior courts said that their existing case management systems are currently meeting all of their needs. In replying to another question, 32 of the 51 superior courts reported that their existing systems will serve them for the foreseeable future. Of particular concern is that just 12 of these 51 superior courts that do not use an interim system submitted responses that were generally positive about CCMS or that did not discuss potential challenges associated with CCMS deployment. Many of the remaining 39 superior courts expressed uncertainty about the statewide case management project. For instance, the Superior Court of Kern County (Kern) reported that it perceives no benefit to the AOC's plan to replace Kern's current systems with CCMS and that it would refuse implementation as currently proposed.

We recognize that the implementation of any new system, especially one as large and complex as CCMS, would likely have a significant impact on any organization and would not occur without some challenges; however, it is critically important for the AOC to resolve the concerns and negative perceptions held by several superior courts. The AOC could aid such an effort by using the results from a survey of the superior courts conducted by a consultant with which the AOC has recently contracted.

Although the AOC contracted with Sjoberg Evashenk Consulting, Inc. (consulting firm) to provide independent oversight of the statewide case management project, the contract does not require that these services be performed consistent with industry standards for a project of this size and scope. Under best practices for system development and to help ensure project success—particularly with large, complex, and costly projects such as CCMS—entities normally contract with consultants to provide two types of independent oversight: independent verification and validation (IV&V) to ensure that software meets requirements and user needs as well as independent project oversight (IPO) to ensure that effective project management practices are in place and in use. The level of rigor of independent oversight should be commensurate with the size, scope, and complexity of the project. Although the Judicial Council directed the AOC in 2003 to continue its efforts in developing the statewide case management project, the AOC did not contract for IV&V services until April 2004 or for IPO services until July 2007. Even when it did contract for IV&V services, the specific level of oversight was limited in scope and duration. Further, as of July 2007 onward, the IV&V work that the consultant was to perform applied only to the development of the civil system and CCMS, and it did not cover the period of actual system deployment. Moreover, many of the services that it contracted for as of July 2007 were not required by the contract to be practiced in a manner that is consistent with industry standards adopted by the Institute of Electrical and Electronics Engineers, Inc.2 and best practices for a project of this size and complexity.

Although the AOC asserted that it had an approach that provided oversight sufficient for a project of this size and scope, it did not document its oversight plan and it could not demonstrate that some practices were performed. Additionally, the AOC relied on its staff—who cannot be considered independent in such a role—to provide portions of the IV&V and IPO oversight, and the AOC has experienced an unexpected 10-month project delay due to quality issues detected during testing. Even with the level of oversight that it was engaged to perform, the consulting firm providing IV&V and IPO services raised significant concerns that the AOC addressed inadequately. In fact, Catalysis Group, our information technology (IT) expert, believes that CCMS may be at substantial risk of future quality problems as a result of the AOC's failure to address certain of the consulting firm's concerns and the quality issues experienced on the project to date. In light of these issues, we believe that before proceeding with its plan to deploy CCMS at three superior courts that will be early adopters of the system (early-adopter courts), the AOC would derive value from conducting an independent review to determine the extent of any quality issues and problems. We do not expect that conducting this review would require a halt in the project; rather, we expect that this review could be performed in concert with the remaining development and testing effort without significant disruption to the statewide case management project.

RECOMMENDATIONS

To understand whether CCMS is a cost-beneficial solution to the superior courts' case management needs, the AOC should ensure that it conducts a thorough analysis of the costs and benefits of CCMS. The AOC should carefully evaluate the results of this analysis and present a recommendation to the Judicial Council regarding the course of action that it believes should be taken with CCMS.

To ensure its management of the statewide case management project is transparent, the AOC should make sure that all key decisions for future activities on CCMS are documented and retained.

The AOC should consider restructuring its current contract to ensure the warranty for CCMS is adequate and covers a time period necessary to ensure that deployment of CCMS has occurred at the three early-adopter courts and they are able to operate the system in a live operational environment.

Regardless of whether it is expressly required by statute, the AOC should report to the Legislature and others the true cost of the statewide case management project, including the costs for the interim systems and CCMS. Also, the AOC should require superior courts to track their past and future costs related to the statewide case management project and then include these in the total cost. Further, the AOC should be clear in disclosing what kind of costs other entities, such as justice partners, will incur that are not included. Finally, the AOC should update its cost estimate for CCMS on a regular basis, as well as when significant assumptions change.

To address the funding uncertainty facing CCMS, the AOC should work with the Judicial Council, Legislature, and governor to develop an overall strategy for CCMS that is realistic, given the current fiscal crisis facing the State.

Additionally, the AOC should use the results from its consultant's survey of the superior courts to identify and better understand the courts' concerns regarding CCMS and the status of their existing case management systems. If the survey results indicate that superior courts have significant concerns about CCMS or that they believe their case management systems will serve them for the foreseeable future, the AOC should take steps to address these concerns and perceptions. Moreover, the AOC should continue to work with the superior courts that have deployed an interim system to ensure that the AOC is promptly and appropriately addressing the courts' concerns with the systems. Although the Judicial Council has the authority to require that the superior courts implement CCMS, it is still critically important to ensure that the AOC addresses the courts' concerns as implementation moves forward.

To make certain that no significant quality issues or problems exist within CCMS, the AOC should retain an independent consultant to review the system before deploying it to the three early-adopter courts. If any quality issues and problems identified by this review can be adequately addressed, and system development can be completed without significant investment beyond the funds currently committed to develop the system, the AOC should deploy it at the early-adopter courts during the vendor's warranty period.

Going forward, we made recommendations to the AOC on how to improve its process for managing future IT projects, including that it complete a thorough analysis of cost and benefits before investing significant resources into development, document and retain all key decisions, ensure that cost estimates are accurate and include all relevant costs, have a long-term funding strategy in place, take steps to foster support from the superior courts and, if applicable, —depending on the project's size and complexity—obtain independent oversight throughout projects as well as appropriately address concerns raised.

AGENCY COMMENTS

Although the Judicial Council, AOC generally agreed with our conclusions and recommendations, it disagreed with some of our conclusions relating to its cost and contract management for the statewide case management project. Additionally, the Judicial Council, AOC disagreed with some of our conclusions and recommendations regarding the independent oversight services on CCMS and whether additional independent oversight should be performed prior to deployment.

1. The superior courts' views cited in this report are from responses to two surveys of court executive officers. The survey questions we asked appear in appendices B and C.

2. The Institute of Electrical and Electronics Engineers, Inc. is a leading developer of international standards that support many projects and services, including IT.