Report 2010-102 All Recommendation Responses

Report 2010-102: Administrative Office of the Courts: The Statewide Case Management Project Faces Significant Challenges Due to Poor Project Management (Release Date: February 2011)

Recommendation #1 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should estimate costs at the inception of projects.

Annual Follow-Up Agency Response From October 2012

The AOC has two artifacts for estimating costs at the inception of a project, the Business Case and the Project Assessment Form, both which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard Solution Development Life Cycle (SDLC). SDLC describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category, medium, or large, are currently envisioned in the current budget climate, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The primary purpose of development of a business case, in support of Judicial Council-approved recommendations for AOC realignment, is to provide a financial justification for undertaking a project, entailing cost analyses for a recommended solution and for alternatives that were explored as well. Costs are broken out into several categories (hardware, software, etc.), as well as project vs. ongoing costs. A copy of the business case template is attached.

The purpose of the Project Assessment Form is to gather high-level information about a request to categorize the project as small, medium, or large, based on estimated level of effort, duration, and cost. Costs are broken out into several categories (hardware, software, etc.), application vs. infrastructure, and project vs. ongoing. A copy of the Project Assessment Form is included below.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #2 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should employ appropriate budget and cost management tools to allow it to appropriately budget, track, manage, and estimate costs.

Annual Follow-Up Agency Response From October 2012

The AOC has two artifacts for budgeting, tracking, managing, and estimating costs, the business case and the project assessment forms, both developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard Solution Development Life Cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC. The Business Case and Project Assessment Form templates are attached.

In addition to completing the business case artifact itself, development of which supports Judicial Council-approved recommendations for AOC realignment, the SDLC calls for revisiting the business case after project inception to facilitate cost management and tracking once an effort is underway.

Another key tool used for budgeting, tracking, managing, and estimating costs is the Information and Technology Services Office's (formerly Information Services Division) zero based budgeting process. Specifically, Information and Technology Services prepares a five-year budget for each Information and Technology Services program. Each program budget is shared with the Fiscal Services Office (formerly Finance Division) and other business partners before being used as a basis for the Information and Technology Services' annual budget allocation. The core purpose of this process is to identify key needs for each project going forward while providing sufficient funding for baseline activities. The other purpose is to provide a budget monitoring tool for project managers throughout the fiscal year. The budget projections produced through this process are also used to inform the Judicial Council, related working groups, and advisory committees. To develop, track and monitor their budgets, each program uses information provided by the Oracle Financial system, which captures all AOC allocations, encumbrances, and expenditures for Information and Technology Services programs.

Finally, the AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health. Formal criteria for the budget health indicator have been established, and are as follows: green, project is tracking to approved budget, which may include one that has been re-baselined per a formally approved change; yellow, a budgetary risk has been identified for which a mitigation strategy is in place; red, a budgetary risk has been identified for which there is not a mitigation strategy in place. An explanation of any yellow or red budget indicators must be provided in writing, and a justification for any indicators that were previously yellow or red but have been reset to green must also be noted.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #3 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should ensure that cost estimates are accurate and include all relevant costs, including costs that superior courts will incur.

Annual Follow-Up Agency Response From October 2012

The AOC has two artifacts to ensure that all relevant costs of an effort are considered, the Business Case and the Project Assessment Form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

Both the business case, development of which supports Judicial Council-approved recommendations for AOC realignment, and the project assessment form, include a comprehensive list of cost categories, including hardware, software, data center, contract services, one-time vs. ongoing costs, and others. The project assessment form includes costs likely to be incurred by entities external to the AOC, such as the superior courts, justice partners, and law enforcement agencies. For example, these may include expenses related to time required of internal staff, potential software and hardware costs, or contracting vendors for design, development or testing. Further, the SDLC calls for revisiting the business case after project inception to validate the accuracy of initial estimates. The AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health, ensuring that significant variances from initial cost estimates are identified and addressed. The SDLC also requires a post-project review, one part of which includes an analysis of the accuracy of cost estimates and the identification of any lessons learned to improve cost management on future efforts. The post-project review template is embedded below.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #4 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should disclose costs that other entities will likely incur to the extent it can reasonably do so.

Annual Follow-Up Agency Response From October 2012

The AOC has two artifacts for ensuring that all relevant costs of an effort are considered, the business case and the project assessment form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned, given the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

Both the business case, development of which supports the Judicial Council-approved recommendations for AOC realignment, and the project assessment form, include a comprehensive list of cost categories, including hardware, software, data center, and contract services, one-time vs. ongoing costs, and costs likely to be incurred by entities external to the AOC, such as the superior courts, justice partners, and law enforcement agencies. For example, this might include expenses related to time required of internal staff, potential software and hardware costs if available, and contracting vendors for design, development or testing.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #5 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should update cost estimates on a regular basis and when significant assumptions change.

Annual Follow-Up Agency Response From October 2012

The AOC has two artifacts for ensuring that cost estimates are updated when significant assumptions change, the business case and the project assessment form, both of which have been developed as part of the Enterprise Methodology and Process (EMP) program. A key component of the EMP program is development and implementation of a standard solution development life cycle (SDLC) that describes a phase-by-phase methodology for projects categorized as medium or large, including standards, processes, and artifacts. While no new projects in either category are currently envisioned in the current budget environment, upcoming applicable maintenance and operations (M&O) efforts and future projects will be required to adhere to the SDLC.

The Business Case and Project Assessment Form templates are embedded below.

The SDLC calls for revisiting the business case, development of which supports Judicial Council-approved recommendations for AOC realignment following project inception, to validate and update accuracy of initial estimates. Likewise, the AOC makes use of a project portfolio management (PPM) tool for weekly project reporting, one element of which includes budget health, ensuring that significant variances from initial cost estimates are identified and addressed. Formal criteria for the budget health indicator have been established, and are as follows: green, the project is tracking to the approved budget,; yellow, a budgetary risk has been identified for which a mitigation strategy is in place; red, a budgetary risk has been identified for which there is no mitigation strategy in place. An explanation of any yellow or red budget indicators must be provided in writing, and a justification for any indicators that were previously yellow or red but have been reset to green must also be noted.

The annual budget process also provides an opportunity to adjust budget projections to reflect changes in program strategy and assumptions. Specifically, Information and Technology Services prepares a five-year budget for each Information and Technology Services program. Each program budget is shared with the Fiscal Services Office (formerly Finance Division) and other business partners before being used as a basis for the Information and Technology Services' annual budget allocation. The core purpose of this process is to identify key needs for each project going forward while providing sufficient funding for baseline activities. The other purpose is to provide a budget monitoring tool for project managers throughout the fiscal year. The budget projections produced through this process are also used to inform the Judicial Council, related working groups and advisory committees. To develop, track and monitor their budgets, each program uses information provided by the Oracle Financial system which captures all AOC allocations, encumbrances and expenditures for Information and Technology Services programs. As the program strategy and assumptions change, Oracle is updated accordingly, to reflect the status of the project.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #6 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should disclose full and accurate cost estimates to the Judicial Council, the Legislature, and stakeholders from the beginning of projects.

Annual Follow-Up Agency Response From October 2013

In October 2012, the Chief Justice created the Judicial Branch Technology Planning Task Force which reports directly to the Judicial Council's broader Technology Committee. The committee oversees the council's policies concerning technology and is responsible in partnership with the courts for coordinating with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners, and stakeholders on technological issues relating to the branch and the courts. The committee is responsible for ensuring that council policies are complied with and that specific projects proceed on schedule and within scope and budget. The committee seeks reports and recommendations from the Administrative Director, the courts, and stakeholders on technology issues. It ensures that technology reports to the council are clear and comprehensive and provide relevant options so that the council can make effective final technology policy decisions. The committee reports on technology affecting the branch and courts at each Judicial Council meeting.

Over the past year, the committee in partnership with the trial courts has completed among other accomplishments: A survey of the trial courts on their technology needs; Creation of the Judicial Branch Technology Initiatives Working Group (JBTIWG) that has assisted the Judicial Council Technology Committee in the overall effort of defining and confirming requirements for branchwide technology initiatives; Regular ongoing meetings to review technology projects and make policy decisions; Working collaboratively with stakeholders to define technology governance and develop a long-term strategic plan for technology; and Hosting a Technology Summit in October 2012 that assembled branch stakeholders for a collaborative discussion on branch technology governance, vision, and roadmap.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2012

The Judicial Branch Technology Committee, in collaboration with working groups and advisory committees, is establishing a branch governance structure that will enable communication to the Judicial Council, the legislature and stakeholders, With the March 27, 2012 Judicial Council decision to end CCMS, the Judicial Council tasked the Technology Committee with overseeing the council's policies concerning technology. The Technology Committee is responsible, in partnership with the courts, to coordinate with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners and stakeholders on technological issues relating to the branch and the courts. The committee is responsible to ensure compliance with council policies and that specific projects proceed on schedule, and within scope and budget.

In May 2012, the committee established the Judicial Branch Technology Initiatives Working Group, consisting of Presiding Judges, Court Executive Officers, and Court Information Officers, to assist in the overall effort of defining and confirming requirements for branchwide technology initiatives to address the short-term critical technology needs for the Judicial Branch, and a long-term strategic vision for the Judicial Branch. As a result of this effort, the Technology Committee will develop a governance structure for technology programs that will provide the oversight, monitoring, transparency and accountability recommended by the BSA and further supported by the Judicial Council recommendations published August 2012. Project implementations will be subject to the approval of the Technology Committee.

In addition to the project oversight provided by the governance structure , the superior courts, the appellate courts, and the Administrative Office of the Courts are all subject to the Judicial Branch Contract Law which requires the approval from the California Technology Agency for projects of the Judicial Council or the courts with total costs estimated at more than five million dollars.

We believe the above measures will provide the necessary oversight to ensure communication of costs at the onset of projects.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #7 To: Administrative Office of the Courts

To better manage costs of future IT projects, the AOC should ensure that it has a long-term funding strategy in place before investing significant resources in a project.

Annual Follow-Up Agency Response From October 2013

In October 2012, the Chief Justice created the Judicial Branch Technology Planning Task Force which reports directly to the Judicial Council's broader Technology Committee. The committee oversees the council's policies concerning technology and is responsible in partnership with the courts for coordinating with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners, and stakeholders on technological issues relating to the branch and the courts. The committee is responsible for ensuring that council policies are complied with and that specific projects proceed on schedule and within scope and budget. The committee seeks reports and recommendations from the Administrative Director, the courts, and stakeholders on technology issues. It ensures that technology reports to the council are clear and comprehensive and provide relevant options so that the council can make effective final technology policy decisions. The committee reports on technology affecting the branch and courts at each Judicial Council meeting.

Over the past year, the committee in partnership with the trial courts has completed among other accomplishments: A survey of the trial courts on their technology needs; Creation of the Judicial Branch Technology Initiatives Working Group (JBTIWG) that has assisted the Judicial Council Technology Committee in the overall effort of defining and confirming requirements for branchwide technology initiatives; Regular ongoing meetings to review technology projects and make policy decisions; Working collaboratively with stakeholders to define technology governance and develop a long-term strategic plan for technology; and Hosting a Technology Summit in October 2012 that assembled branch stakeholders for a collaborative discussion on branch technology governance, vision, and roadmap.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2012

The Judicial Branch Technology Committee, in collaboration with working groups and advisory committees, is establishing a branch governance structure that will enable communication to the Judicial Council, the Legislature and all stakeholders, With the March 27, 2012 Judicial Council decision to end CCMS, the Judicial Council tasked the Technology Committee with overseeing the council's policies concerning technology. The Technology Committee is responsible in partnership with the courts for coordinating with the Administrative Director of the Courts and all internal committees, advisory committees, commissions, working groups, task forces, justice partners and stakeholders on technological issues relating to the branch and the courts. The committee is responsible for ensuring that council policies are complied with and that specific projects proceed on schedule and within scope and budget.

In May 2012, the committee established the Judicial Branch Technology Initiatives Working Group consisting of Presiding Judges, Court Executive Officers, and Court Information Officers, to assist in the overall effort of defining and confirming requirements for branchwide technology initiatives, to address the short-term critical technology needs for the Judicial Branch and a long-term strategic vision for the Judicial Branch. As a result of this effort, the Technology Committee will develop a governance structure for technology programs to provide the oversight, monitoring, transparency and accountability recommended by the BSA and further supported by the Judicial Council recommendations published August 2012. Project implementations will be subject to the approval of the Technology Committee.

In addition to the project oversight provided by the governance structure , the superior courts, appellate courts, and the Administrative Office of the Courts are subject to the Judicial Branch Contract Law, which requires the approval from the California Technology Agency for projects of the Judicial Council or the courts with total costs estimated at more than five million dollars.

We believe these measures will provide the necessary oversight to ensure a long term funding strategy.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #8 To: Administrative Office of the Courts

Although the Judicial Council has the legal authority to compel the courts to adopt CCMS, to better foster superior court receptiveness to deploying CCMS, the AOC should continue to work with the superior courts that have deployed the civil system to ensure it is addressing their concerns in a timely and appropriate manner.

Annual Follow-Up Agency Response From August 2014

At the Judicial Council meeting of August 22, 2014 the council approved the Court Technology Governance and Strategic Plan which has the three primary components:

1. Technology Governance and Funding Model

2. Strategic Plan for Technology

3. Tachtical Plan for Technology

Below is the item number of the agenda item and the address at which it can be located.

Item 4 3:20-4:20 p.m.

Judicial Branch Administration: Court Technology Governance and Strategic Plans (Action Required)

- Presentation

- Audiocast (41:57)

http://www.courts.ca.gov/documents/jc-20140822-item4.pdf

As previously discussed, at the time of the above being formally adopted by the council the issue would be considered fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From January 2014

At yesterdays Judicial Council meeting the council approved the new IT governance, strategy, tactical, and funding proposal. As in all things of this nature, it will go out for public comment before finalization at the June meeting of the council. They have been implemented now and this does correspond to your request to see them formalized and in place for the remaining two issues that are outstanding. The documents are now approved, in place, and formalized. Below is the council summary and the links to the documents.

Technology Governance, Strategy, and Funding: The council reviewed a report and accompanying proposal from its Technology Planning Task Force, which provides a structure, roadmap, and process for managing technology initiatives and increases the transparency and accountability of how funds are managed and allocated for technology projects in the judicial branch. The task force is comprised of judicial officers, court executive officers, court information technology officers, and other stakeholders representing 17 trial courts, the appellate courts, and the public.

The council approved the proposal in concept to be used in support of the budget change proposal process for technology initiatives. The task force will post the proposal for public comment and then re-submit it for final approval at a future council meeting.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

The Administrative Office of the Courts' links did not transmit over with their response. The links are as follows:

http://www.courts.ca.gov/documents/jc-20140123-itemI.pdf http://www.courts.ca.gov/3046.htm#acc20898.

Annual Follow-Up Agency Response From October 2013

The recommendation that the AOC continue to work with the superior courts that have deployed the civil system to ensure it is addressing their concerns in a timely and appropriate manner has been adhered to. The V3 support team continues to hold weekly meetings, attended by court project managers, technical analysts, and operational staff, where court representatives discuss operational issues and prioritize items for next software release. Following established processes, any enhancements and defects exceeding a pre-defined level of effort (LOE) are escalated for approval. In addition to the weekly group meetings, individual court meetings are held each week to provide each court the opportunity to discuss their specific issues specific. To further support the courts, metrics are used to track compliance with service-level agreements, application performance, and reliability. Over the past 12 months there have only been a few severity 1 (critical) issues recorded and only one in the last six months. The CCMS V3 application has been stable.

As a further indication of progress, the V2/V3 maintenance workstream has concluded that the V3 civil system is a significant branch asset that is working well for the courts. The workstream, established by the Technology Committee and represented by staff from superior courts in the counties of Orange, Sacramento, San Diego, Ventura, San Joaquin, and Fresno, acknowledge that V3 supports approximately 25% of all cases across the state and it is stable and the courts consider V3 a viable solution.

The current status ofV3 represents substantial progress in implementing the state auditor recommendations 3.1 b and 3 .lc. - To further demonstrate the commitment and substantial work effort of the Technology Committee and designated workstreams, we have compiled documentation which was sent to the State Auditor in February 2013.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2012

Currently the Information Technology and Information Services Office (ITSO) [formerly the Information Services Division (ISD) of the Administrative Office of the Courts (AOC) conducts weekly conference calls with five superior courts that have deployed CCMS V3. The V3 support project manager facilitates weekly meetings which are attended by court project managers, technical analysts, and operational staff. During these meetings court representatives discuss operational issues and prioritize items for the next software release. Following established processes, any enhancements and defects exceeding a pre-defined level of effort (LOE) are escalated to the governance committee for approval. In addition to the weekly group meetings, weekly meetings are held with each individual court, providing an opportunity to discuss issues specific to their court.

To further support the courts, metrics are maintained to track compliance to service level agreements, as well as application performance and reliability. Over the past 12 months there has been only one severity 1 (critical) issue recorded. The CCMS V3 application has been extremely stable.

Regarding the future, the Judicial Council Technology Committee created the Judicial Branch Technology Initiatives Working Group and developed multiple workstreams to address various technology issues facing the branch. Their goal is to make recommendations to the Judicial Council to specifically address the technology issues. One such workstream is the V/2V3 Maintenance workstream tasked with determining how the courts that are currently using V2 or V3 will be supported. The V2/V3 workstream is chaired by the Court Executive Officer for Orange County and the Court Information Technology Officer from Fresno. It is a collaborative effort. Representatives from V3 and V2 courts including Orange, Sacramento, San Diego, Ventura, San Joaquin, and Fresno, with participation by AOC Information and Technology Services staff. The courts are preparing a proposal regarding the future of the V2 and V3 case management systems. The document addresses maintenance and support, governance, funding, addition of courts, hosting, and life expectancy of the case management systems. Current Service Level Agreements (SLAs) are being met.

In consideration of current support activities and those planned for the future, we believe the concerns of the courts using the civil case management system, CCMS V3, are being addressed in a timely and appropriate manner.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #9 To: Administrative Office of the Courts

Although the Judicial Council has the legal authority to compel the courts to adopt CCMS, to better foster superior court receptiveness to deploying CCMS, the AOC should work with superior courts to address concerns about hosting data at the California Court Technology Center (Technology Center). Further, the AOC should take steps to ensure that superior courts do not lose productivity or efficiencies by hosting data at the Technology Center.

Annual Follow-Up Agency Response From August 2014

At the Judicial Council meeting of August 22, 2014 the council approved the Court Technology Governance and Strategic Plan which has the three primary components:

1. Technology Governance and Funding Model

2. Strategic Plan for Technology

3. Tachtical Plan for Technology

Below is the item number of the agenda item and the address at which it can be located.

Item 4 3:20-4:20 p.m.

Judicial Branch Administration: Court Technology Governance and Strategic Plans (Action Required)

- Presentation

- Audiocast (41:57)

http://www.courts.ca.gov/documents/jc-20140822-item4.pdf

As previously discussed, at the time of the above being formally adopted by the council the issue would be considered fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From January 2014

At yesterdays Judicial Council meeting the council approved the new IT governance, strategy, tactical, and funding proposal. As in all things of this nature, it will go out for public comment before finalization at the June meeting of the council. They have been implemented now and this does correspond to your request to see them formalized and in place for the remaining two issues that are outstanding. The documents are now approved, in place, and formalized. Below is the council summary and the links to the documents.

Technology Governance, Strategy, and Funding: The council reviewed a report and accompanying proposal from its Technology Planning Task Force, which provides a structure, roadmap, and process for managing technology initiatives and increases the transparency and accountability of how funds are managed and allocated for technology projects in the judicial branch. The task force is comprised of judicial officers, court executive officers, court information technology officers, and other stakeholders representing 17 trial courts, the appellate courts, and the public.

The council approved the proposal in concept to be used in support of the budget change proposal process for technology initiatives. The task force will post the proposal for public comment and then re-submit it for final approval at a future council meeting.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

The Administrative Office of the Courts' links did not transmit over with their response. The links are as follows:

http://www.courts.ca.gov/documents/jc-20140123-itemI.pdf http://www.courts.ca.gov/3046.htm#acc20898.

Annual Follow-Up Agency Response From October 2013

The recommendation that the AOC continue to work with the superior courts that have deployed the civil system to ensure it is addressing their concerns in a timely and appropriate manner has been adhered to. The V3 support team continues to hold weekly meetings, attended by court project managers, technical analysts, and operational staff, where court representatives discuss operational issues and prioritize items for next software release. Following established processes, any enhancements and defects exceeding a pre-defined level of effort (LOE) are escalated for approval. In addition to the weekly group meetings, individual court meetings are held each week to provide each court the opportunity to discuss their specific issues specific. To further support the courts, metrics are used to track compliance with service-level agreements, application performance, and reliability. Over the past 12 months there have only been a few severity 1 (critical) issues recorded and only one in the last six months. The CCMS V3 application has been stable.

As a further indication of progress, the V2/V3 maintenance workstream has concluded that the V3 civil system is a significant branch asset that is working well for the courts. The workstream, established by the Technology Committee and represented by staff from superior courts in the counties of Orange, Sacramento, San Diego, Ventura, San Joaquin, and Fresno, acknowledge that V3 supports approximately 25% of all cases across the state and it is stable and the courts consider V3 a viable solution.

The current status ofV3 represents substantial progress in implementing the state auditor recommendations 3.1 b and 3 .lc. - To further demonstrate the commitment and substantial work effort of the Technology Committee and designated workstreams, we have compiled documentation which was sent to the State Auditor in February 2013.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2012

The CCTC provides consistent, cost-effective, and secure hosting services, including ongoing maintenance and operational support, data network management, desktop computing and local server support, tape back-up and recovery, help desk services, email services, a dedicated service delivery manager, and a disaster recovery program. In the event of a significant interruption of court services, the Disaster Recovery program ensures that infrastructure and network services, and trial and appellate court applications hosted in the CCTC can be safely and securely backed-up, redirected, and restored.

CCTC services allow the courts to rely on the skills and expertise of their maintenance and support teams to configure and install software and hardware upgrades and address other minor and critical issues. Courts also benefit from enterprise hardware and software license agreements, as well as bulk volume discounts in purchasing. In addition, centralized changes (e.g. hardware and software patches) are much easier and more efficient to install, while the central help desk support provides the courts a single point of contact.

To address the needs of the courts, each program has a delivery manager who works directly with the courts to address day-to-day issues and concerns, as well as extended challenges. Weekly, the delivery manager meets with CCTC staff on behalf of the courts to address any service issues.

Regarding the future, the Judicial Council Technology Committee created the Judicial Branch Technology Initiatives Working Group and developed multiple workstreams to address various technology issues facing the branch. Their goal is to make recommendations to the Judicial Council to specifically address the technology issues. One such workstream is the V/2V3 Maintenance workstream tasked with determining how the courts currently using V2 or V3 will be supported going forward. Another workstream is the Technology Roadmap, tasked with defining the technology vision for the branch. Both workstreams are collaborative efforts led by trial court executives and sponsored by judicial officers. The issue of hosting will be addressed by both workstreams as they address the question of local versus central hosting. At the conclusion of the project, recommendations will be provided to the Judicial Council.

In consideration of current support activities and those planned for the future, we believe the concerns of the courts hosted at the California Courts Technology Center are being addressed to guard against potential loss of productivity and or efficiencies. System reliability and availability at the CCTC has been stable and help desk tickets are being handled expeditiously. Performance is governed contractually through strict Service Level Agreements (SLAs) and monitoring, including financial consequences to the Technology Center vendor, SAIC, should SLAs ever be missed. The AOC has never had to exercise this contract clause due to missed SLAs.

The Technology Roadmap Workstream is working on the future of judicial branch technology.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #10 To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should obtain IV&V and IPO services at the beginning of the projects and ensure this independent oversight is in place throughout and follows best practices and industry standards appropriate for the size and complexity of the project.

Annual Follow-Up Agency Response From October 2012

The AOC continues to use the guidelines and framework established for graduated project oversight of information technology projects, including independent oversight, from the Information Technology Project Oversight Framework handbook when appropriate. The oversight level will be according to the evaluation of criticality and the risk level of the project. This will be initiated during the concept stage of the project. The AOC will also continue to operate under industry guidelines and standards of the Institute of Electrical and Electronics Engineers (IEEE) for verification and validation (V&V) activities as established in Standard 1012-2012. Independent V&V activities will be performed with the established standard's guidelines. For example, if a vendor is performing project oversight work, a separate vendor will perform the independent V&V activities. This will be done within the review and according to the recommendations of the California Technology Agency, as required by Government Code section 68511.9.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #11 To: Administrative Office of the Courts

To understand whether CCMS is a cost-beneficial solution to the superior courts' case management needs, the AOC should continue with its planned cost-benefit study and ensure it completes this study before spending additional significant resources on the project. The AOC should ensure that this study includes a thorough analysis of the cost and benefits of the statewide case management project, including a consideration of costs and benefits it believes cannot be reasonably quantified. The AOC should carefully evaluate the results of the study and present a recommendation to the Judicial Council regarding the course of action that should be taken with CCMS. Further, the AOC should fully share the results of the study as well as its recommendation to all interested parties, such as the superior courts, justice partners, the Legislature, and the Information Office. The AOC should update this cost-benefit analysis periodically and as significant assumptions change.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 132)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

.

Recommendation #12 To: Administrative Office of the Courts

To ensure the statewide case management project is transparent, the AOC should make sure all key decisions for future activities on CCMS are documented and retained.

1-Year Agency Response

The AOC stated all key decisions will be documented and all documentation provided to or produced by the CCMS governance committees and the CCMS Project Management Office will be retained throughout the life of the CCMS project. It also stated all available documentation predating this new governance model will also be retained throughout the life of the CCMS project. The AOC stated that CCMS documentation will be available to the public in a manner consistent with rule 10.500 of the California Rules of Court, which strives for transparency of judicial administrative records and to ensure the public’s right of access to such records. (See 2013-406, p. 132)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #13 To: Administrative Office of the Courts

To ensure its contract with the development vendor protects the financial interests of the State and the judicial branch, the AOC should consider restructuring its current contract to ensure the warranty for CCMS is adequate and covers a time period necessary to ensure that deployment of CCMS has occurred at the three early adopter courts and they are able to operate the system in a live operational environment.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 132)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #14 To: Administrative Office of the Courts

If the Judicial Council determines that CCMS is in the best interest of the judicial branch and it directs the AOC to deploy the system statewide, assuming funding is available, the AOC should ensure that any contract it enters into with a deployment vendor includes cost estimates that are based on courts’ existing IT environments and available resources to assist with deployment activities.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 132)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #15 To: Administrative Office of the Courts

If the Judicial Council determines that CCMS is in the best interest of the judicial branch and it directs the AOC to deploy the system statewide, assuming funding is available, the AOC should ensure that any contract it enters into with a deployment vendor includes well-defined deliverables.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 133)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #16 To: Administrative Office of the Courts

If the Judicial Council determines that CCMS is in the best interest of the judicial branch and it directs the AOC to deploy the system statewide, assuming funding is available, the AOC should ensure that any contract it enters into with a deployment vendor includes adequate responsibility is placed on the vendor for conducting key steps in the deployment of the system.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 133)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #17 To: Administrative Office of the Courts

The Judicial Council should make certain that the governance model for CCMS ensures that approval of contracts and contract amendments that are significant in terms of cost, time extension, and/or change in scope occur at the highest and most appropriate levels, and that when contracts or contract amendments above these thresholds are approved, that the decision makers are fully informed regarding both the costs and benefits.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 133)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #18 To: Administrative Office of the Courts

To ensure that any future IT projects are in the best interest of the judicial branch and the State, the AOC should complete a thorough analysis of the project’s cost and benefits before investing any significant resources and time into its development, and update this analysis periodically and as significant assumptions change.

1-Year Agency Response

The AOC stated it has been working diligently with the Technology Agency since its review of CCMS. The AOC further stated it has taken steps to integrate the Technology Agency’s recommendations into its existing technology project management process. The AOC reported this includes working with the Technology Agency on project concept documents and the project charters for future IT projects and using project planning documents more similar to those typically used for executive branch IT projects. (See 2013-406, p. 133)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #19 To: Administrative Office of the Courts

To ensure that any future IT projects are in the best interest of the judicial branch and the State, the AOC should document and retain all key decisions that impact the project in general, including the goals of the project.

1-Year Agency Response

The AOC indicates incorporating the Technology Agency’s recommendations into its existing processes, and using and retaining project concept documents, project charters, and other project planning documents more similar to those typically used for executive branch IT projects. (See 2013-406, p. 134)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #20 To: Administrative Office of the Courts

To ensure that any future IT projects are in the best interest of the judicial branch and the State, the AOC should better structure contracts with development and deployment vendors to protect the financial interests of the judicial branch and ensure the contracts provide for adequate warranty periods.

1-Year Agency Response

The AOC stated it will continue to work with the best qualified legal counsel to ensure that its development and deployment contracts protect the financial interests of the judicial branch and the State. The AOC also stated it will include appropriate warranty periods in IT projects and will ensure that any future development and deployment contracts address the length and timing of a warranty period to ensure necessary protection. (See 2013-406, p. 134)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #21 To: Administrative Office of the Courts

To ensure that the financial implications of the statewide case management project are fully understood, the AOC should report to the Judicial Council, the Legislature, and stakeholders a complete accounting of the costs for the interim systems and CCMS. This figure should be clear about the uncertainty surrounding some costs, such as those that the AOC and superior courts will incur for deployment of CCMS.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 134)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #22 To: Administrative Office of the Courts

The AOC should require superior courts to identify their past and future costs related to the project, particularly the likely significant costs that superior courts will incur during CCMS deployment, and include these costs in the total cost.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 134)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #23 To: Administrative Office of the Courts

The AOC should be clear about the nature of the costs that other entities, such as justice partners, will incur that are not included in its total.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 135)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #24 To: Administrative Office of the Courts

The AOC should update its cost estimate for CCMS on a regular basis as well as when significant assumptions change.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 135)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #25 To: Administrative Office of the Courts

To address the funding uncertainty facing CCMS, the AOC should work with the Judicial Council, the Legislature, and the governor to develop an overall strategy that is realistic given the current fiscal crisis facing the State.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 135)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #26 To: Administrative Office of the Courts

Although the Judicial Council has the legal authority to compel the courts to adopt CCMS, to better foster superior court receptiveness to deploying CCMS, the AOC should use the results from its consultant’s survey of the superior courts to identify and better understand the courts’ input and concerns regarding CCMS, including the manner in which the project has been managed by the AOC. To the extent the survey results indicate courts have significant concerns regarding CCMS or that they believe their case management systems will serve them for the foreseeable future, the AOC should take steps to address these concerns and overcome any negative perceptions and modify its deployment plan for CCMS accordingly.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 141)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #27 To: Administrative Office of the Courts

The AOC should continue working with local and state justice partners to assist them in their future efforts to integrate with CCMS, and in particular provide local justice partners the information needed to estimate the costs involved.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 142)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #28 To: Administrative Office of the Courts

Before embarking on future statewide IT initiatives and to ensure it secures appropriate support from users of the systems being proposed, the AOC should determine the extent to which the need for the IT initiative exists, including the necessary information to clearly demonstrate the extent of the problem the IT initiative will address.

1-Year Agency Response

The AOC stated it has both formal and informal processes and procedures in place to identify and assess the need for statewide technology improvements for the judicial branch in partnership with the courts. The AOC also stated it is committed to these processes and will continue to leverage these opportunities. As technology project needs are identified through these many communication channels, the AOC stated project concept documents are drafted that include statements of the problem, anticipated costs and benefits of the IT solution, impacts on courts and court operations, and known risks. (See 2013-406, p. 143)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #29 To: Administrative Office of the Courts

Before embarking on future statewide IT initiatives and to ensure it secures appropriate support from users of the systems being proposed, the AOC should take steps to ensure that superior courts support the solution the AOC is proposing to address the need, which could include conducting a survey of courts to determine their level of support.

1-Year Agency Response

The AOC stated regional meetings provide a solid foundation for the AOC and the courts to share information to learn about, better understand, and evaluate statewide technology needs. The AOC also stated the Judicial Council’s Court Technology advisory committee, trial court presiding judges advisory committee, and court executives advisory committee provide additional avenues of communication that enhance the exchange of information between and among the AOC and the courts to influence the direction and strategies for future statewide technology improvements. The AOC indicated that statewide meetings of presiding judges and court executive officers build on those committee meetings to ensure that superior court feedback is received. (See 2013-406, p. 143)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #30 To: Administrative Office of the Courts

Before embarking on future statewide IT initiatives and to ensure it secures appropriate support from users of the systems being proposed, the AOC should if necessary, determine whether other stakeholders, including local and state justice partners, support the IT initiative.

1-Year Agency Response

The AOC stated its Project Review Board is to ensure that all branch-wide technology projects follow a structured analysis protocol that will produce the information required to adequately assess the need for and value of the project proposal. The AOC further stated court and stakeholder surveys will be included in this structured analysis protocol. (See 2013-406, p. 143)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #31 To: Administrative Office of the Courts

To provide for an appropriate level of independent oversight on CCMS, the AOC should expand and clarify the scope of oversight services and require that oversight consultants perform oversight that is consistent with best practices and industry standards.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 144)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #32 To: Administrative Office of the Courts

To ensure that no gaps in oversight occur between CCMS development and deployment, the AOC should ensure that it has IV&V and IPO services in place for the deployment phase of CCMS. Further, to allow for independent oversight of the IV&V consultant, the AOC should use separate consultants to provide IV&V and IPO services.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 144)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #33 To: Administrative Office of the Courts

To ensure no significant quality issues or problems exist within CCMS, the AOC should retain an independent consultant to review the system before deploying it to the three early adopter courts. This review should analyze a representative sample of the requirements, code, designs, test cases, system documentation, requirements traceability, and test results to determine the extent of any quality issues or variances from industry standard practices that would negatively affect the cost and effort required of the AOC to operate and maintain CCMS. If any quality issues and problems identified by this review can be adequately addressed, and system development can be completed without significant investment beyond the funds currently committed, the AOC should deploy it at the early adopter courts during the vendor’s warranty period.

1-Year Agency Response

In March 2012 the Judicial Council voted to halt deployment of CCMS; thus, this recommendation is no longer relevant. (See 2013-406, p. 144)

California State Auditor's Assessment of 1-Year Status: Will Not Implement

Recommendation #34 To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should employ separate firms for IV&V and IPO services to allow for the IPO consultant to provide independent oversight on the IV&V consultant as well as the project team’s response to IV&V findings.

1-Year Agency Response

The AOC stated it will work closely with the Technology Agency on all future IT projects that will have a cost in excess of $5 million, and will carefully consider its recommendations for such projects, including those relating to oversight and risk mitigation. (See 2013-406, p. 145)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #35 To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should ensure that the staff performing IV&V and IPO services have experience and expertise that is commensurate with the size, scope, and complexity of the project they are to oversee.

1-Year Agency Response

The AOC stated it will work closely with the Technology Agency on all future IT projects that will have a cost in excess of $5 million, and will carefully consider its recommendations for such projects, including those relating to oversight and risk mitigation. (See 2013-406, p. 145)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #36 To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should ensure that independent oversight is not restricted in any manner and that all parties—the IV&V and IPO consultants, senior management, the project management team, and the development vendor—understand that the IV&V and IPO consultants are to have complete access to all project materials.

1-Year Agency Response

The AOC stated it will work closely with the Technology Agency on all future IT projects that will have a cost in excess of $5 million, and will carefully consider its recommendations for such projects, including those relating to oversight and risk mitigation. (See 2013-406, p. 145)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Recommendation #37 To: Administrative Office of the Courts

To ensure that future major IT projects receive appropriate independent oversight over technical aspects and project management, the AOC should address promptly and appropriately the concerns that independent oversight consultants raise.

1-Year Agency Response

The AOC stated it concurs with the importance of the identification of concerns raised by IV&V and IPO consultants and that their concerns be reported and monitored to ensure they are appropriately addressed. The AOC also stated concerns raised by IV&V and IPO consultants will be taken off watch status only after careful consideration and discussion of all risks and mitigation efforts that must occur to ensure that system function is unaffected. (See 2013-406, p. 145)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

All Recommendations in 2010-102

Agency responses received after June 2013 are posted verbatim.