Report 2014-116 Recommendation 14 Responses

Report 2014-116: California Department of Consumer Affairs' BreEZe System: Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost (Release Date: February 2015)

Recommendation #14 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

Annual Follow-Up Agency Response From October 2020

As indicated in CDT's October 2019 Annual Follow-Up Agency response, implicit in the submission of a Special Project Report (SPR) with the associated "Special Project Report Executive Approval Transmittal" certification by a department is: (i) the completion of a cost-benefit analysis undertaken with all due diligence through their governance processes, and (ii) the financial viability of the project. The California Department of Technology (CDT) approves the SPR after a rigorous analysis of the information submitted in the SPR, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Approval of the SPR signifies CDT's acknowledgment of the department's self-attestation of the completion of the cost benefit analysis reflecting the investment priority and financial viability of the project. Additionally, the SPR is subject to financial oversight and control by the Department of Finance, which further informs CDT final approval of the SPR.

It follows therefore, that a department would refrain from submitting an SPR if it is unable to substantiate the financial viability of continuing the project after completion of a cost benefit analysis. CDT is revising its SPR instructions to provide additional guidance on the steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. The updated SPR instructions will be posted to the website on or before November 17, 2020.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, CDT is a control agency and we expected it to ensure departments are appropriately certifying that they have evaluated the cost-benefit of continuing a project. Thus, we would expect CDT to perform a review of documentation supporting a department's certification to ensure it is justified. Further, although it has revised its SPR preparation instructions to reflect what steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy, it is still in the process of finalizing these instructions. Until these shortcomings are addressed, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2019

To address the audit concern of having no thresholds relating to increasing IT project costs, the Department of Technology (CDT) finalized an updated Project Status Report (PSR) and included in an update to SIMM 45 and associated Technology letter (TL 2017-07) in December 2017. The PSR includes instructions for determining the variance calculations relating to thresholds.

CDT has determined that the responsibility to determine the cost-benefit of all investments, including IT project investments rests with State entities. The director for the state entity ultimately has the responsibility to ensure that these decisions are undertaken with all due diligence through their governance processes. These entities must certify this through the "Special Project Report Executive Approval Transmittal" as updated April 2017.

The CDT analysis of the Special Project Reports (SPR) focuses on:

- budget and cost details for justification of further expenditures of the project.

- key performance factors of scope and schedule.

- health of the inflight project, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Unaddressed deficiencies in execution efficiency and effectiveness could result in the need for further substantiation of any added investment and may potentially result in non-approval of SPRs. Under some conditions this could result in further escalation

up to and including all remedies such as a corrective action plan, suspension, or reassignment, or termination.

Completion Status:

CDT asserts it has implemented threshold calculations for SIMM 45 in December 2017.

CDT has implemented new procedures to address CSA's concern for state agencies to conduct a cost-benefit of its IT investments via the Special Project Report Executive Approval Transmittal in April 2017.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, CDT is a control agency and we expected it to ensure departments are appropriately certifying that they have evaluated the cost-benefit of continuing a project. Thus, we would expect CDT to review documentation supporting a department's certification to ensure it is justified. Further, it has not yet updated its SPR preparation instructions, dated June 2014, to reflect what steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. Until these shortcomings are addressed, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2018

The Project Status Report (PSR) has been finalized and was included in an update to SIMM 45 and associated Technology letter (TL 2017-07) in December 2017. The PSR includes instructions for determining the variance calculations relating to thresholds.

The responsibility to determine the cost-benefit of all investments, including IT project investments rests with State entities. The director for the state entity ultimately has the responsibility to ensure that these decisions are undertaken with all due diligence through their governance processes. These entities must certify this through the "Special Project Report Executive Approval Transmittal" as updated April 2017.

The CDT analysis of the Special Project Reports (SPR) focuses on:

- the budget and cost details for justification of further expenditures of the project.

- other key performance factors of scope and schedule.

- the health of the inflight project, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Unaddressed deficiencies in execution efficiency and effectiveness could result in the need for further substantiation of any added investment and may potentially result in non-approval of SPR's. Under some conditions this could result in further escalation

up to and including all remedies such as a corrective action plan, suspension, or reassignment, or termination.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT has developed instructions for the development of an SPR that include thresholds relating to cost increases and schedule delays. Further, a focus of the IPO report is time management and cost, and as part of its new escalation process, CDT requires concerns raised by IPO and IV&V regarding costs and schedule to be escalated if not addressed by the respective department. These activities will help CDT justify why a project should continue.

Finally, CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, although CDT has made this change to its template, it has not yet updated its SPR preparation instructions, dated June 2014, to reflect what steps an agency should take if they cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. Until it does so, we will report this recommendation as partially implemented.

Annual Follow-Up Agency Response From November 2017

A new SPR template has been developed, and is ready for rollout at the end of January 2018.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2016

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to compare the relevant information from the financial worksheets of the original or most recently approved project plan to the proposed new project plan included in the SPR. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revision to the SPR analysis template is expected to be finalized by December 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to capture the relevant information concerning cost benefits analysis. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revised SPR analysis template is anticipated to be issued by April 2016.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

This recommendation is currently under development and we expect to be able to report progress for the next scheduled update.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

60-Day Update:

CalTech will add a section to the Special Project Report Template that will require the state entity to provide sufficient information on the cost-benefit or risk avoidance (as applicable) assessment that the state entity used to justify the approval of the SPR.

Progress on this initiative will be reported in our 6 month status update.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2014-116

Agency responses received are posted verbatim.