Report 2014-116 All Recommendation Responses

Report 2014-116: California Department of Consumer Affairs' BreEZe System: Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost (Release Date: February 2015)

Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include Consumer Affairs' plan for implementing BreEZe at those regulatory entities included in the project's third phase, including a timeline for the implementation.

Description of Legislative Action

As of January 2020, the Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

AB 507 was introduced on February 23, 2015, and would require the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill would also require the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

California State Auditor's Assessment of 1-Year Status: Legislation Introduced

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include the total estimated costs through implementation of the system at the remaining 19 regulatory entities and the results of any cost-benefit analysis it conducted for phase 3.

Description of Legislative Action

As of January 2020, the Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

AB 507 was introduced on February 23, 2015, and would require the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill would also require the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

California State Auditor's Assessment of 1-Year Status: Legislation Introduced

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of 6-Month Status: No Action Taken

Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include a description of whether and to what extent the system will achieve any operational efficiencies resulting from implementation by the regulatory entities.

Description of Legislative Action

As of January 2020, the Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

AB 507 would have required the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill also would have required the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

California State Auditor's Assessment of 1-Year Status: Legislation Proposed But Not Enacted

Description of Legislative Action

Legislation has not been introduced to address this recommendation

California State Auditor's Assessment of 6-Month Status: No Action Taken

Recommendation #4 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns raised by its IPO specialist and its IV&V consultant.

6-Month Agency Response

The department has implemented a process which involves summarizing concerns raised by its IPO Specialist and its IV&V consultant and the tracking Consumer Affairs' actions. This monitoring process is documented monthly in a Corrective Action Plan, which we have provided for the month of August 2015 (BreEZe CAP-1507) to CSA separately.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

60-Day Update:

Following approval of the Section 11 and subsequent execution of contract amendment #3 with Accenture, CalTech IT Project Oversight Division will follow-up with DCA and confirm DCA's completion of the conditions below which were placed on DCA for approving SPR 3.1. These are:

1. In order to ensure transparency and an appropriate level of decision making with a proper focus on scope and cost management, the DCA Project team must incorporate the BreEZe governance process and structures (including the Executive Steering Committee and Change Control Board), the roles and responsibilities of participants, their authority, and criteria for selection into a Governance Plan document within 30 business days following SPR 3.1 approval.

2. Within 30 business days following SPR 3.1 approval, the Project must provide the Department of Technology with an organization chart which shows the reporting relationships of the individuals who will be responsible for contract management, risk management, change management and schedule management. Beginning in April 2015, the Project must submit an updated project organizational chart identifying positions filled in the prior month and remaining open positions with the monthly Project Status Report (PSR) submission.

3. The Project must continue to maintain documentation of the current status and response to all Independent Verification and Validation (IV&V) and Independent Project Oversight Consultant (IPOC) findings which must be provided to the Department of Technology monthly along with the submission of monthly PSR.

CalTech will closely monitor Consumer Affairs' responsiveness and will report in its 6 month status update.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #5 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should require Consumer Affairs to develop and follow all project management plans and ensure that the BreEZe project team leads receive all required training.

1-Year Agency Response

Four BreEZe project team leads received California Project Management Methodology (CA-PMM) Boot Camp for Project Managers training since the last report. See the attached report.

In addition, all of the required plans have been completed, including those due in September 2015:

a) Quality Management Plan

b) Change Management Plan

c) Document Management Plan

The December 2015 Independent Project Oversight Report (IPOR) will be forwarded to the auditor for additional reference.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

There have been no new project team leads since the contract amendment was approved and therefore no training has been required.

Most required plans have been completed. However, the following three plans are still in process and will not be due until the middle of September 2015:

a) Quality Management Plan

b) Change Management Plan

c) Document Management Plan

We will report on the status of these plans in our next status update.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

60-Day Update:

Following approval of the Section 11 and subsequent execution of contract amendment #3 with Accenture, CalTech IT Project Oversight Division will follow-up with DCA and confirm DCA's completion of the conditions below which were placed on DCA for approving SPR 3.1. These are:

4. With the increase in staffing for the Project PMO and for organizational change management and because of the change in the system development methodology, the DCA project team must review and update the Project Management Plans listed below in the following priority order:

Within 30 business days of the implementation of contract amendment 3 for Design, Development, and Implementation (DDI):

a) Communications Management Plan

b) Transition Plan

Within 90 business days of the implementation of contract amendment 3 for DDI:

c) Maintenance Plan (if impacted by DDI contract amendment 3)

d) Organizational Change Management Plan

e) Quality management Plan

f) Change Management Plan

g) Document Management Plan

5. A baseline work plan for the remainder of the project must be completed and provided to the Department of Technology within 30 business days following the implementation of contract amendment 3 for DDI.

6. CalTech will also require that DCA provide updated information about the knowledge and experience of the individuals serving in the leadership and critical role positions for the project along with any plans for training for these staff. CalTech will determine the sufficiency of these plans and require that all necessary training take place within 120 days of the execution of contract amendment 3.

CalTech will closely monitor this situation and report in our 6 month status update.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should document key discussions with Consumer Affairs in which significant concerns are raised about Accenture and assessments of the department's performance by the IV&V consultant and the IPO specialist.

6-Month Agency Response

Attachment 2 will be sent separately via e-mail.

The Department's IT Project Oversight Division (ITPOD) established a procedure for documenting significant concerns and issues that are discussed in Portfolio meetings with Department executives. See Attachment 2.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

60-Day Update:

Over the next 30 days, ITPOC will establish a new procedure for documenting significant concerns and agreed upon actions discussed during meetings with Consumer Affairs.

The new procedures are expected to in place by June 2015.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #7 To: Technology, California Department of

If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project.

1-Year Agency Response

This recommendation is no longer applicable as DCA has completed Phase 2 implementation. To date, the BreEZe project remains within the budget approved in SPR 3.1. Through December 2015, the Project continues to operate under budget. Staffing and other costs have been below anticipated levels.

As of the end of December 2015, $917,560 remains in the Unanticipated Tasks Fund for future Requests for Change (RFC).

California State Auditor's Assessment of 1-Year Status: Resolved

Consumer Affairs implemented phase 2 of BreEZe and, based on documentation provided by CalTech, the project did not face escalating costs. As a result, it is no longer necessary for CalTech to require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project. Thus, we have reported the status of this recommendation as resolved.

6-Month Agency Response

Consumer Affairs has reported it will conduct an analysis of the costs and benefits of moving forward after the completion of Release #2.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

60-Day Update:

In the event that the BreEZE project continues to face escalating costs, CalTech will require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned in SPR 3.1. Failure on the part of DCA to do so will result in a recommendation of ITPOC to the Director to suspend or terminate the project.

CalTech will closely monitor this situation and report in our 6 month status update.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #8 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments undertake detailed business process analyses before submitting FSRs to CalTech for approval.

Annual Follow-Up Agency Response From October 2016

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #9 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to justify the type of IT solution they are proposing, such as a COTS or a fully customized system.

Annual Follow-Up Agency Response From October 2016

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #10 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to inform the resulting RFPs and that they ensure that the RFPs contain adequate requirements.

Annual Follow-Up Agency Response From October 2016

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #11 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to follow its IT policies, including developing all necessary plans and receiving all required training.

Annual Follow-Up Agency Response From November 2017

Statewide Information Management Manual (SIMM) 45 updated for departments to follow CDT policies.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2016

California Project Management Framework (CA-PMF) was published in May 2016. See TL 16-04. The CA-PMF offers guidance and insight on project management methods and approach (through the use of resources, tools, and templates), as well as narratives detailing the justifications for why specific activities should be performed. The CA-PMF is designed to be scalable to accommodate the various needs of projects.

Prior to completion of the CA-PMF, California Department of Technology (CDT) decided to separate policy mandates from project management guidance. In particular, CDT determined that the priority for policy mandates should be on reportable IT projects. As a result, CDT modified the IT Project Oversight Framework - SIMM 45 in July 2016 to include the oversight related components previously published in the CA-PMM.

These are:

- Project Management Risk Assessment

- Project Complexity Assessment

- Project Status Reports

- Independent Project Oversight Reports

Information Technology Project Oversight Division (ITPOD) has drafted an escalation process to raise awareness and foster appropriate actions concerning significant risks and issues to increasingly higher levels of CDT leadership. ITPOD has also drafted a Corrective Action Plan framework to notify departments that remediation is required. ITPOD intends to incorporate both of these processes to the IT Project Oversight Framework after state entities are provided with the opportunity to ask questions or provide comments - anticipated to be completed by December 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

Work on revisions to the (CA-PMM) is underway with implementation anticipated in July 2016. The new version will be re-named to: California Project Management Framework (CA-PMF). Release of the new CA-PMF will include the a policy clarification that departments must follow the CA-PMF or another documented methodology based on PMBOK (Project Management Institute Project Management Body of Knowledge). CA-PMM was written as a set of guidelines - not requirements.

ITPOD will document deficiencies concerning compliance with IT policies and when these are not remediated within a reasonable period of time, ITPOD will use an escalation process to bring greater attention to the resulting risks and issues to increasingly higher levels of leadership. Updates to the ITPOD Project Oversight Escalation process are under review, and implementation is expected March 2016.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Work on CA-PMM is underway with implementation anticipated in July 2016. Updates to the Project Oversight Escalation process are in development and implementation is planned for September 2015.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

60-Day Update:

CalTech is undertaking a project to update and enhance the CA-PMM. This will provide the opportunity to strengthen the direction given to state entities concerning the Department's IT policies. At the same time, ITPOC is enhancing and updating the Project Oversight Escalation process to ensure that ITPOC management is apprised of project compliance deficiencies and so that the recommendation for remediation action are developed.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #12 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should ensure that departments obtain IPO and IV&V services as soon as an IT project is approved. Additionally, CalTech should document key discussions with any department in which the IV&V consultant and the IPO specialist raise significant concerns about a project, and ensure that the respective department responds promptly to, and adequately addresses, the concerns that the IPO specialist and the IV&V consultant raise.

Annual Follow-Up Agency Response From November 2017

Written instructions have been completed and incorporated into SIMM 45 on the requirement for obtaining both IPO and IV&V services.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2016

ITPOD established the new positions approved in the FY 2016-17 State Budget. Efforts to fill these positions are on-going.

Starting in the latter half of 2015, ITPO began assigning staff on a part-time basis to provide assistance and evaluate departments going through PAL. This facilitates the planning process for ITPOD concerning when a project will be approved in Stage 4 and then will require the assignment of a full time oversight resource. Projects approved at stage 4 may have conditions that will be required depending on any specific risks or issues identified during the PAL. The assigned ITPOD staff will monitor compliance as the assigned project progresses.

For more than two years, ITPOD has included significant or unanswered findings from IV&V in the Independent Project Oversight Report (IPOR) on all projects subject to CDT oversight. And, as mentioned above, ITPOD has drafted an escalation process to raise awareness and foster appropriate actions concerning significant risks and issues to increasingly higher levels of CDT leadership. ITPOD has also drafted a Corrective Action Plan framework to notify departments that remediation is required. ITPOD intends to incorporate both of these processes to the IT Project Oversight Framework after state entities are provided with the opportunity to ask questions or provide comments - anticipated to be completed by December 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Department submitted a Budget Change Proposal for additional ITPOD staff, which has been included in the Governor's budget for FY 16/17. ITPO staff will be assigned to projects that are determined to be medium or high complexity in Stage 2 of the Project Approval Lifecycle. Projects approved at stage 4 may have conditions that will be required depending on identified risks or issues. The assigned ITPO staff will monitor compliance as the assigned project progresses.

As of June 2015 and reported in the 6th month update, the Department's IT Project Oversight Division (ITPOD) established a procedure for documenting significant concerns and issues that are discussed in Portfolio meetings with Department executives. Updates to the Project Oversight Escalation process are under review, and implementation is expected March 2016.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The Department continues to have discussions with the Department of Finance about an appropriate way to augment the ITPOD staffing allocation when new projects are expected to begin.

See the above status report (Attachment 2) for recommendation # 6 regarding discussing significant concerns during meetings with departments. The escalation process is planned for issuance September 2015.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

60 Day Update:

CalTech will engage the Department of Finance in a discussion about an appropriate way to augment the ITPOC staffing allocation when new projects are expected to begin.

Over the next 30 days, ITPOC will establish a new procedure for documenting significant concerns and agreed upon actions discussed during meetings with departments. This process will include follow-up and escalation procedures.

This new procedure is expected to be in place by June 2015.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #13 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to appropriately address deficiencies identified in CalTech's approval of any SPRs as conditions that must be met, and to do so according to specified timelines. If the deficiencies are not adequately addressed within the specified timelines, CalTech should take action to suspend the project until the department has either resolved the identified deficiencies or adequately documented its justification for not addressing the deficiencies.

Annual Follow-Up Agency Response From November 2017

Written procedures were developed and implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

The technology department revised SIMM Section 45 - Appendix G - by including a section in the IPO report template focused on the respective state department's actions to address conditions identified in the Technology Department's approval of any SPRs. Further, the technology department revised its Issue Escalation Process to more fully describe how issues or risks, including stagnant remediation of any conditions in the approval of an SPR, will be escalated to state department officials and to management of the technology department. Further, the Issue Escalation Process includes when and what the technology department should require as remedial actions, or the sanctions that it will impose when departments do not comply with these remedial actions.

Annual Follow-Up Agency Response From October 2016

ITPOD publishes the most recent project approval conditions and the status of each in the IPOR. If the project fails to make progress on the project approval conditions by the dates established in the approval letter, the IPOD manager notifies the Branch Chief and Deputy Director for discussion with the department and project leadership. This process has been successfully used in 2016, but with the publication of the Escalation and Corrective Action Plan policies referenced above, departments will have greater understanding of the importance of addressing the project approval conditions.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

Due to funding constraints and workload, the anticipated Project Planning & Lifecycle Automation system will be delayed until next year. But, ITPOD is now using a manual process to track compliance with project approval conditions. This includes documenting the conditions in the IPOR and reporting on progress each month. If the project fails to make progress on the project approval conditions, the IPO manager will follow the new draft Escalation Process that is expected to be finalized by March 2016. We are also developing a follow-on process for issuing a "Corrective Action" (CAP) requirement on an IT project. This will take the form of both policy and formal ITPOD process. Because the CAP will take the form of published policy, we expect to finalize it by June 2016.

Together the Escalation process and the CAP policy will contribute to the framework necessary to evaluate project problems and ensure appropriate attention and accountability are employed so that the problems can be resolved.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

This capability has been included in the requirements for the Department's new IT system project: Project Planning and Lifecycle Automation. Target date for the new system is September 2016. Until then, a manual process is required. The manual process is expected to be completed by September 2015.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

60-Day Update:

CalTech ITPOC will develop a standardized process to track the results of a state entities compliance with any/all conditions that CalTech places on approval of a Special Project Report. In the event a state entity fails to respond adequately to the conditions, the IPO will follow Project Oversight Escalation Process and ITPOC management will determine and/or recommend the appropriate remedial action.

The new process is expected to be implemented by July 2015.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #14 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

Annual Follow-Up Agency Response From October 2020

As indicated in CDT's October 2019 Annual Follow-Up Agency response, implicit in the submission of a Special Project Report (SPR) with the associated "Special Project Report Executive Approval Transmittal" certification by a department is: (i) the completion of a cost-benefit analysis undertaken with all due diligence through their governance processes, and (ii) the financial viability of the project. The California Department of Technology (CDT) approves the SPR after a rigorous analysis of the information submitted in the SPR, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Approval of the SPR signifies CDT's acknowledgment of the department's self-attestation of the completion of the cost benefit analysis reflecting the investment priority and financial viability of the project. Additionally, the SPR is subject to financial oversight and control by the Department of Finance, which further informs CDT final approval of the SPR.

It follows therefore, that a department would refrain from submitting an SPR if it is unable to substantiate the financial viability of continuing the project after completion of a cost benefit analysis. CDT is revising its SPR instructions to provide additional guidance on the steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. The updated SPR instructions will be posted to the website on or before November 17, 2020.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, CDT is a control agency and we expected it to ensure departments are appropriately certifying that they have evaluated the cost-benefit of continuing a project. Thus, we would expect CDT to perform a review of documentation supporting a department's certification to ensure it is justified. Further, although it has revised its SPR preparation instructions to reflect what steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy, it is still in the process of finalizing these instructions. Until these shortcomings are addressed, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2019

To address the audit concern of having no thresholds relating to increasing IT project costs, the Department of Technology (CDT) finalized an updated Project Status Report (PSR) and included in an update to SIMM 45 and associated Technology letter (TL 2017-07) in December 2017. The PSR includes instructions for determining the variance calculations relating to thresholds.

CDT has determined that the responsibility to determine the cost-benefit of all investments, including IT project investments rests with State entities. The director for the state entity ultimately has the responsibility to ensure that these decisions are undertaken with all due diligence through their governance processes. These entities must certify this through the "Special Project Report Executive Approval Transmittal" as updated April 2017.

The CDT analysis of the Special Project Reports (SPR) focuses on:

- budget and cost details for justification of further expenditures of the project.

- key performance factors of scope and schedule.

- health of the inflight project, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Unaddressed deficiencies in execution efficiency and effectiveness could result in the need for further substantiation of any added investment and may potentially result in non-approval of SPRs. Under some conditions this could result in further escalation

up to and including all remedies such as a corrective action plan, suspension, or reassignment, or termination.

Completion Status:

CDT asserts it has implemented threshold calculations for SIMM 45 in December 2017.

CDT has implemented new procedures to address CSA's concern for state agencies to conduct a cost-benefit of its IT investments via the Special Project Report Executive Approval Transmittal in April 2017.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, CDT is a control agency and we expected it to ensure departments are appropriately certifying that they have evaluated the cost-benefit of continuing a project. Thus, we would expect CDT to review documentation supporting a department's certification to ensure it is justified. Further, it has not yet updated its SPR preparation instructions, dated June 2014, to reflect what steps a department should take if it cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. Until these shortcomings are addressed, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2018

The Project Status Report (PSR) has been finalized and was included in an update to SIMM 45 and associated Technology letter (TL 2017-07) in December 2017. The PSR includes instructions for determining the variance calculations relating to thresholds.

The responsibility to determine the cost-benefit of all investments, including IT project investments rests with State entities. The director for the state entity ultimately has the responsibility to ensure that these decisions are undertaken with all due diligence through their governance processes. These entities must certify this through the "Special Project Report Executive Approval Transmittal" as updated April 2017.

The CDT analysis of the Special Project Reports (SPR) focuses on:

- the budget and cost details for justification of further expenditures of the project.

- other key performance factors of scope and schedule.

- the health of the inflight project, including risk and issues impacting budget and cost, as a reflection of execution efficiency and effectiveness of project performance.

Unaddressed deficiencies in execution efficiency and effectiveness could result in the need for further substantiation of any added investment and may potentially result in non-approval of SPR's. Under some conditions this could result in further escalation

up to and including all remedies such as a corrective action plan, suspension, or reassignment, or termination.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

CDT has developed instructions for the development of an SPR that include thresholds relating to cost increases and schedule delays. Further, a focus of the IPO report is time management and cost, and as part of its new escalation process, CDT requires concerns raised by IPO and IV&V regarding costs and schedule to be escalated if not addressed by the respective department. These activities will help CDT justify why a project should continue.

Finally, CDT updated its SPR template to require departments to certify that they have considered the cost benefit analysis associated with the proposed project changes and certify that the changes are consistent with its strategy. However, although CDT has made this change to its template, it has not yet updated its SPR preparation instructions, dated June 2014, to reflect what steps an agency should take if they cannot certify the SPR because the cost benefit analysis is not consistent with the department's strategy. Until it does so, we will report this recommendation as partially implemented.

Annual Follow-Up Agency Response From November 2017

A new SPR template has been developed, and is ready for rollout at the end of January 2018.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2016

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to compare the relevant information from the financial worksheets of the original or most recently approved project plan to the proposed new project plan included in the SPR. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revision to the SPR analysis template is expected to be finalized by December 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to capture the relevant information concerning cost benefits analysis. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revised SPR analysis template is anticipated to be issued by April 2016.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

This recommendation is currently under development and we expect to be able to report progress for the next scheduled update.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

60-Day Update:

CalTech will add a section to the Special Project Report Template that will require the state entity to provide sufficient information on the cost-benefit or risk avoidance (as applicable) assessment that the state entity used to justify the approval of the SPR.

Progress on this initiative will be reported in our 6 month status update.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #15 To: Consumer Affairs, Department of

To make certain that Consumer Affairs has a project team that consists of staff trained in managing IT projects, it should ensure that all the BreEZe project team leads obtain all required project management training.

1-Year Agency Response

The Department has completed a comprehensive project management training schedule to ensure that the project leads have the required project management training, in compliance with Statewide Information Management Manual Section 17E (California Project Management Methodology (CA PMM)). See Attachment 1 for complete training schedule.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

To fully implement this recommendation, the Department has put in place a project management training schedule to ensure that the project leads obtain the required project management training, in compliance with Statewide Information Management Manual Section 17E (California Project Management Methodology (CA PMM)). Two of the four identified team leads have been scheduled for the required CA PMM Basic training, which will be completed by the end of September 2015. The remaining two identified team leads' required CA PMM Basic training should be completed by January 2016. Additionally, the Project Director and Project Manager have Project Management Professional (PMP) certifications. For the new lead staff identified in the Department's SPR 3.1, the Department has included additional funding to ensure the project is prepared to train those staff.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

To fully implement this recommendation, the Department has put in place a project management training schedule to ensure that the project leads obtain the required project management training, in compliance with Statewide Information Management Manual Section 17 (California Project Management Methodology). Additionally, the Project Director and Project Manager have Project Management Professional (PMP) certifications. For the new lead staff identified in the Department's SPR 3.1, the Department has included additional funding to ensure the project is prepared to train those staff. (Attachment 1)

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #16 To: Consumer Affairs, Department of

Consumer Affairs should develop a process to ensure that it prepares all required project management documents and undertakes all oversight activities related to BreEZe as CalTech requires so that it can prevent or identify and monitor future problems as they arise. This includes taking steps to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise.

60-Day Agency Response

The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase. Additionally the Department, in collaboration with Department of Technology (CalTech), has implemented a process to clearly document all activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. This documentation identifies the steps taken to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise and is reviewed in its entirety with CalTech oversight staff each month at a minimum. (Attachments 2 & 3)

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #17 To: Consumer Affairs, Department of

To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost-benefit analysis to ensure that BreEZe is a cost-effective solution to meet these regulatory entities' business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost-benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change.

Annual Follow-Up Agency Response From October 2021

The Department (department) of Consumer Affairs' Business Modernization Plan has successfully delivered initial technology releases for 6 of 16 boards and bureaus (boards) through 2021 and has achieved many planning milestones for the remaining boards. The boards that have deployed modernized online services as of September 2021 are the:

1. California Acupuncture Board,

2. Board of Chiropractic Examiners,

3. Board of Professional Engineers, Land Surveyors, and Geologists,

4. Court Reporters Board,

5. Bureau for Private Postsecondary Education, and

6. Contractors State License Board. The Contractors State License Board implemented their modernized online services after the last reporting period.

Since the last update, two additional boards have completed business process mapping: the Board of Accountancy and the Speech-Language Pathology & Audiology & Hearing Aid Dispensers Board. Also, one additional board has completed the Project Approval Lifecycle (PAL) Stage 1: the Speech-Language Pathology & Audiology & Hearing Aid Dispensers Board. Finally, six boards completed their PAL Stage 2 since the last update: the Speech-Language Pathology & Audiology & Hearing Aid Dispensers Board, the Cemetery and Funeral Bureau, the Bureau of Household Goods and Services, the California Architects Board/Landscape Architects Technical Committee, the Structural Pest Control Board, and the Bureau of Automotive Repair. The chart below reflects the current progress.

Table 1. Phase 3 PAL and Business Activities Status

Phase 3 boards/bureaus with completed business process mapping: 14

PAL Stage 1 Completed: 15

PAL Stage 2 Completed: 13

PAL Stage 3 Completed: 7

PAL Stage 4 Completed: 7

Completed Deployment of Modernized Software: 6

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2020

The Department (department) of Consumer Affairs' Business Modernization Plan of 2017 has effectively created a structured methodology for planning and implementing business process improvements and efficient online services to better serve the 3.2 million licensees and the department's many stakeholder groups. The plan has successfully delivered initial technology releases for 5 of 16 boards and bureaus (boards) in 2020 and has achieved many planning milestones for the remaining boards. The boards that have deployed modernized online services in 2020 are: the California Acupuncture Board, the Board of Chiropractic Examiners, the Board of Professional Engineers, Land Surveyors, and Geologists, Court Reporters Board, and the Bureau for Private Postsecondary Education. The chart below reflects the current progress.

Table 1. Phase 3 PAL and Business Activities Status

Phase 3 boards/bureaus with completed business process mapping: 12

PAL Stage 1 Completed: 14

PAL Stage 2 Completed: 7

PAL Stage 3 Completed: 7

PAL Stage 4 Completed: 7

Completed Deployment of Modernized Software: 5

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

DCA provided documentation demonstrating that several regulatory entities have completed Stage 2 of the PAL process, and provided documentation that it believes constitutes cost benefit analyses. However, it stated that there are nine regulatory entities that have not yet completed Stage 2 of the PAL process. Thus, the cost-benefit analyses for these remaining entities have not yet been finalized. Until such analyses are developed and approved, we will report this recommendation as partially implemented.

Annual Follow-Up Agency Response From November 2019

As reported in 2018, the Department of Consumer Affairs (DCA) has established a standard for planning, analyzing, documenting and justifying all reportable technological investments. Our standards, outlined in the department's 2017 Business Modernization Plan (updated in 2018.), require that the cost/benefit of an investment be analyzed and documented in Stage 2 of the Department of Technology's (CDT) Project Approval Lifecycle (PAL). Executive leadership from the board/bureau, DCA, the Business, Consumer Services and Housing Agency, the Department of Finance, and CDT then must approve the PAL documents validating the cost/benefit. The recommendation calls out specifically further investments in Breeze, but DCA has extended this requirement to all reportable technological investments.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

DCA provided documentation demonstrating that four regulatory entities have completed Stage 2 of the PAL process, and provided documentation that it believes constitutes cost benefit analyses. However, there are several regulatory entities that have not yet completed Stage 2 of the PAL process. Thus, the cost-benefit analyses for these remaining entities have not yet been finalized. Until such analyses are developed and approved, we will report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2018

The Department of Consumer Affairs, in collaboration with phase 3 regulatory entities, has developed a Business Modernization Plan (and accompanying reports) to effectively facilitate the analysis, approval, and potential transition to a new licensing and enforcement platform. The Plan outlines our approach to identifying board/bureau specific business needs, determining cost-effective options, and proposes initial schedules. The Plan also incorporates the impacts and requirements of the Project Approval Lifecycle (PAL) - the project approval process required by the California Department of Technology. Each board or bureau will perform a cost- benefit analysis, or determine the cost effectiveness of each alternative platform during Stage 2 of the PAL process. The Financial Analysis Worksheets (FAW) facilitate fiscal planning and comparison and will be used to determine cost benefit on a board by board basis.

The phase 3 regulatory entities are in the following stages of PAL and business activities status: two entities have PAL Stage 1 completed, four entities for PAL Stage 1 in progress, two entities have PAL Stage 2 in progress, and nine entities have completed business process mapping.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2017

The Department of Consumer Affairs, in conjunction with phase 3 regulatory entities, has developed a Business Modernization Plan (Plan) to effectively facilitate the analysis, approval, and potential transition to program specific licensing and enforcement platforms. The Plan outlines our approach to identifying board/bureau specific business needs, determining cost effective options, and proposes initial schedules. The Plan also considers the impacts and requirements of the Project Approval Lifecycle (PAL), the project approval process required by the California Department of Technology. Each board/bureau will perform a cost benefit analysis, or determine the cost effectiveness of each alternative platform, during Stage 2 of the PAL process. The Financial Analysis Worksheets (FAW) facilitate fiscal planning and comparison, and will be used to determine cost benefit on a board by board basis.

Currently there are four boards/bureaus actively documenting their current business processes to identify needs and requirements. The remaining boards/bureaus are working to inventory their business processes and schedule documentation workshops with departmental Organizational Change Management staff. One program has submitted Stage 1 of the PAL process and a second is scheduled to submit theirs within 90 days.

The thorough planning, business analysis, and program-specific nature of this effort will ensure success for our remaining boards and bureaus (formerly Release/Phase 3).

Implementation Timeframe: Proposed timelines are included and maintained in a board/bureau's individual Business Modernization Reports and may change due to business resource contention.

Individual board/bureau progress through business modernization will vary based on individual program resources, current legislative/business landscape, complexity, fiscal landscape, and the PAL process.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2016

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2016

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare all existing BreEZe functionality as delivered at the completion of Release 2, to the Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe would be the most cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare all existing BreEZe functionality as delivered at the completion of Release 2, to the Release 3 boards and bureaus' business needs and current systems' functionality to identify necessary BreEZe enhancements. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #18 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, system requirements that are specific to each regulatory entity as applicable.

1-Year Agency Response

The Department, working with the California Department of Technology, will continue to follow all statewide IT policy applicable to the Systems Development Life Cycle. For all future IT projects, the Department has developed a formal policy so that any future IT project will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application. See Attachment 3 for aforementioned policy.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Department, working with the California Department of Technology, will continue to utilize/follow the Systems Development Life Cycle model/process (the Systems Development Life Cycle model describes the process for planning, creating, testing, and deploying an information system).

For all future IT projects, the Department will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application.

California State Auditor's Assessment of 6-Month Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it includes system requirements that are specific to each regulatory entity as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

60-Day Agency Response

The Department, working with the California Technology Agency, will continue to utilize/follow the Systems Development Life Cycle process for all future IT projects. The Department will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application.

California State Auditor's Assessment of 60-Day Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.

Recommendation #19 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, a project team that is qualified and experienced.

Annual Follow-Up Agency Response From October 2016

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team that is qualified and experienced. With the release of Technology Letter 16-07 in August 2016, all four Stages of the Project Approval Lifecycle are complete and in place.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team that is qualified and experienced.

California State Auditor's Assessment of 1-Year Status: Pending

According to the Department of Technology's one-year response to recommendations eight through ten regarding its four stage project review and approval process, Stage 4 of that approval process is scheduled for final release in July 2016. Thus, until that stage is finalized and includes steps to ensure that any future IT project includes a project team that is qualified and experienced, we will continue to report the status of this recommendation as pending.

6-Month Agency Response

The Department has implemented additional procedures for IT projects based on lessons learned. For any new IT project, a project staffing plan will be required prior to embarking on the project. At a minimum, the components of the project staffing plan will include the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs.

California State Auditor's Assessment of 6-Month Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, its project team that oversees any new IT project is qualified and experienced, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response

60-Day Agency Response

The Department will develop a project staffing plan prior to embarking on any new IT project. Items that will be included in the project staffing plan are the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs.

California State Auditor's Assessment of 60-Day Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.

Recommendation #20 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT system is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, development of and adherence to all project management plans.

60-Day Agency Response

The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase. (Attachment 2)

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #21 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, planning and implementation of effective organizational change management.

1-Year Agency Response

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and assist in the implementation of new or revised business processes specific to each entity's business. At the conclusion of BreEZe Release 2 OCM activities the OCM documentation will be available. The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase (Attachment 2). This Department process requires any future IT project to include appropriate organizational change management activities as a project critical success factor.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and assist in the implementation of new or revised business processes specific to each entity's business. At the conclusion of BreEZe Release 2 OCM activities the OCM documentation will be available. Prior to any new IT project in the future, the Department will initiate this new organizational change management process making any refinements based on additional lessons learned and on the needs of the new project.

California State Auditor's Assessment of 6-Month Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it includes planning and implementation of effective organizational change management as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

60-Day Agency Response

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes. This process will be used for all future IT projects.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities. Thus, although Consumer Affairs has taken action to assist the phase 2 regulatory entities as it relates to organizational change management, not until it develops and finalizes a plan for future IT projects will we be able to fully assess its progress in implementing this recommendation.

Recommendation #22 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, timely response to concerns that the IV&V consultant and the IPO specialist raise.

Annual Follow-Up Agency Response From October 2016

The Department, in collaboration with Department of Technology, implemented a process to document and report activities the project undertakes in response to Project Oversight findings related to BreEZe. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with Project Oversight staff no less than once per month. The Department has developed a standard response template (Attachment 1) that will ensure future IT projects have the tool to facilitate clear communication with oversight staff and provides timely documented response to IV&V and/or IPO identified concerns.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The Department, in collaboration with Department of Technology, implemented a process to document and report activities the project undertakes in response to Project Oversight findings related to BreEZe. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with Project Oversight staff no less than once per month. The Department is developing a standard process and response template that will ensure future IT projects provide timely documented response to IV&V and/or IPO identified concerns.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The Department, in collaboration with Department of Technology (CalTech), implemented a process to document and report activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with CalTech oversight staff at least once a month. These reports are available and can be requested from CalTech's IPO and IV&V staff. Prior to any new IT project in the future, the Department will follow this or a similar model to ensure timely responses to oversight staff concerns.

California State Auditor's Assessment of 6-Month Status: Pending

Although Consumer Affairs indicates that it collaborated with CalTech and implemented a process to document and report activities the project undertakes in response to concerns raised by the IV&V consultant and IPO specialist, this process is specific to BreEZe. As a result, not until Consumer Affairs develops a standard process that staff are required to use on any new IT projects it may undertake in the future will we consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

60-Day Agency Response

The Department, in collaboration with Department of Technology (CalTech), has implemented a process to clearly document all activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. This documentation identifies the steps taken to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise and is reviewed in its entirety with CalTech oversight staff each month at a minimum. This process will be used for all future IT projects.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities. Thus, although it reported that it has developed a process for responding to oversight findings related to BreEZe, not until it develops a plan for future IT projects that adresss this recommendation will we be able to fully assess its progress in implementing this recommendation.

Recommendation #23 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, sufficient staffing.

Annual Follow-Up Agency Response From October 2016

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available, sufficient, and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle (fully implemented as of August 29, 2016 per Technology Letter 16-07) will ensure that any future IT project at the Department includes a project team of sufficient size.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available, sufficient, and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team of sufficient size.

Additionally, the Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase (Attachment 2). This Department process requires any future IT project to include a Human Resources (Staffing) Plan as a project critical success factor.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

According to the Department of Technology's one-year response to recommendations eight through ten regarding its four stage project review and approval process, Stage 4 of that approval process is scheduled for final release in July 2016. Thus, until that stage is finalized and includes steps to ensure that any future IT project includes a project team that is qualified and experienced, we will continue to report this recommendation as not fully implemented. However, because Consumer Affairs has developed a project management plan development procedure, which includes a Human Resources Management plan, we will report this recommendation as partially implemented.

6-Month Agency Response

As stated in the response to Recommendation #19, a project staffing plan will be required prior to embarking on any new IT project. Further to ensure sufficient staffing, the plan will include obtaining commitments from key staff (boards and bureaus) before actual work begins.

California State Auditor's Assessment of 6-Month Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it has sufficient staffing as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

60-Day Agency Response

The Department will develop a project staffing plan prior to embarking on any new IT project. Items that will be included in the project staffing plan are the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs. Additionally, the plan will include obtaining commitments from key staff (boards and bureaus) before actual work begins.

California State Auditor's Assessment of 60-Day Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.

Recommendation #24 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should document its reasons for approving any deviations from standard contract language.

Annual Follow-Up Agency Response From October 2016

In August 2015, the Department's Procurement division piloted its draft desk procedures for approving any deviations from standard contract language. Desk procedures finalization and full implementation for all IT reportable projects were completed May 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

In August 2015, the Department's Procurement division piloted its draft desk procedures for approving any deviations from standard contract language. Desk procedures finalization and full implementation for all IT reportable projects is expected by March 2016

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The Department's Procurement division currently documents the reasons for any deviations from standard contract language (GSPD 401IT) utilizing formal approval threshold criteria. The Procurement division has developed draft desk procedures for this area, which are currently under internal review and expect to be finalized by September 2015.

California State Auditor's Assessment of 6-Month Status: Pending

Until CalTech finalizes its draft desk procedures for this area, we will continue to report the status of this recommendation as pending.

60-Day Agency Response

CalTech is currently developing the below new criteria. The following is the GSPD 401IT Paragraph Modification Threshold Criteria that STPD has proposed:

To modify a "carve out" paragraph of the GSPD (i.e., "unless otherwise specified in the Statement of Work":

- The Department must show, and CalTech must agree, that there is added clarity to the contract or value added to an important State position in the agreement as a result of changing the contract term.

To modify a "non-carve out" paragraph of the GSPD:

- The Department must show, and CalTech must agree, that a critical Department objective cannot be met without changing the contract term.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #25 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language gives departments the rights to the source code necessary to complete the project if a contract is terminated for convenience.

60-Day Agency Response

Since April 2014, CalTech Procurement ensures the contract language gives the rights to the source code necessary to complete the project if a contract is terminated for convenience utilizing standard contract language (GSPD 401IT).

This action has been completed

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #26 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language contains assurances that state funds will not be used for unauthorized purposes.

60-Day Agency Response

Since April 2014, CalTech Procurement ensures contract language contains assurances that state funds will not be used for unauthorized purposes utilizing standard contract language (GSPD 401IT).

This action has been completed

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #27 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contracting departments can appropriately manage IT maintenance support contracts by including service-level agreements in these contracts.

60-Day Agency Response

Since April 2014, CalTech Procurement ensures that contracting departments can appropriately manage IT maintenance support contracts by including service level agreements in these contracts utilizing standard contract language (GSPD 401IT) or special contract provisions.

This audit recommended action has been completed.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #28 To: Technology, California Department of

To ensure that contracting departments comply with requirements, CalTech should periodically monitor the status of contracts and enforce adherence to procurement requirements. For example, CalTech should ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract.

Annual Follow-Up Agency Response From October 2018

SIMM 45 and the newly released Project Status Report (PSR), issued December 2017, specifically address scope changes and the management of change requests processes as a monthly review area of CDT's Oversight analysts. Furthermore, there is second level review as Oversight managers review all entities' PSR and IPORs.

The PSRs, which are initially reviewed by Oversight analysts, asks the following questions:

Variance detail chart:

1. Was work done that was not planned?

2. Were there any changes to the scope?

3. Were any deliverables or milestones removed?

Look Ahead view:

1. Is there any unplanned work that needs to be done?

2. Are there any expected or recommended changes to the scope?

3. Are there any deliverables that should be removed from the plan?

Additionally, the IPOR Assessment Criteria section in the IPOR template includes the following criteria for the Scope Management area:

- Are change requests are consistent with the approved scope of the project and are managed and monitored through completion and close out?

If any of these items identified above are perceived as a risk by Oversight analysts or managers, the Oversight framework (SIMM 45) specifically identifies our Statewide Technology Procurement Division as a point of escalation for Level 2. The Oversight manager is also directed to escalate unmitigated variances as outlined in the Project Oversight Escalation Process that was updated in December 2017.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

CDT has updated its assessment criteria within SIMM Section 45 regarding the contents of IPO reports to specifically state that IPO is to ensure change requests are consistent with the proposed scope of the project and they are managed and monitored through completion and close out. Further, its escalation process allows for the escalation of issues to its statewide technology procurement department and further specifies that change requests deemed to be outside of the approved scope or budget should be reviewed by its procurement department for validation. Thus, this recommendation is fully implemented.

Annual Follow-Up Agency Response From November 2017

CDT has updated the SPR preparation instructions to reflect potential alternatives if cost-benefit analysis is incomplete.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

In following up with CalTech regarding its implementation of this recommendation, we learned that it has not yet developed procedures to ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract. Until it develops such procedures, we will report this recommendation as pending.

Annual Follow-Up Agency Response From October 2016

ITPOD will examine change requests in the normal course of the oversight activity and if any are identified as exceeding the scope and intent of the current contract, ITPOD will raise that issue with the department and will bring the issue to the attention of the State Technology and Procurement Division (STPD), as needed. STPD will conduct a more detailed review to make the final determination whether or not the changes exceed the scope of the respective contract. Procedural documentation of these new oversight activities are expected to be completed by December 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

It is important to recognize that change requests are often necessary and proper. ITPOD uses SIMM 45, Appendix F: Project Oversight Review Checklist to evaluate the conditions which necessitate changes to the contract as well as the processes for ensuring adequate decision making at the appropriate level for issuing a change request. This includes verifying that key specification documents (e.g. contracts, requirement specifications and/or contract deliverables) and software products are under formal configuration control, with items to be controlled and specific staff roles and responsibilities for configuration management identified in a configuration management plan. This also includes verification that a formal governance process is in place, which necessitates the documentation of the reasons for approval of change requests.

ITPOD will examine change requests in the normal course of the oversight activity and if any are identified as exceeding the scope and intent of the current contract, ITPOD will raise that issue with the department and will bring the issue to the attention of the State Technology and Procurement Division (STPD) as needed. STPD will conduct a more detailed review to make the final determination whether or not the changes exceed the scope of the respective contract.

California State Auditor's Assessment of 1-Year Status: Pending

In following up with CalTech regarding its implementation of this recommendation, we learned that it has not yet developed procedures to ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract. Until it develops such procedures, we will report this recommendation as pending.

6-Month Agency Response

The Department's Procurement division will continue to work with ITPOD to identify if a Change Request has deviated outside a procurement requirement. The division will then be notified to conduct a more detailed review to determine if the changes exceed the scope of the respective contract.

We expect to be able to report progress for the next scheduled update.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

To address the audit recommendation that CalTech review all contract requests, STPD has formally requested staff augmentation. We will be able to provide an update on the progress of this additional resource request in our 6-month status update report.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #29 To: Consumer Affairs, Department of

To ensure that it complies with all terms of the BreEZe contract, Consumer Affairs should develop and follow a process for doing so, including documenting how it has met applicable contract terms, such as forwarding all change requests to CalTech.

1-Year Agency Response

The Department has developed a process to ensure that it complies with all terms of the BreEZe contract. This process and documentation format has been added to the project's Contractual Obligations Management Plan (Attachment 4).

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Department has a robust process in place to document how each system requirement within the contract has been met which is traceable from initial contract requirement to the implemented system. Following the pending execution of amendment 3 to the BreEZe implementation contract, the contract manager will use a similar robust process for all contract administrative requirements. This process shall document how all terms of the BreEZe contract have been satisfied, including contract terms such as forwarding all change requests to California Department of Technology.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department has a robust process in place to document how each system requirement within the contract has been met which is traceable from initial contract requirement to the implemented system. Following the pending execution of amendment 3 to the BreEZe implementation contract, the contract manager will develop a similar robust process for all contract administrative requirements. This process shall document how all terms of the BreEZe contract have been satisfied, including contract terms such as forwarding all change requests to CalTech.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #30 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor the timeliness of its processing of applications by type and the cause of any delays.

Annual Follow-Up Agency Response From November 2017

As a result of the Auditor's January 2017 response, the BRN completed development of more than 15 reports to formally track and monitor applications by type and deficiency. Some reports are reviewed daily, weekly, and monthly by managers as noted in the Succession Plan: Licensing Management Manual (SPLMM). Instructions and guidance for usage of these reports by management are formally outlined in the SPLMM on pages 10-14. Management's review of the reports is to ensure applications are processed in accordance with regulatory timeframes, identify delays in application processing, and remedy any causes of such delay.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From January 2017

Beginning in May 2016 the BRN has been able to formally track and monitor applications by type by utilizing the new Quality Business Interactive Reporting Tool (QBIRT). The BRN has been tracking application deficiency periods and now can track the specific cause for the delay because the BRN further refined the QBIRT report to track deficiencies within BreEZe to identify specific causes of application delays. The BRN reviews this report on a bi-weekly basis to ensure applications are evaluated on a timely basis and to remedy any causes for delay within BRNs purview.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Although BRN was able to provide reports demonstrating that it now has the capability to track and monitor the timeliness of its processing of applications by type and the cause of any delays, it has not yet formalized its process for reviewing these reports. Thus, until BRN sets forth this review process in its policies and procedures, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2016

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement to track application processing at specified points within the application life cycle and the data has been used to track the timeliness of application processing by type. This enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for the BRN starting July 1, 2015.

Additionally, starting in May 2016 the BRN has been able to formally track and monitor applications by type by utilizing the new Quality Business Interactive Reporting Tool (QBIRT). The BRN can track application deficiency periods, but cannot currently track the specific cause for the delay. The BRN is working with DCA BreEZe staff to update and create new specific deficiency tracking points within BreEZe to allow for more detailed application deficiency tracking which will allow the BRN the ability to identify specific causes of application delays.

The BRN anticipates implementation of these changes by July 1, 2017 and once completed this recommendation will be considered fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for use starting July 1, 2015.

Reports were released into production on January 19, 2016, when release 2 was deployed in BreEZe. BRN staff is verifying data to determine the accuracy of information provided in the reports.

The BRN continues to use data extracts that were created to help ensure applications are processed in a timely manner.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for use starting July 1, 2015.

The report is still under development for use by boards and bureaus utilizing BreEZe and is expected to be available by July 1, 2016.

In the interim, staff members are working with DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

BRN staff is working with Department of Consumer Affairs (DCA) and has initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement request was presented at the March 24, 2015 Change Control Board (CCB) for review and approval. The CCB approved the system enhancement and expects to have the enhancement in BreEZe for use by July 1, 2015.

In addition, the BRN is requesting an additional system enhancement to track the initial review date for each application type as this was not included in the DCA's original enhancement request. It is hoped this additional system enhancement will be included in the release by July 1, 2015.

In the interim, staff members are working with the DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #31 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor its pending workload of applications by type and original receipt date.

Annual Follow-Up Agency Response From October 2016

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which provides us with statistical information for incomplete applications. The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members worked with DCA to design and develop data extracts to help ensure applications are processed in a timely manner. Additionally, starting in May 2016, the BRN has been able to create application specific reports and formally track and monitor applications by utilizing the Quality Business Interactive Reporting Tool(QBIRT).

The BRN continues to maintain the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to help ensure applications are processed within the regulatory time frame.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members worked with DCA to design and develop data extracts to help ensure applications are processed in a timely manner.

Reports were released into production on January 19, 2016, when release 2 was deployed in BreEZe. BRN staff is verifying data to determine the accuracy of information provided in the reports.

The BRN continues to maintain the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to help ensure applications are processed within the regulatory time frame.

California State Auditor's Assessment of 1-Year Status: Pending

Although BRN indicated it has released reports into production that address this recommendation, based on additional follow-up with BRN regarding its response, the assistant executive director explained that BRN is not comfortable with the data in the report regarding its pending workload and is working with Consumer Affairs to resolve its concerns. Given the uncertainty surrounding the accuracy of these reports, we will continue to report this recommendation as pending until these concerns are alleviated.

6-Month Agency Response

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members are working with DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

The addition of temporary staff, including an additional supervisor, has allowed the licensing supervisor to more closely monitor the age of each pending application.

Formal reports are being developed to capture this information by participating boards/bureaus and DCA to be available by July 1, 2016. Until the reports are available the BRN maintains the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to ensure applications are processed within the regulatory time frame.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

BRN staff is working with the Department of Consumer Affairs (DCA) and has initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement request was presented at the March 24, 2015 Change Control Board (CCB) for review and approval. The CCB approved the system enhancement and expects to have the enhancement in BreEZe for use by July 1, 2015.

Beginning April 1, 2015, the BRN is utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members are working with the DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

The addition of temporary staff, including an additional supervisor, has allowed the licensing supervisor to more closely monitor the age of each pending application.

Until the system enhancements are complete the BRN maintains the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to ensure applications are processed within the regulatory time frame.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #32 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe in order to identify the workload capability of each of its units, such as the licensing support unit; to the extent it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified.

Annual Follow-Up Agency Response From January 2017

The BRN has analyzed the information from the SOLID report to assess efficiency and workload capability which was included with our data collection to request additional resources in a BCP for FY 17/18 which was included in the Governor's budget that was released on January 10, 2017.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

The Department of Consumer Affairs' SOLID developed and delivered a final report that detailed and documents the licensing business processes, average processing times, and recommendations prior to June 30, 2015.

The BRN is analyzing the information from the SOLID report to assess efficiency and workload capability to include in our existing request for additional resources.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Until BRN has finalized its analysis of the information in the SOLID report and determined whether additional resources are necessary to manage its workload, we will continue to report this recommendation as partially implemented.

6-Month Agency Response

The Department of Consumer Affairs' SOLID developed and delivered a final report that detailed and documents the licensing business processes, average processing times, and recommendations prior to June 30, 2015.

The BRN is analyzing the information from the SOLID report to assess efficiency and workload capability to include in our existing request for additional resources.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The Board of Registered Nursing (BRN) is currently doing the following to meet the recommendation:

- The BRN met with the Department of Consumer Affairs (DCA) SOLID Training Solutions to outline the project on February 12, 2015. BRN subject matter experts met with SOLID and designed licensing process maps for each license type in March 2015.

- SOLID developed and delivered business process flowcharts and summary procedures for each licensing type to BRN for review. SOLID and BRN staff will meet on April 8, 2015 to discuss and finalize the flowcharts.

The BRN will do the following to meet the recommendation:

- SOLID will deliver a template that details instructions and the process to develop average time frames to process applications for each license type.

- BRN staff will conduct time trials to process all application types to develop average processing time frames.

- SOLID will develop and deliver a final report that details and documents the licensing business processes, average processing times, and any recommendations no later than June 30, 2015.

- The BRN will analyze the information from the SOLID report to assess efficiency and workload capability to include in a request for additional resources.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #33 To: Registered Nursing, Board of

To ensure that BRN continues to process applications within regulatory time frames, it should continue its efforts to refine its business processes to increase efficiency and reduce the amount of time applications are pending its review.

Annual Follow-Up Agency Response From January 2017

The BRN has implemented changes related to SOLID and staff recommendations and on an ongoing basis has taken additional steps to improve and refine our business processes. The BRN has refined the application review process for California schools eliminating the need to review an application multiple times to determine eligibility to take the exam and obtain a license. An additional business process change in October 2016 to accept transcripts electronically via the DCA Cloud system from California approved RN schools has contributed to the improved processing times. As a result, the BRN has conducted the majority of initial application evaluations within the regulatory timeframe since July 2016.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2016

The BRN has worked with the Department of Consumer Affair's (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN has implemented changes related to the recommendations and on an ongoing basis has taken additional steps to improve and refine our business processes. The BRN will continue to analyze and accept staff feedback regarding its business processes to develop and implement new and revised procedures for the initial licensure application process. The BRN continues to assess efficiency and workload capability and has identified the need for additional resources. Requests for additional staff have been submitted.

This recommendation is considered implemented, but is also considered ongoing as business processes evolve due to ongoing enhancements to the BreEZe system.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Based on information provided by BRN, it continues to face significant delays in processing license applications. As a result, until it processes applications within regulatory time frames, we will continue to report this recommendation as partially implemented.

1-Year Agency Response

The BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN is moving forward to implement changes related to the recommendations and taking additional steps to improve and refine our business processes.

The BRN continually analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

The BRN is assessing efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

The BRN will submit requests for additional staff when adequate funding is available to sustain the request.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

The BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN is moving forward to implement changes related to the recommendations and taking additional steps to improve and refine our business processes.

The BRN continually analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

The BRN is assessing efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The BRN continuously analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

Once the BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions in early July 2015 to document all of its business processes and review any recommendations, the BRN will assess efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #34 To: Consumer Affairs, Department of

Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner, with particular attention to understanding their reporting needs and improving the system's reporting capabilities.

60-Day Agency Response

The Department has a maintenance and operation (M&O) contract in place which provides dedicated resources to resolving issues for Release 1 entities. Additionally, we have established three user groups (Licensing, Enforcement, and Reports) and multiple forums for Release 1 executives and their staff to ensure the issues are clearly communicated to the Department and addressed in a timely manner.

In addition to the M&O contract, the Department established a dedicated reports development team in March 2013 to address the difficulties and deficiencies with the available BreEZe reports. This team works with the Reports User Group and is charged with developing, prioritizing, and standardizing reports.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #35 To: Consumer Affairs, Department of

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

Annual Follow-Up Agency Response From October 2021

Annual Follow-Up Agency [DCA] Response October 2021

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements was issued October 4, 2021 via an all staff Policy Directive OIS 21-01.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements will be submitted to the DCA Policy Committee in early 2021.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2019

The Department of Consumer Affairs (DCA) has adopted this recommendation as a standard for all training plans associated with reportable technological investments. In addition, beginning with the implementation of the data platform for the Bureau for Cannabis Control in 2018, DCA has adopted a more agile development and implementation process. This agile, iterative approach brings new functionality to market on a monthly or bi-monthly basis. Such a quick turn-around naturally requires training to be provided on an ongoing basis. This approach contrasts with previously used waterfall methods that may have a multi-year delay between business collaboration and implementation. The recommendation calls out specifically further training in Breeze, but DCA has extended this requirement to all reportable technological investments.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Although DCA claims that it has adopted this recommendation as a standard for all training plans associated with reportable technological investments, DCA explained that it has not yet formalized a policy to reflect this recommendation. DCA indicated that it intends on formalizing a policy in 2020, when the first group of regulatory entities are scheduled to begin their projects.

Annual Follow-Up Agency Response From October 2018

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (i.e. PAL). Because phase 3 entities will be partnering with the Department in completing the PAL, the Department will ensure planned training activities are occurring as close to the planned rollouts within each entity's approved baseline project schedule. During BreEZe's second phase, training activities were completed on 12/11/2015 and the second release rollout occurred on 1/19/2016.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2017

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (PAL). The Department will ensure planned training activities occur as close to the rollout date within each entity's approved project schedule that is baselined through the PAL process.

Implementation Timeframe: The dates may vary as each entity proceeds through the planning stages of the PAL.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

In its one-year response to this recommendation, Consumer Affairs explained and provided supporting documentation that it had successfully executed a comprehensive multi-stage training curriculum for all BreEZe regulatory entities included in phase 2, which resulted in providing BreEZe system training up to the penultimate business day before system rollout. Because of this, we assessed the status of this recommendation as partially implemented. Until Consumer Affairs provides training activities as close to the rollout date as possible for the phase 3 regulatory entities, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2016

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during the functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #36 To: Consumer Affairs, Department of

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should work with the regulatory entities to develop training that is specific to each entity's business processes.

Annual Follow-Up Agency Response From October 2021

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners, and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements was issued October 4, 2021 via an all staff Policy Directive OIS 21-01.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements will be submitted to the DCA Policy Committee in early 2021.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2019

A primary tenet of the Department of Consumer Affairs' Business Modernization Plan is to analyze the business needs of the boards/bureaus on an individual basis and has been implemented for all boards/bureaus that have begun their business activities. From those specific and individualized requirements come board-specific functionality that therefore must be trained on a board-specific basis. The Business Modernization's focus on the board-level requirements at all levels, ensures that training is delivered in the same manner. The recommendation calls out specifically further training in Breeze, but DCA has extended this requirement to all reportable technological investments.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Although DCA indicates that it has adopted this recommendation as a standard for all training associated with reportable technological investments, DCA explained that it has not yet formalized a policy to reflect this recommendation. DCA indicated that it intends on formalizing a policy in 2020, when the first group of regulatory entities are scheduled to begin their projects.

Annual Follow-Up Agency Response From October 2018

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (i.e. PAL). Because phase 3 entities will be partnering with the Department in completing the PAL, the Department will ensure planned training activities include business-process specific training elements. During BreEZe's second release, the Department supplemented the planned training activities in the Solution Vendor (i.e. Accenture) contract with a separate Organizational Change Management (OCM) contract. The OCM activities included developing numerous business process-specific artifacts that the phase 2 entities used prior to and after launch to effectively execute business activities in the BreEZe system.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2017

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (PAL). The Department will ensure planned training activities include business-process specific training elements.

Implementation Timeframe: For phase 3 regulatory entities, the dates may vary as each entity proceeds through the planning stages of the PAL.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

In its one-year response, Consumer Affairs explained and provided supporting documentation that it deployed an Organizational Change Management team to assist the phase 2 regulatory entities with their transition to BreEZe. Specifically, the Organizational Change Management team was charged with working with the phase 2 regulatory entities to develop and execute training that was specific to each entity's business processes. Because of this, we assessed this recommendation as partially implemented. However, until Consumer Affairs works with the phase 3 regulatory entities to develop training that is specific to each entity's business processes, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2016

The Department deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team was charged with working with the Release 2 boards and bureaus to develop and execute training that was specific to each entity's business processes. This training was the last stage of the multi-stage training approach detailed in response to Recommendation #35. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Department has deployed a skilled Organizational Change Management (OCM)team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team was charged with working with the Release 2 boards and bureaus to develop and execute training that was specific to each entity's business processes. This training was the last stage of the multi-stage training approach detailed in response to Recommendation #35. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2014-116

Agency responses received are posted verbatim.