Report 2014-116 Recommendation 35 Responses

Report 2014-116: California Department of Consumer Affairs' BreEZe System: Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost (Release Date: February 2015)

Recommendation #35 To: Consumer Affairs, Department of

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

Annual Follow-Up Agency Response From October 2021

Annual Follow-Up Agency [DCA] Response October 2021

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements was issued October 4, 2021 via an all staff Policy Directive OIS 21-01.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

This recommendation has been adopted as standard for all non-delegated IT initiatives at the Department of Consumer Affairs (DCA). This adoption is illustrated by the first group to deploy functionality under the Business Modernization umbrella. The Business Modernization Cohort 1, consisting of the California Acupuncture Board, the Bureau for Private Postsecondary, the Board of Chiropractic Examiners and the Board for Professional Engineers, Land Surveyors and Geologists, deployed its first release on September 16, 2020. The effort required that training sessions were tailored to the specific program where applicable and the sessions were held within 60 days of implementation. The training sessions were held 3 weeks prior to deployment. A departmental policy outlining these requirements will be submitted to the DCA Policy Committee in early 2021.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2019

The Department of Consumer Affairs (DCA) has adopted this recommendation as a standard for all training plans associated with reportable technological investments. In addition, beginning with the implementation of the data platform for the Bureau for Cannabis Control in 2018, DCA has adopted a more agile development and implementation process. This agile, iterative approach brings new functionality to market on a monthly or bi-monthly basis. Such a quick turn-around naturally requires training to be provided on an ongoing basis. This approach contrasts with previously used waterfall methods that may have a multi-year delay between business collaboration and implementation. The recommendation calls out specifically further training in Breeze, but DCA has extended this requirement to all reportable technological investments.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Although DCA claims that it has adopted this recommendation as a standard for all training plans associated with reportable technological investments, DCA explained that it has not yet formalized a policy to reflect this recommendation. DCA indicated that it intends on formalizing a policy in 2020, when the first group of regulatory entities are scheduled to begin their projects.

Annual Follow-Up Agency Response From October 2018

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (i.e. PAL). Because phase 3 entities will be partnering with the Department in completing the PAL, the Department will ensure planned training activities are occurring as close to the planned rollouts within each entity's approved baseline project schedule. During BreEZe's second phase, training activities were completed on 12/11/2015 and the second release rollout occurred on 1/19/2016.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From November 2017

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (PAL). The Department will ensure planned training activities occur as close to the rollout date within each entity's approved project schedule that is baselined through the PAL process.

Implementation Timeframe: The dates may vary as each entity proceeds through the planning stages of the PAL.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

In its one-year response to this recommendation, Consumer Affairs explained and provided supporting documentation that it had successfully executed a comprehensive multi-stage training curriculum for all BreEZe regulatory entities included in phase 2, which resulted in providing BreEZe system training up to the penultimate business day before system rollout. Because of this, we assessed the status of this recommendation as partially implemented. Until Consumer Affairs provides training activities as close to the rollout date as possible for the phase 3 regulatory entities, we will continue to report this recommendation as partially implemented.

Annual Follow-Up Agency Response From October 2016

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during the functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2014-116

Agency responses received are posted verbatim.