Report 2014-116 Recommendations

To make certain that Consumer Affairs has a project team that consists of staff trained in managing IT projects, it should ensure that all the BreEZe project team leads obtain all required project management training.

Consumer Affairs should develop a process to ensure that it prepares all required project management documents and undertakes all oversight activities related to BreEZe as CalTech requires so that it can prevent or identify and monitor future problems as they arise. This includes taking steps to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise.

To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost-benefit analysis to ensure that BreEZe is a cost-effective solution to meet these regulatory entities' business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost-benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change.

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, system requirements that are specific to each regulatory entity as applicable.

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, a project team that is qualified and experienced.

If Consumer Affairs determines that a new IT system is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, development of and adherence to all project management plans.

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, planning and implementation of effective organizational change management.

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, timely response to concerns that the IV&V consultant and the IPO specialist raise.

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, sufficient staffing.

To ensure that it complies with all terms of the BreEZe contract, Consumer Affairs should develop and follow a process for doing so, including documenting how it has met applicable contract terms, such as forwarding all change requests to CalTech.

Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner, with particular attention to understanding their reporting needs and improving the system's reporting capabilities.

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should work with the regulatory entities to develop training that is specific to each entity's business processes.

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include Consumer Affairs' plan for implementing BreEZe at those regulatory entities included in the project's third phase, including a timeline for the implementation.

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include the total estimated costs through implementation of the system at the remaining 19 regulatory entities and the results of any cost-benefit analysis it conducted for phase 3.

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include a description of whether and to what extent the system will achieve any operational efficiencies resulting from implementation by the regulatory entities.

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor the timeliness of its processing of applications by type and the cause of any delays.

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor its pending workload of applications by type and original receipt date.

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe in order to identify the workload capability of each of its units, such as the licensing support unit; to the extent it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified.

To ensure that BRN continues to process applications within regulatory time frames, it should continue its efforts to refine its business processes to increase efficiency and reduce the amount of time applications are pending its review.

To help ensure the success of the BreEZe project going forward, CalTech should ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns raised by its IPO specialist and its IV&V consultant.

To help ensure the success of the BreEZe project going forward, CalTech should require Consumer Affairs to develop and follow all project management plans and ensure that the BreEZe project team leads receive all required training.

To help ensure the success of the BreEZe project going forward, CalTech should document key discussions with Consumer Affairs in which significant concerns are raised about Accenture and assessments of the department's performance by the IV&V consultant and the IPO specialist.

If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project.

Continue implementing the STAR project for state departments and ensure that its framework requires departments undertake detailed business process analyses before submitting FSRs to CalTech for approval.

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to justify the type of IT solution they are proposing, such as a COTS or a fully customized system.

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to inform the resulting RFPs and that they ensure that the RFPs contain adequate requirements.

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to follow its IT policies, including developing all necessary plans and receiving all required training.

To ensure that IT projects have the oversight needed to better position them for success, CalTech should ensure that departments obtain IPO and IV&V services as soon as an IT project is approved. Additionally, CalTech should document key discussions with any department in which the IV&V consultant and the IPO specialist raise significant concerns about a project, and ensure that the respective department responds promptly to, and adequately addresses, the concerns that the IPO specialist and the IV&V consultant raise.

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to appropriately address deficiencies identified in CalTech's approval of any SPRs as conditions that must be met, and to do so according to specified timelines. If the deficiencies are not adequately addressed within the specified timelines, CalTech should take action to suspend the project until the department has either resolved the identified deficiencies or adequately documented its justification for not addressing the deficiencies.

To ensure that IT projects have the oversight needed to better position them for success, CalTech should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should document its reasons for approving any deviations from standard contract language.

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language gives departments the rights to the source code necessary to complete the project if a contract is terminated for convenience.

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language contains assurances that state funds will not be used for unauthorized purposes.

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contracting departments can appropriately manage IT maintenance support contracts by including service-level agreements in these contracts.

To ensure that contracting departments comply with requirements, CalTech should periodically monitor the status of contracts and enforce adherence to procurement requirements. For example, CalTech should ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract.