Report 2014-120 Recommendation 4 Responses
Report 2014-120: California Public Utilities Commission: It Needs to Improve the Quality of Its Consumer Complaint Data and the Controls Over Its Information Systems (Release Date: April 2015)
Recommendation #4 To: Public Utilities Commission
To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should develop and implement tools by September 30, 2015, to measure the quality management team program's effectiveness.
Annual Follow-Up Agency Response From October 2021
The Quality Management Team (QMT) supervisor produces a QMT report every two weeks that identifies the most common errors when processing a written complaint case. The QMT reports are also used to facilitate discussions at team meetings to ensure consistency. The Quality Management Team (QMT) is comprised of Consumer Affairs Representatives that ensure accuracy by performing a review of the coding (e.g., categorizing and attributes) for all written cases. Additionally, a Quality Assurance Unit (QAU) was established that currently focuses on internal unit training and overseeing the call monitoring quality program. The QAU also uses the QMT reports to facilitate internal trainings as needed.
- Completion Date: September 2021
California State Auditor's Assessment of Annual Follow-Up Status: Pending
CPUC provided data addressing complaints for a two-week period that were reviewed by QMT supervisors, but it did not demonstrate how it was measuring the degree to which these reviews improved the quality of data collection over time. CPUC indicated that it has a quality assurance project underway that will produce automated reporting of the QMT's efforts and effectiveness. It expects to implement the project in the second quarter of 2022.
- Auditee did not substantiate its claim of full implementation
- Auditee did not address all aspects of the recommendation
Annual Follow-Up Agency Response From November 2020
Due to the loss of IT Resources assigned to higher priority CPUC projects, the timeline for completion has changed. Currently, development progress is at 90% complete. In order to complete the implementation of this recommendation, CPUC needs to finish the CIMS-QA Reports, Testing (unit testing, test scripts, and User Acceptance Testing (UAT)), Training, and User Guide.
- Estimated Completion Date: 5/19/2021
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2019
Updated 10/14/19 - Partially Implemented.
Due to the loss of IT Resources being assigned to higher priority CPUC projects, the timeline for completion has slipped. New IT Resources are being assigned to the project. A learning curve will be there for these new IT Resources. Development progress is at 57% currently. Remaining is finishing up the programming for the CIMS-QA code, Reports, Testing, UAT.
- Estimated Completion Date: 8/31/2020
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2018
Although the project requirements and design phases were completed in 2017, the lack of IT resources in 2018 resulted in significant delays in automating and improving CIMS quality management processes. Specifically, the CPUC's IT Applications Programming staff only made limited progress in application development between January 2018 and July 2018 as staff resources were reassigned to higher priority CPUC projects. In addition, IT Project Management staff was replaced with new staff in June 2018. As a result of the delays, the contract for vendor resources also expired and the Branch is in the process of securing new vendor resources to complete technical manuals once project development and testing are complete. In August 2018, IT began shifting some IT Applications Program resouces back to the Branch's project and correspondingly estimates that the project will be complete and the automation be in place in July 2019. As of September 30, 2018, the project is on track to meet July 2019 completion.
- Estimated Completion Date: 07/2019
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From November 2017
In 2017, the Branch has secured resources to automate and improve parts of the quality management processes within the CIMS database. Resources include the CPUC's IT Applications Programming and Project Management units as well as a vendor specializing in business analysis. The project was chartered on 11/09/2016 by the CPUC as the "Consumer Information Management System - Audit Response Mitigation for Quality Assurance". The project was approved by the California Department of Technology in a Stage 1 Business Analysis on 01/23/2017. (Public Utilities Commission (8660): 8660-082 CIMS Audit Response Mitigation for Quality Assurance) The project requirements and design phases were approved on 03/17/2017 and 07/05/2017, respectively. The applications development was initiated on 08/14/2017. It is estimated that the project will complete and the automation be in place in mid-2018.
- Estimated Completion Date: 6/30/2018
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2016
The Branch has updated its plan to measure and improve the effectiveness of its quality management team (see attached "CAB Quality Management Team Enhancement Plan"). As described in the plan, the Branch has completed the quality management process analysis described as Phase I. In Phase II, the Branch has begun to analyze baseline measures of its processes. That analysis has been completed for 2013-14 and 2014-15; it is anticipated that the analysis for 2015-16 will be completed in late 2016. Moreover, the Branch is actively pursuing a project to automate and improve parts of the quality management processes (Phases III and IV). As noted in the prior audit status responses, these process improvements will require resources from outside of the Branch, including support from the CPUC's IT unit and the database vendor, as well as additional staffing resources for CAB to ensure optimal quality management. To that end, the Branch's project request to build a database module to automate quality management processes was updated and approved in August 2016. The Branch began work in September 2016 securing funding to use a vendor to create a business analysis for the project. As of September 28, 2016, the Branch received budget approval to move forward with the project. It is anticipated that the project will begin in November 2016, with an estimated duration of six to nine months.
- Estimated Completion Date: 12/30/2016
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
The Branch has created a draft plan to measure and improve the effectiveness of its quality management team. As part of the draft plan, the Branch has begun to analyze baseline measures of its process. The Branch continues to research the feasibility of automating parts of the quality management processes and continues to research ability of its database to create multiple alerts to enable case progress to be better measured. Preliminary findings are the process improvements will require resources from outside of the Branch, including support from CPUC IT and the database vendor.
- Estimated Completion Date: 4/9/2017
- Response Date: April 2016
California State Auditor's Assessment of 1-Year Status: Pending
6-Month Agency Response
Branch has analyzed current QMT processes and is researching the feasibility of automating parts of the processes. Preliminary findings are that process improvements will require resources from outside of the branch, including support from CPUC IT and the CIMS database vendor. Current estimates are that IT resources will not be available until early to mid-2016.
- Estimated Completion Date: Late 2016.
- Response Date: October 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Branch is working with IT to expand its measurement capabilities in CIMS to assist in quality management team efforts.
- Estimated Completion Date: September 30, 2015
- Response Date: June 2015
California State Auditor's Assessment of 60-Day Status: Pending
All Recommendations in 2014-120
Agency responses received are posted verbatim.