Report 2012-122 Recommendation 5 Responses

Report 2012-122: Mental Health Services Act: The State's Oversight Has Provided Little Assurance of the Act's Effectiveness, and Some Counties Can Improve Measurement of Their Program Performance (Release Date: August 2013)

Recommendation #5 To: Health Care Services, Department of

To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should collaborate with the Accountability Commission to develop and issue guidance or regulations, as appropriate, to counties on how to effectively evaluate and report on the performance of their MHSA programs.

Annual Follow-Up Agency Response From October 2019

DHCS entered into a vendor contract to prepare the guidance recommended by the CSA. The contractor provided DHCS with recommendations on uniform procedures, metrics, and templates that counties could use for assessing and reporting the performance of MHSA programs. DHCS expects to release guidance to counties by December 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From August 2019

DHCS entered into a vendor contract to prepare the guidance recommended by the CSA. The contractor provided DHCS with recommendations on uniform procedures, metrics, and templates that counties could use for assessing and reporting the performance of MHSA programs. DHCS expects to release guidance to counties by December 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Health Care Services has received information from a vendor that could form the basis of guidance to counties for assessing and reporting the performance of MHSA programs; it expects to release that guidance by the end of 2019.

Annual Follow-Up Agency Response From November 2018

DHCS entered into a vendor contract to improve data quality and completeness in the CSI and the MHSA DCR. The vendor created reports of findings (e.g., data entry errors, anomalies, missing data) and delineated the need for improved infrastructure and maintenance of data quality. The vendor has also provided customized technical assistance (e.g., weekly webinars, monthly site visits) for counties and Electronic Health Record (EHR) vendors to improve data quality. For sustainability of improvement efforts, DHCS extended the vendor's contract to FY 2018-19 and is negotiating a contract for FY 2019-20. DHCS plans to transition vendor-provided technical assistance and system training to DHCS staff within the contract timeframe for continued county support. During FY 2019-20, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs. Full implementation is expected by June 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2017

DHCS has contracted with a vendor to perform data clean up on the data included in the DHCS Client Services Information System and MHSA Data Collection and Reporting System. The purpose of this project is to provide DHCS, the MHSOAC, and the counties with accurate and complete data that may be used to track performance outcomes for Community Services and Supports, Full Service Partnership clients. The project will be completed over the next 12-18 months, with DHCS maintaining data quality thereafter. Counties have already received their comprehensive data error reports, and technical assistance calls to support the interpretation and use of these reports are currently underway. After the data clean up project is complete, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

As a first step toward evaluating and reporting on MHSA-funded programs, DHCS determined that it is necessary to work with counties to improve the quality of the Full Service Partnership data reported to the DHCS MHSA Data Collection and Reporting System. DHCS sought, and was provided with, a re-appropriation of MHSA funds to strengthen data quality, which will be the focus of our efforts in FY 2016-17. Thereafter, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

DHCS is planning to start discussions with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

DHCS continues to support the Mental Health Services Oversight and Accountability Commission's (MHSOAC) Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, data monitoring and reporting for compliance, and ongoing data system planning and coordination meetings.

As part of the review and development of MHSA regulations, DHCS continues to consult with the Accountability Commission to determine any regulatory changes needed to ensure data are collected consistently and in support of the Evaluation Master Plan.

If needed, draft regulations will be developed in consultation with the Accountability Commission and made available for public comment in the winter of 2015, with final regulations adopted during the fall of 2016.

DHCS also continues the development of the Performance Outcomes System Plan for Medi-Cal Specialty Mental Health Services for Children and Youth (POS). In September of 2013, DHCS staff began reviewing the status of county data submission to the Client and Services Information (CSI) System. As of June 30, 2014, 79% of counties had submitted CSI data within the 60-day submission window. Data review and cleanup, as well as timely submission of data, is critical to the quality of data for the POS and other mental health services evaluation efforts. Beginning in October 2014, DHCS will provide quality improvement reports to the counties in order to prepare for the initial outcomes reports for specialty mental health, which are scheduled to be released in December 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

DHCS continues to support the Accountability Commissions Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, and data monitoring and reporting compliance.

DHCS continues its ongoing effort to identify outcomes/measures for children and youth as part of the Performance Outcomes System Plan for Medi-Cal Specialty Mental Health Services for Children and Youth (POS). The POS system plan was submitted to the Legislature on November 1, 2013 and the system implementation plan was submitted on January 1, 2014. Since some of the evaluation efforts in the POS and the Evaluation Master Plan are similar, DHCS and the Accountability Commission continue to collaborate to develop and issue guidance to counties on how to effectively evaluate and report on program performance.

DHCS continues to work with counties on their submission of mental health data. As part of the MHSA training and technical assistance contract, the contractor will work with counties to develop evaluation reports on several evidence-based childrens mental health services. The contractor will provide counties with technical assistance during the evaluation, and afterwards as part of quality improvement. During contract development, DHCS will continue to review deliverables related to county data reporting and performance outcomes evaluation with the Accountability Commission.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS continues to support and further the activities of the Accountability Commission's Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, and data monitoring and reporting compliance efforts.

DHCS and the Accountability Commission continue to collaborate on various data workgroup efforts to help in creating uniform, standard data reporting and outcome evaluations.

DHCS continues to work with counties on their submission of mental health data. As part of the MHSA training and technical assistance contract, DHCS will work with the contractor and the Accountability Commission to develop training for counties on data reporting and performance outcomes evaluation in support of local and statewide performance outcome objectives and standards.

DHCS will meet with the Accountability Commission beginning in November 2013 to plan for the county guidance and training needed to effectively evaluate and report on the performance of MHSA funded programs.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2012-122

Agency responses received are posted verbatim.