Appendix A

Scope and Methodology

The Joint Legislative Audit Committee (Audit Committee) directed the California State Auditor to develop and verify information related to the implementation of the LPS Act by Los Angeles and two additional counties. We selected San Francisco and Shasta as the additional counties for review. Table A below lists the objectives that the Audit Committee approved and the methods we used to address them.

Table A
Audit Objectives and the Methods Used to Address Them
AUDIT OBJECTIVE		METHOD
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed and evaluated relevant federal and state laws, rules, regulations, and best practices related to the LPS Act, including laws related to the broader mental health systems within which counties implement involuntary holds.
2	Review the statewide oversight of the implementation of the LPS Act.	Documented and assessed the roles and responsibilities of Health Care Services, State Hospitals, the Judicial Council, Justice, and the Oversight Commission by evaluating their oversight responsibilities and relevant data. Documented Health Care Services’ rulemaking history related to the LPS Act and other relevant mental health issues for the last five fiscal years. Documented procedures related to Health Care Services’ approval and inspection of county treatment facilities and inspection and certification of facilities that provide Medi‑Cal specialty mental health services. Evaluated State Hospitals’ LPS Act waitlist projections and its usage planning for state facility capacity. Determined that the Judicial Council provides training and educational resources for judges who hear LPS Act conservatorship cases. Evaluated Justice’s process for monitoring and maintaining data relevant to the LPS Act by assessing how it tracks treatment facility data. Interviewed agency staff about their roles, responsibilities, and perspectives.
3	By county and for each of the most recent three years, determine the following: a. The number of individuals placed under initial involuntary holds, the referral sources for those holds, and the number of individuals placed under repeated initial holds. b. The number of individuals placed under subsequent holds. c. The number of individuals placed into new and renewed LPS conservatorships and the referral source for those conservatorships. d. The average length of LPS conservatorships. e. The number of terminated LPS conservatorships and the reasons for the termination.	Obtained Justice’s mental health holds data to determine, for fiscal years 2014–15 through 2018–19, the number of individuals placed on 72‑hour holds, repeated 72‑hour holds, 14‑day holds, and conservatorships. Because of statewide data limitations, conducted a case file review to determine the referral sources for a selection of 30 72‑hour holds. Analyzed Justice’s data for all fiscal years available to identify the number of individuals placed on holds during fiscal years 2014–15 through 2018–19 who had been placed on repeated holds during their lifetime. Justice’s data are limited to individuals placed on involuntary holds because they were determined to be dangerous to themselves or others. Because of statewide data quality issues, completed manual and automated data identification of duplicate records for the three counties in the audit to ensure, to the extent possible, accuracy in calculating the number of repeated holds per individual. Because of statewide data limitations, conducted a case file review to determine the referral sources, average length of conservatorships, and reasons for termination for a selection of 60 conservatorship cases.
4	Assess the counties’ implementations of the LPS Act for the last three years and compare the counties to one another by reviewing at least the following: a. The counties’ definitions of the criteria for involuntary treatment holds and whether each county has consistently applied its definitions. b. The counties’ criteria for placing individuals into LPS conservatorships and making least‑restrictive‑environment determinations and whether the counties have consistently followed these criteria.	Evaluated each county’s process for placing individuals on short‑term holds, including how it interpreted criteria and whether it applied those criteria consistently, and compared the three counties’ processes. Used county, law enforcement, and Justice data to select 10 72‑hour hold cases, 10 14‑day hold cases, and 20 cases in which 72‑hour holds might have been placed but were not in each county from fiscal years 2016–17 through 2018–19. Evaluated the selected cases to assess why the holds were placed, the start and stop times of the holds, the histories of the individuals, and the connection to subsequent treatment. Evaluated each county’s process for placing individuals on conservatorships, including how it determined who should be placed on conservatorship and how it ensured the confidentiality of sensitive information. Used county data to select 20 conservatorship cases from each county for case file review. We selected conservatorships that ended during fiscal years 2016–17 through 2018–19, including up to five cases per county involving individuals who had been found incompetent to stand trial. We verified that the county records we used to make our selection of case files were sufficiently complete for our purposes. Evaluated superior court processes by reviewing judicial orders and transcripts granting and terminating conservatorship for elements such as the evidence used to support the final decisions and whether individuals placed on conservatorships retained their right to consent to or refuse medications. Documented the placements of individuals on conservatorships and evaluated the extent to which counties provided care in the least restrictive setting appropriate to individuals’ needs. To do so, we reviewed counties’ policies and processes for placing individuals in appropriate levels of care and whether individuals moved from more restrictive to less restrictive levels of care during conservatorships.
5	Assess whether any differences between county approaches to involuntary holds, conservatorships, or the associated care provided to individuals should be addressed through changes to state law or regulation.	Documented county policies and assessed applications of policies through a review of case files to determine whether counties implemented involuntary holds and conservatorships consistently. Reviewed laws in other states to identify possible best practices or potentially beneficial changes to state law or regulation. Interviewed nonprofit stakeholders to identify concerns regarding treatment and rights protections for individuals placed on involuntary holds and conservatorships.
6	Determine how the counties fund their implementations of the LPS Act and whether access to funding is a barrier to the implementation of the LPS Act.	Evaluated the Medi‑Cal process and reimbursements for psychiatric patients to identify possible barriers to treatment and mental health coverage for Medi‑Cal beneficiaries. Documented or determined each county’s unspent MHSA fund balance and evaluated the county’s stated reasons for maintaining that balance. Documented and assessed existing statewide reporting requirements for mental health funding and outcomes.
7	Assess the availability of treatment resources in each county and, to the extent possible, determine whether there are barriers to achieving the intent of the LPS Act. In doing so, at the minimum, consider the number of LPS facilities in each county and the availability of rehabilitative programs during and after conservatorships.	Determined the range of services each county has available to individuals treated through the LPS Act. Assessed the extent to which counties have connected individuals who have been on LPS Act holds to assisted outpatient treatment and full‑service treatment programs. Relied on records kept by counties that indicated the enrollment in these programs. Reviewed the completeness and accuracy of these records by comparing to enrollment data held by Health Care Services. Although we identified some errors in this review that could affect the precision of the numbers we present in this report, there is sufficient evidence overall to support the findings and conclusions we present in this report. Obtained assisted outpatient treatment and full‑service treatment enrollment data from the counties to calculate various three‑day hold statistics for individuals enrolled in assisted outpatient treatment and full‑service partnership programs. Evaluated barriers to implementation of assisted outpatient treatment in each county. Documented facilities designated for evaluation and treatment under the LPS Act in each county and in a selection of three additional counties. Evaluated Los Angeles’s treatment facility referral and waitlist tracking logs to identify barriers to placement of individuals receiving services through the LPS Act. Evaluated existing reports regarding oversight, cost, and outcomes for assisted outpatient treatment and full‑ service partnership treatment approaches. Documented and assessed State Hospitals’ admissions and discharge practices, admissions waitlist, facility inventory, and recent and possible future need for capacity expansion.
8	Review and assess any other issues that are significant to the audit.	Documented contextual information and background statistics for issues related to mental illness, including homelessness, incarceration, and substance abuse.

Source: Analysis of Audit Committee’s audit request number 2019‑119, state law, and information and documentation identified in the column titled Method.

Assessment of Data Reliability

The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer‑processed information that we use to support our findings, conclusions, and recommendations. In performing this audit, we relied on Justice’s mental health hold data to calculate various statistics, including the number of repeat holds, in Los Angeles, San Francisco, and Shasta. To evaluate these data, we reviewed existing information about the data, interviewed agency officials knowledgeable about the data, and performed electronic testing of the data. We determined that Justice’s data does not consistently track a unique person identifier that can be used to identify multiple holds for a single individual. Further, we found that medical providers and courts had sometimes submitted mental health hold data to Justice using different variations of individuals’ names. To help account for these issues, we removed duplicate hold records and performed manual and automated deduplication work to group holds by person. However, we were unable to uniquely identify individuals related to 5 percent of the holds during our audit period. As a result, we found these data are of undetermined reliability for our purposes. Although these issues may affect the precision of the numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations.

We also obtained State Hospitals’ pre‑admission data to determine the number of people on its waitlist and how long they had been waiting. To evaluate these data, we interviewed agency officials knowledgeable about the data and performed electronic testing of the data. However, we did not perform accuracy and completeness testing of the data because source documents are located at various locations throughout the State, making such testing cost‑prohibitive. As a result, these data are of undetermined reliability. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations.

Appendix B

Detailed Proposal for Reporting Framework

Counties provide a range of programs and services to individuals with mental illnesses. However, as we discuss in Chapter 3, no reporting framework currently exists that makes it easy for stakeholders to understand the types of services counties provide, how they fund those services, and the impacts of those services on people’s lives. In the course of our review of three counties’ mental health systems, we created an example of a framework that would address that issue, which we present in detail in Table B. We based our framework on the services that counties provide, with the goal of categorizing those services simply but in a way that allows for useful comparisons between the various categories. We believe this kind of framework could help the State collect and report information from counties that would allow stakeholders—including the Legislature—to better evaluate mental health spending and outcomes statewide.

Table B
Example Reporting Framework for County Mental Health Programs and Services
COMPONENTS	EXAMPLES OF PROGRAMS AND SERVICES	POSSIBLE PROGRAM AND SERVICE OUTCOMES
*Emergency Services* Short‑term emergency or crisis services.	Urgent care and emergency room services. Mobile crisis response teams. Law enforcement and mental health teams.	Response time of first responders. Emergency room wait time and length of stay. Frequency and timeliness of linkage to subsequent services.
*Inpatient Care* Extended treatment/care in facility settings.	Services in state hospital facilities. Services in general acute hospitals or acute psychiatric hospitals. Services in residential care facilities.	Availability of beds/timeliness of placement by facility type. Medication compliance. Frequency and timeliness of linkage to subsequent services.
*Intensive Outpatient Services* Community‑based programs with individualized support and case management that coordinate care for clients with serious mental illnesses.	Full service partnership programs that include personal case managers. Assisted outpatient treatment programs that include personal case managers and low staff‑to‑client ratios. Intervention treatment programs for schizophrenia and early psychosis.	Population served and population with unmet needs. Medication compliance. Clients’ health status and other quality‑of‑life measures. Clients’ incidences of hospitalization, incarceration, and other negative outcomes.
*Basic Social Supports* Community‑based programs and services primarily focused on meeting basic needs, such as food, clothing, and shelter.	Supportive housing and shelter programs. Drop‑in centers providing temporary safety, food, clothing, and hygiene facilities, as well as linkages to mental health services and supports.	Population served and population with unmet needs. Average length of stay (for housing and shelter). Frequency and timeliness of linkage to concurrent or subsequent services.
*General Outpatient Services* Medical services and supports, such as evaluation and medication, provided on an outpatient and as‑needed basis.	Drop‑in and appointment‑based programs that provide evaluations, pharmaceuticals, and placements into treatment programs. Health centers that provide a variety of services, such as assessments, medication management, and psychotherapy sessions.	Population served and population with unmet needs. Frequency and timeliness of linkage to concurrent or subsequent services. Medication compliance (when applicable). Clients’ health status and other quality‑of‑life measures (when applicable). Clients’ incidences of hospitalization, incarceration, and other negative outcomes (when applicable).
*Community Wellness Supports* Social programs and supports available in the community to improve individuals’ wellness.	Wellness centers. Peer support and resource centers Programs that offer social support groups and other wellness‑based activities, as well as referrals to services and supports such as food, clothing, and medical attention.	Population served and population with unmet needs. Frequency and timeliness of linkage to concurrent or subsequent services. Client‑reported wellness and satisfaction with programs and supports.
*Outreach and Education* Outreach, education, and training to provide information about available services; educate staff and community members; and encourage well‑being.	Stigma reduction programs. Implicit bias forums/trainings. Parenting programs aimed at enhancing parents’ knowledge, skills, and confidence as a preventive measure for their children. Suicide prevention campaigns to inform the community about related resources.	Population served or affected by outreach and education efforts. Impact of efforts on individuals’ engagement with treatment. Community awareness of and attitudes toward available services.

Source: Analysis of county documents such as MHSA reports and continuums of care, state law, other documents about the range of mental health services available, and discussions with county and state staff.

Appendix C

Additional Data About Involuntary Holds and Conservatorships

The Audit Committee asked us to provide a variety of summary information related to involuntary holds and conservatorships in the counties we reviewed. The following tables summarize additional or more detailed results of our review of data related to the involuntary holds and conservatorships we discuss throughout the report. Because statewide data on conservatorships are limited, we provide information about conservatorship referrals, durations, and terminations based on our review of 60 case files in the three counties that we reviewed during this audit.

Table C.1
The Number of Involuntary Holds per Fiscal Year Has Generally Increased
TYPE OF INVOLUNTARY HOLD	2014–15	2015–16	2016–17	2017–18	2018–19
*Los Angeles*
72‑hour‑hold	71,018	72,508	73,830	80,047	81,505
14‑day‑hold	15,828	14,156	15,038	15,497	15,820
Conservatorship	4,389	4,919	4,660	4,623	4,698
*San Francisco*
72‑hour‑hold	4,524	4,086	3,718	4,033	3,837
14‑day‑hold	448	580	592	798	897
Conservatorship	531	531	525	537	601
*Shasta*
72‑hour‑hold	631	581	504	403	670
14‑day‑hold	148	220	235	246	310
Conservatorship	60	81	86	69	94

Source: Analysis of Justice’s mental health holds data.

Table C.2
Many Individuals Were Placed on Multiple Involuntary Holds
	INDIVIDUALS WITH AT LEAST ONE HOLD OF THIS TYPE	INDIVIDUALS WITH ONLY ONE HOLD OF THIS TYPE	INDIVIDUALS WITH MORE THAN ONE HOLD OF THIS TYPE	AVERAGE NUMBER OF HOLDS FOR INDIVIDUALS WITH MULTIPLE HOLDS OF THIS TYPE
72‑Hour Hold
Los Angeles	166,447	94,425 (57%)	72,022 (43%)	6.2
San Francisco	14,010	9,647 (69%)	4,363 (31%)	4.3
Shasta	2,206	1,701 (77%)	505 (23%)	2.8
14‑Day Hold
Los Angeles	57,130	33,574 (59%)	23,556 (41%)	4.1
San Francisco	3,428	2,401 (70%)	1,027 (30%)	2.9
Shasta	962	763 (79%)	199 (21%)	2.5

Source: Analysis of Justice’s mental health holds data.

Note: This analysis includes the lifetime total number of holds for individuals with a hold or conservatorship from fiscal years 2014–15 through 2018–19. However, we excluded 5 percent of the holds in our audit period from this analysis because we could not associate each of these holds with a unique individual for reasons such as a blank date of birth or a likely fictitious name.

Table C.3
Most Individuals Placed on Conservatorship Were Subject to Multiple Conservatorship Orders
	INDIVIDUALS WITH AT LEAST ONE CONSERVATORSHIP ORDER*	INDIVIDUALS WITH ONLY ONE CONSERVATORSHIP ORDER	INDIVIDUALS WITH MORE THAN ONE CONSERVATORSHIP ORDER	AVERAGE NUMBER OF CONSERVATORSHIP ORDERS FOR INDIVIDUALS WITH MULTIPLE CONSERVATORSHIP ORDERS
*Los Angeles*	7,242	1,324 (18%)	5,918 (82%)	5.5
*San Francisco*	813	160 (20%)	653 (80%)	4.4
*Shasta*	152	40 (26%)	112 (74%)	3.7

Source: Analysis of Justice’s mental health holds data.

Note: This analysis includes the lifetime total number of conservatorship orders for individuals with a hold or conservatorship from fiscal years 2014–15 through 2018–19. Conservatorship orders include orders renewing a conservatorship after one year and orders establishing new conservatorships.

* A small percentage of these individuals did not experience a conservatorship from fiscal years 2014–15 through 2018–19, but experienced at least one conservatorship in their lifetime. Nevertheless, these individuals continued to interact with the mental health system by being placed on involuntary holds during our audit period.

Table C.4
Designated Professionals at Treatment Facilities and Correctional Facilities Made the Referrals for the Conservatorships We Tested
	REFERRALS FROM TREATMENT FACILITIES	REFERRALS FROM CORRECTIONAL FACILITIES
*Los Angeles*	14	6
*San Francisco*	16	4
*Shasta*	18	2

Source: Analysis of 60 conservatorship case files.

Note: State law allows designated professionals at treatment facilities and county jails to recommend conservatorships for gravely disabled individuals. We deliberately included some cases involving the criminal justice system in our review of 20 case files from each county. Therefore, the information presented here is not necessarily indicative of the sources of conservatorship referrals generally.

Table C.5
The Conservatorships We Tested Lasted About Three Years on Average
	AVERAGE LENGTH OF CONSERVATORSHIP
*Los Angeles*	2 years and 8 months
*San Francisco*	3 years and 6 months
*Shasta*	3 years and 3 months

Source: Analysis of 60 conservatorship case files.

Table C.6
Except in Los Angeles, Most Conservatorships We Tested Ended When Individuals Were Able to Provide for Their Basic Needs
	COUNTY OR COURT DETERMINED THE INDIVIDUAL WAS NO LONGER GRAVELY DISABLED	INDIVIDUAL LEFT TREATMENT FACILITY WITHOUT AUTHORIZATION	CONSERVATORSHIP TERMINATED BECAUSE COURT COULD NOT PROCEED*
*Los Angeles*	8	5	7
*San Francisco*	14	5	1
*Shasta*	18	2	0

Source: Analysis of 60 conservatorship case files.

* The absence of a doctor’s testimony, which we discuss in Chapter 1, was the most frequent reason why courts could not proceed. In two other cases, the courts could not proceed because of individuals’ specific circumstances rather than because of a systemic problem.