Use the links below to skip to the specific section you wish to view:

Appendix A

Scope and Methodology

The Joint Legislative Audit Committee (Audit Committee) directed the California State Auditor to perform an audit of selected LEAs’ and charter schools’ youth suicide prevention efforts as well as several other related objectives. Table A lists the audit objectives and the methods we used to address them.

Table A
Audit Objectives and the Methods Used to Address Them
Audit Objective		Method
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, regulations, policies, procedures, and other related background material.
2	Provide and analyze statistical information related to suicide and self‑inflicted injuries of youth ages 12 to 19 in California during the past 10 years. To the extent possible, summarize this information by county.	Analyzed Public Health and Office of Statewide Health Planning and Development data to calculate the suicide and self‑harm rates by county from 2009 through 2018 for youth ages 12 to 19. We calculated the incidents per 100,000 youth ages 10 to 19, per year using population data from the U.S. Census Bureau. Analyzed Public Health and Office of Statewide Health Planning and Development data and calculated the number of individuals ages 12 to 19 that hospitals admitted for instances of self‑harm from 2009 through 2018. Self‑harm incidents we identified before October 2015 may not be comparable with those identified in or after October 2015. Beginning in October 2015, health care organizations shifted from reporting medical information under the International Classification of Diseases (ICD) Ninth Revision, to the ICD Tenth Revision which allows for more specificity. For this reason, Public Health advises against comparing incidents reported before October 2015 with incidents reported after that date. However, Public Health also acknowledged that an instance classified as self‑harm in the ICD Ninth Revision would also generally be classified as self‑harm under the Tenth Revision. In our presentation of the total number of self‑harm instances, we chose not to distinguish between the Ninth and Tenth Revisions.
3	Identify and analyze the roles of state‑level entities, including Public Health, in overall suicide prevention and as it relates to LEAs and charter schools.	Interviewed staff and reviewed relevant materials from Education, Health Care Services, the Oversight Commission, and Public Health to determine their roles in the oversight of youth suicide prevention. Reviewed Education’s model suicide prevention policy and determined that it is in compliance with state law.
4	Interview relevant stakeholders and subject matter experts to identify best practices to prevent suicides—including for students in categories with high suicide rates—that may be appropriate for LEAs or charter schools to implement.	Interviewed staff from the Trevor Project, the American Foundation for Suicide Prevention, and the California Coalition for Youth and reviewed related resources regarding best practices for suicide prevention programs. Reviewed the efforts of a selection of other states that have enacted laws and programs related to youth suicide prevention in schools. Interviewed staff at the Assembly Select Committee on Youth Mental Health to obtain its perspective on youth suicide prevention. Interviewed staff at a selection of local mental health agencies that partner with LEAs to provide mental health services. Examined academic and nonprofit research to determine what factors reduce the incidence of youth suicide. Reviewed selected provisions of Education’s model policy to verify that they reflect best practices.
5	For a selection of five LEAs and charter schools, perform the following related to their suicide prevention efforts:	Based on suicide and self‑harm rates, we selected Kern, Mendocino, and San Francisco counties for review. We then selected the largest LEA within each of these counties: Kern High School District in Kern County, Ukiah Unified in Mendocino County, and San Francisco Unified in San Francisco County. Additionally, we selected one charter school from each county: Heartland Charter in Kern County, Redwoods Charter in Mendocino County, and Gateway Charter in San Francisco County.
	a. Identify the extent to which each LEA and charter school tracks student suicides and attempted suicides.	Interviewed staff at each LEA to determine whether it tracked suicide or self‑harm data. None of the school districts or charter schools we selected track aggregate suicide or self‑harm data. To the extent available, reviewed LEAs’ counseling records to determine the percentage of the student population that received mental health services over the last three academic years. However, the LEAs did not consistently track or record data. As a result, it was not possible to calculate comparable rates of mental health service use.
	b. Determine whether the LEA and charter school have a pupil suicide prevention policy and whether that policy complies with relevant criteria.	Analyzed selected LEAs’ suicide prevention policies and procedures to determine compliance with state law and Education’s model policy. Interviewed staff at Education and the School Boards Association to discuss the development and adoption of published suicide prevention model policies.
	c. Assess the process used to develop each LEA and charter school’s pupil suicide prevention policy, and determine whether it ensured that the policy was developed in conjunction with appropriate stakeholders and experts.	Interviewed staff and reviewed policy development meeting notes from the selected school districts and charter schools to assess whether they developed suicide prevention policies and procedures in conjunction with the types of stakeholders and community organizations identified in state law and Education’s model policy.
	d. Analyze any suicide prevention training provided by the LEA and charter school, and perform the following:
	i. Identify who receives this training.	Reviewed training policies and procedures to determine whether the LEAs conduct suicide prevention training and which staff receive the training. Reviewed and analyzed training records from charter schools and one campus at each school district to determine whether teachers and staff received suicide prevention training during the 2019–20 academic year.
	ii. Identify how often and how the training is provided.	Interviewed staff and reviewed training policies, procedures, and materials to determine how and how often the selected school districts and charter schools conduct the training.
	iii. Analyze the extent to which training includes how to identify appropriate mental health services within the school and community, and when and how to refer those services.	Analyzed training materials to determine whether they include the mental health services available at the school and in the community, and when and how to refer students to those services.
	iv. Identify the content of the training, including any content related to LGBTQ and other students in categories with elevated risk.	Reviewed training materials to assess whether they included content related to students in categories with an elevated risk of suicide.
	e. Assess each LEA and charter school’s preparedness for responding to and assisting students after incidents of student suicide and attempted suicide.	Assessed whether the policies include crisis intervention plans and response plans, as well as whether those plans incorporate best practices such as student reentry protocols after a suicide attempt.
	f. Identify and assess efforts by the LEA and charter school to help students, including but not limited to the provision of mental health services and access to hotlines, materials, and other resources.	Identified employed and contracted mental health professional positions at the selected LEAs and analyzed documents and data to determine mental health professional staffing levels at middle and high schools and their related cost for fiscal year 2018–19. Further, we identified the state, federal, and local revenue sources that LEAs used to fund those positions. Identified enrollment of students at middle and high schools at each of the selected LEAs, calculated ratios of mental health professionals to students for fiscal year 2018–19, and compared these ratios to the ratios that Education recommended. Additionally, we calculated the average cost of mental health professionals per student for fiscal year 2018–19. Obtained suicide prevention‑related posters, handouts, and presentations to determine if LEAs performed outreach, and reviewed student IDs at each LEA to ensure that they included a suicide hotline phone number in compliance with state law. Five of the six LEAs included the required information on student IDs; the sixth, Redwoods Charter, did not but stated it will do so beginning in the 2020–21 academic year.
	g. Determine the extent to which the LEA and charter school’s practices align with best practices identified in Objective 4.	Reviewed the compliance of the selected LEAs’ suicide prevention policies identified in Objective 5b74 with elements of Education’s model policy representing best practices.
6	Identify best practices used at the selected LEAs that could be implemented by charter schools and best practices used at selected charter schools that could be implemented by LEAs, and areas where charter schools and LEAs would benefit from coordinating their efforts.	Interviewed staff at each charter school to determine the level of coordination of suicide prevention and mental health services between the charter school and its authorizing LEA. Reviewed school district and charter school suicide prevention policies and procedures to identify best practices.
7	Review and assess any other issues that are significant to the audit.	Interviewed LEA staff and analyzed funding sources for mental health professional spending to determine whether the selected LEAs use MHSA funds to pay for mental health providers. Interviewed staff at Health Care Services to determine how LEAs become Medi‑Cal providers. We obtained and reviewed documents describing CMS approval of billing option program changes and when the changes took effect. Analyzed state law to identify the entities eligible to participate in the billing option program and determined the total number of eligible LEAs. We obtained Health Care Services billing option program provider data to determine the number of LEAs that participated in the program as Medi‑Cal providers. We interviewed the selected LEAs to determine whether they were aware of, or participating in, the billing option program. Interviewed staff at Education and Health Care Services to determine how both agencies provide information about the program and its benefits to participating and nonparticipating LEAs. We obtained and reviewed Health Care Services’ email list to identify the number of LEAs and organizations to which it was providing billing option program information as of July 2020. Interviewed staff at the Mendocino County Office of Education and Ukiah Unified regarding their method of consolidating billing option program administrative responsibilities at the county office of education. We reviewed the memorandum of understanding between the parties to identify their responsibilities under the partnership. Additionally, we reviewed the Mendocino County Office of Education Medi‑Cal provider enrollment form to identify the LEAs participating in the billing option program through the county office of education during fiscal year 2019–20. Interviewed staff at Alameda and Fresno counties, which the Mental Health Services Oversight and Accountability Commission identified as examples of counties collaborating with LEAs to provide students with mental health services, to determine how they used MHSA funds. Fresno uses some MHSA funds to increase the number of mental health providers on school campuses, while Alameda uses MHSA funds to promote the services it provides at its school‑based health centers.

Source: Audit Committee’s audit request number 2019‑125, planning documents, and information and documentation identified in the table column titled Method.

Appendix B

Methods Used to Assess Data Reliability

In performing this audit, we relied on electronic data files that we obtained from multiple state and local agencies. The U.S. Government Accountability Office, whose standards we are statutorily obligated to follow, requires us to assess the sufficiency and appropriateness of computer‑processed information we use to support our findings, conclusions, or recommendations. Table B describes the analyses we conducted using data from the information systems we used, our methods for testing them, and the results of our assessments.

Table B
Assessment of Data Reliability
Data Source	Purpose	Method and Result	Conclusion
Education California Longitudinal Pupil Achievement Data System (CALPADS) and DataQuest, which is populated from CALPADS.	To determine the reported number of enrolled students and staff at each LEA, and to determine LEA middle and high school enrollment by grade.	Performed dataset verification procedures, electronic testing of key data elements, and reviewed existing information, and did not identify any significant issues.	Sufficiently reliable for the purposes of this audit. However, we did not evaluate the accuracy of the LEA‑reported information.
Public Health Vital Death Data	To determine the number of incidences of youth (ages 12–19) suicide by county.	Performed dataset verification procedures, electronic testing of key data elements, and reviewed existing information, and did not identify any significant issues.	Sufficiently reliable for the purposes of this audit.
Office of Statewide Health Planning and Development Hospital Encounter Data	To determine the number of instances of youth (ages 12–19) self‑harm by county.	Performed dataset verification procedures, electronic testing of key data elements, and reviewed existing information, and did not identify any significant issues.	Sufficiently reliable for the purposes of this audit.
Payroll and staffing data for: San Francisco Unified Kern High School District Heartland Charter Ukiah Unified Redwoods Charter	To determine mental health professional staffing levels, their associated costs, and the source of funding.	We performed dataset verification procedures, electronic testing of key data elements, and reviewed existing information. In addition, we shared the results of our analysis with each LEA, and obtained their confirmation of the results. However, we did not perform accuracy or completeness testing because the supporting documentation is maintained at various facilities across the state and COVID‑19 made travel to these sites to conduct such testing impractical.	We concluded that the data are of undetermined reliability. Although we recognize that this limitation may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations.
California Health and Human Services Agency Open Data Portal	To determine the number of individuals enrolled in Medi‑Cal by age 20 as of July 2019	We performed electronic testing of key data elements and reviewed existing information. We did not identify any issues.	Because these data are used primarily for background or contextual information and do not materially affect findings, conclusions, or recommendations, we determined that a data reliability assessment was not necessary.
Department of Finance Demographic Research Unit Population Projections	To determine the State’s population through age 20 as of July 2019.	We performed electronic testing of key data elements. We did not identify any issues.	Because these data are used primarily for background or contextual information and do not materially affect findings, conclusions, or recommendations, we determined that a data reliability assessment was not necessary.
Health Care Service Billing Option Program LEA Master List	To determine the total number of LEAs participating in the LEA billing option program.	We performed dataset verification procedures and electronic testing of key data elements. We did not identify any issues.	Because these data are used primarily for background or contextual information that do not materially affect findings, conclusions, or recommendations, we determined that a data reliability assessment was not necessary.
Health Care Services LEA billing option program email listserv	To determine the total number of LEAs and organizations on Health Care Services billing option program mailing list.	We performed dataset verification procedures and electronic testing of key data elements. We did not identify any issues.	Because the data are used primarily for background or contextual information that do not materially affect findings, conclusions, or recommendations, we determined that a data reliability assessment was not necessary.

Source: Analysis of documents, interviews, and data from Education, Public Health, the Office of Statewide Health Planning and Development, Health Care Services, Department of Finance, California Health and Human Services Agency, and selected LEAs.