Appendix

Scope and Methodology

The Joint Legislative Audit Committee (Audit Committee) directed the California State Auditor to review how the county and city identified, contained, and treated the recent hepatitis A outbreak. Table A.1 lists the audit objectives that the Audit Committee approved and the methods we used to address them.

Table A.1
Audit Objectives and the Methods Used to Address Them
AUDIT OBJECTIVE		METHOD
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, regulations, and policies applicable to the county’s and city’s responsibilities in responding to the hepatitis A outbreak. Reviewed other background material, including hepatitis A guidance obtained from the websites for WHO, CDC, and CDPH, as well as national hepatitis A outbreak information from CDC’s website. Identified and evaluated the roles and responsibilities of public health and medical officials at the state, county, and city levels.
2	Evaluate the county’s response to the hepatitis A outbreak by determining the following: a. Which criteria the county used to determine that the increased number of hepatitis A cases was, in fact, an outbreak and an emergency and whether the county identified, contained, and treated the hepatitis A outbreak in accordance with legal requirements and established protocols. b. Whether the county’s efforts to identify, contain, and treat the hepatitis A outbreak before the county officially declared a public health emergency in September 2017 were consistent with legal requirements and established protocols for managing an infectious disease outbreak, including the release of location data and communication between the medical directors of local municipalities. c. What steps the county has taken to prevent another infectious disease outbreak and how those steps may be helpful for other jurisdictions.	Interviewed relevant county and CDPH staff. Obtained and evaluated the county’s relevant policies and procedures related to identifying, containing, and treating infectious diseases. Obtained and examined information relevant to the county’s efforts regarding vaccination, sanitation, and education in response to the hepatitis A outbreak. Reviewed information we obtained from the county on hepatitis A cases, vaccinations, and vaccination events, as well as other relevant data. We also reviewed the correlation between the timing of administered vaccines and the identification of new cases. Obtained and evaluated HHSA’s justifi cations for withholding certain location data from affected jurisdictions. Reviewed and analyzed the county’s after action report and other documentation to identify measures to be taken to better respond to future outbreaks.
3	Evaluate the city’s response to the hepatitis A outbreak by determining the following: a. Which criteria the city used to determine that the increased number of hepatitis A cases was, in fact, an outbreak and an emergency and whether the city identified, contained, and treated the hepatitis A outbreak in accordance with legal requirements and established protocols. b. Whether the city’s efforts to identify, contain, and treat the hepatitis A outbreak before the county officially declared the public health emergency in September 2017 were consistent with legal requirements and established protocols for managing an infectious disease outbreak, including the release of location data and communication between the medical directors of local municipalities. c. What steps the city has taken to prevent another infectious disease outbreak and how those steps may be helpful for other jurisdictions.	Interviewed relevant city staff. Reviewed and analyzed documents relevant to the city’s ability to proclaim a local emergency, pass emergency ordinances, and enact other emergency-related orders. Reviewed and analyzed the city’s after action report and the programs, procedures, and guidance the city is implementing to better respond to future hepatitis A outbreaks.
4	Determine whether the county or city was aware of the potential for an infectious disease outbreak among a particular population before March 2017 and, if so, whether they had the resources to prevent an infectious disease outbreak of this magnitude.	Interviewed relevant county and city staff. Reviewed relevant documents regarding the potential risk of a disease outbreak among the homeless and illicit drug-using populations. Reviewed relevant documents regarding the resources available to the city and county to prevent an infectious disease outbreak of this magnitude. Determined that the State has plans for general emergency response. Although the plans do not specifically address outbreaks among particular populations, they address infectious disease outbreaks. Information from CDC indicates that the county was one of the first jurisdictions in the United States and the first county in California to experience this type of hepatitis A outbreak in the post-vaccine era.
5	Determine whether the city’s policy for approving public works contracts in response to declared emergencies differs between contracts awarded any other time and whether the city approved contracts related to the hepatitis A emergency in accordance with applicable legal requirements and its own policy.	Interviewed relevant city staff. Obtained and evaluated the city’s policies and procedures related to contracting for goods and services in emergency and nonemergency situations. For outbreak-related contracts, we also analyzed whether the city complied with its policies. Obtained, reviewed, and analyzed information related to the nine contracts that the city used during the local health emergency. For the contracts regarding right-of-way sanitation, we reviewed and analyzed charges from and payments to the vendor. Determined that the city appropriately followed its processes for awarding emergency contracts related to the hepatitis A outbreak and for paying for services under the emergency contract for right-of-way sanitation.
6	Review and assess any other issues that are significant to the audit.	Obtained and evaluated relevant information regarding CDPH’s roles and responsibilities related to controlling the spread of infectious diseases and reporting outbreaks to other communities. Interviewed relevant CDPH staff to gain an understanding of their roles and to determine their perspectives on the effectiveness of the county’s response to the outbreak. Obtained and evaluated CDPH’s hepatitis A outbreak response plan.

Sources: Analysis of the Audit Committee’s audit request number 2018-116, as well as information and documentation identified in the table column titled Method.

Assessment of Data Reliability

In performing this audit, we obtained electronic data files extracted from the data sources listed in Table A.2. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer-processed information that we use to support our findings, conclusions, or recommendations. Table A.2 describes the analyses we conducted using the data from these sources, our methods for testing, and the results of our assessments. Although these determinations may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations.

Table A.2
Methods of Assessing Data Reliability
Data Source	Purpose	Method and Result	Conclusion
County Web Confidential Morbidity Reporting System	To determine the number of cases, hospitalizations, and deaths resulting from the hepatitis A outbreak and to determine where and when these cases occurred.	We performed data-set verification procedures and did not identify any significant issues. We conducted electronic testing of key data elements and did not identify any significant issues. Because this system is paperless, we were unable to perform completeness or accuracy testing. Furthermore, we did not perform a review of the controls over these data because of the significant resources required to conduct such an analysis. To gain some assurance of the accuracy of the data, we compared the data to the case numbers that the county calculated and found that the totals materially agreed with our calculations.	Undetermined reliability for the purpose of this audit. Although this determination may affect the precision of numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations.
County San Diego Immunization Registry (SDIR)	To determine the number of hepatitis A vaccinations administered in the county during the hepatitis A outbreak, when they were administered, and the type of organization that administered them.	We performed data-set verification procedures by comparing the total number of records in the data file to the data description provided by the county and by reviewing key fields for errors. For example, we reviewed the date-of-birth field to ensure that only adult vaccinations were included in the data. We did not note any concerns. We also reviewed the county’s procedures for identifying and removing test records that do not represent real patients from the data, and we identified discrepancies related to the county’s removal of test records from the data because test records are not clearly labeled as such in the data. We assessed the possibility of such discrepancies materially affecting our conclusions as low. We performed completeness testing by comparing the total number of vaccinations in the SDIR data to other reports generated by the county and found the totals to be materially close. Because over 400 user/provider facilities can input data into SDIR, it would have been cost-prohibitive to perform in-depth sampling and testing of records to determine their accuracy.	Undetermined reliability for the purpose of this audit. Although this determination may affect the precision of numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations.

Source: Analysis of various documents, interviews, and data from the county.