Appendix A

Scope and Methodology

The Audit Committee directed the California State Auditor’s Office to conduct an audit of the Commission’s decision-making practices and its implementation of the merit system. The audit scope included nine objectives. The table below lists the objectives that the Audit Committee approved and the methods we used to address them.

Audit Objectives and the Methods Used to Address Them

AUDIT OBJECTIVE		METHOD
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, regulations, Commission rules, a collective bargaining agreement, and other background materials applicable to the Commission and the District.
2	Determine whether the Commission has adhered to fair decision-making practices that are consistently applied in its processes used to make personnel decisions.	To identify best practices, reviewed relevant rules and other documentation and interviewed representatives from three human resources organizations that perform similar functions: the State Personnel Board, the State Center, and the Los Angeles Unified School District Personnel Commission (LAUSD Commission). Compared the Commission’s rules for making personnel decisions related to issues such as classifying and disciplining employees to the best practices that we identified to identify potential changes the Commission could make to better ensure the fairness of its decisions. Judgmentally selected and reviewed 20 decisions that the Commission made from fiscal years 2016–17 through 2018–19. We examined reclassifications, out-of-class claims, and disciplinary appeals to determine whether the Commission made the decisions in compliance with its rules. Because the Commission does not comprehensively track all complaints, reviewed its speaker logs for Commission meetings from fiscal years 2016–17 through 2018–19 to identify complaints made in those meetings. Identified four complaints from the speaker logs and 21 email complaints, and reviewed the related documentation to determine whether the Commission handled these complaints adequately. Judgmentally selected and reviewed 12 Commission meetings from fiscal years 2016–17 through 2018–19 to determine whether the Commission documented each non-agenda speaker during those meetings.
3	Review the accountability measures that the Commission uses and determine whether they are sufficient for a public hiring agency.	Interviewed Commission staff and assessed the Commission’s accountability measures to determine if it can ensure that it makes personnel decisions fairly and consistently. Interviewed representatives from the State Personnel Board, State Center Commission, and LAUSD Commission to identify any accountability measures they use to assess performance that may be appropriate for the Commission to consider adopting.
4	Review the Commission’s candidate application and examination process and determine whether it is fair, relevant, and structured in a way that allows employees equal opportunities to promote.	Reviewed lists of the examination and eligibility appeals the Commission received and examinations it conducted from fiscal years 2016–17 through 2018–19. From these lists, we judgmentally selected and reviewed five examinations for which at least one applicant filed an examination or eligibility appeal. We selected 10 applicants per examination—five that the Commission determined were eligible to take the examination and five that the Commission determined were not eligible—for a total of 50 applications. We also selected an additional 94 applications that the Commission determined were not eligible—for a total of 144 applications of all types—to better assess whether the Commission appropriately disqualified applicants. We describe this work in more detail in Objective 9. Attempted to review additional records concerning job examinations but were unable to do so because of the circumstances described in the section below titled Scope Limitation. Evaluated key steps in the Commission’s examination application process, such as screening for minimum qualifications, to determine whether the process is fair, consistent, and structured in a way that allows the Commission to select and advance employees on the basis of merit after fair and open competition. We determined the Commission did not verify high school transcripts. As a result, the Commission did not determine whether 13 candidates met minimum qualifications. However, we verified the Commission required applicants to demonstrate full-time, paid work experience relevant to the job for which the examination was being held. Evaluated key steps in the Commission’s examination process, such as conducting examinations and creating eligibility lists, to determine whether that process is fair, consistent, and structured in a way that allows it to select and advance employees on the basis of merit after fair and open competition. Assessed whether the Commission’s implementation of its application and examination processes was fair by determining whether it adhered to key rules and steps in its processes. Collected and reviewed best practices related to the application and examination process from the State Personnel Board, State Center Commission, and LAUSD Commission to identify potential opportunities for the Commission to improve its processes.
5	Review the Commission’s merit system rules and determine whether they are consistent with state law and the applicable memorandums of understanding.	Compared the relevant state law and the Commission’s rules related to reclassifications, examinations, out-of-class claims, appeals, and discipline for classified employees. Assessed the Commission’s rules to determine whether they are consistent with the bargaining agreement between the staff union—which represents a majority of classified staff—and the District.
6	Determine whether there are sufficient management controls for the Commission’s budgets and expenditures.	Interviewed a commissioner, the assistant director, and District staff to assess the District’s oversight of the Commission’s budget and whether it is sufficient to minimize the risk of fraud, waste, and abuse related to the Commission’s budget and its expenditures. Reviewed budget documentation from the District and Commission from fiscal years 2016–17 through 2018–19 to determine the effect of the Commission’s expenditures on the District’s budget.
7	Identify the Commission’s process for determining and considering how its decisions affect the District’s budget.	Interviewed a commissioner and the assistant director to determine the extent to which the Commission has a process to assess how its decisions affect the District’s budget. We interviewed District staff to obtain their perspectives on the Commission’s decisions and their effect on the District’s budget. Reviewed documentation that demonstrates how the Commission’s decisions affect the District’s budget.
8	Review the Commission’s appeal process and determine whether it adequately considers employee grievances.	Reviewed the Commission’s rules for reviewing and processing appeals. Reviewed rules and other documentation related to appeals and grievances from the State Center Commission, LAUSD Commission, the State Personnel Board, and the California Department of Human Resources. Compared these to the Commission’s rules to identify potential opportunities for the Commission to better ensure that it adequately considers employee appeals and grievances. Judgmentally selected 10 appeals the Commission decided on from fiscal years 2016–17 through 2018–19, including some appeals related to the examinations we selected to review for Objective 4. Evaluated the appeal decisions to determine whether the Commission followed its rules and time frame when addressing the appeals. Determined that the Commission made its decisions in a reasonable time frame. Obtained and reviewed emails sent to and from the former director and assistant director from fiscal years 2016–17 through 2018–19 to determine whether Commission staff received complaints and addressed them appropriately.
9	Review and assess any other issues that are significant to the audit.	After receiving allegations from several District employees about the Commission using inconsistent practices and unfairly disqualifying applicants from examinations, selected 10 additional examinations posted from July 2017 through September 2020, and up to 10 disqualified applicants for each, for a total of 94 applicants. We assessed whether the Commission appropriately disqualified these applicants and used consistent practices when doing so. We also selected 20 individuals whom the Commission debarred during the same time period and reviewed relevant documentation and obtained Commission perspective to assess whether the Commission communicated required information and timely informed these individuals. For 10 debarments, we also assessed whether the Commission used a consistent approach when deciding to debar the individual. We performed various procedures to address the other allegations we received, as Table 2 outlines. Judgmentally selected four Commission staff based on their length of employment at the Commission and their job positions and interviewed them about the Commission’s work environment. Reviewed the Commission’s organization charts for fiscal years 2014–15 through 2019–20 to assess the extent of its staff turnover.

Source: Analysis of the Audit Committee’s audit request number 2020‑111 and audit workpapers.

Scope Limitation

To review the Commission’s candidate application and examination process as Objective 4 requires, we initially selected five examinations and obtained 10 applications associated with each of the examinations. We attempted to obtain an additional selection of the same number of examinations and associated applications; however, the Commission’s director stated that, because of the COVID-19 pandemic, Commission staff were unable to access the Commission’s physical location and hard copy records. As a result, our review was limited to the 50 applications we had initially obtained rather than the 100 applications we planned to review. After further discussion with Commission staff, we obtained remote access to its application and examination system and were able to review an additional 94 applications. However, because this system only contains documentation from fiscal year 2017–18 and later, and it does not contain certain documentation—such as rating sheets—it was not possible to fully perform our intended review. For this reason, our review of the additional applications was limited to assessing the Commission’s screening of applications for minimum qualifications. Although this limitation affected the scope of our review, we obtained sufficient evidence in total to support the findings and conclusions we present in this report.

Assessment of Data Reliability

The U.S. Government Accountability Office, whose standards we are statutorily obligated to follow, requires us to assess the sufficiency and appropriateness of computer-processed information we use to support our findings, conclusions, or recommendations. In performing this audit, we relied on the following data and systems:

Examinations

We relied on summaries of the Commission’s eligibility lists to identify the number of examinations the Commission offered multiple times within a 12-month period and to make a selection of those examinations for further review. Because we used these data solely for selecting examinations, we reviewed key elements to ensure that they contained logical data and performed completeness testing and data-set verification procedures. We did not identify any issues. To verify the completeness of these summaries, we compared the total number of examinations the Commission reported in its annual report for each fiscal year of the audit period to the total number of examinations on the summaries. We determined that these data were sufficiently complete for the purpose of selecting items for further review.

Public Complaints

We used the Commission’s public meeting speaker log to identify the number and nature of complaints it received from fiscal years 2016–17 through 2018–19. To assess the completeness of these data, we judgmentally selected 12 Commission meetings that occurred during those fiscal years and reviewed the meeting minutes to determine whether each speaker from the meeting whose comment required Commission action or follow-up was described in the log. We determined that the Commission included all relevant speakers from these meetings in its log and that these data were sufficiently reliable for the purpose of identifying non‑agenda speaker complaints from Commission meetings.

Appeals

We relied on a list of the appeals the Commission received from fiscal years 2016–17 through 2018–19 to select appeal cases and examinations for further review. The Commission maintains hard copy appeal files, but due to the COVID-19 pandemic, we were unable to haphazardly select items from those files to compare to the list. Therefore, to assess the completeness of this list, we compiled a list of appeals from our review of certain employee emails and determined whether the appellant described in the email was listed on the appeals logs. We found that the appeals list was incomplete, as one of the seven appeals we identified in emails was not listed on the appeals log. According to the assistant director, this was due to a clerical error. Although the population of appeals from which we selected items was incomplete, this list was the best source of such data available and there is sufficient evidence in total to support our findings and recommendations.

Commission’s Application and Examination System

We used data from the Commission’s application and examination system to make a selection of rejected applicants, candidates, and eligible candidates for further review. We also obtained documents from this system related to applications and examinations. Due to the COVID-19 pandemic, we were unable to obtain source documents and could not perform data reliability testing on this system; therefore, the data are of undetermined reliability. However, there is sufficient evidence in total to support our findings and recommendations.

Number of District Employees

We used data from the California Community Colleges Chancellor’s Office’s management information systems data mart to determine the District’s total number of employees, number of classified employees, and number of enrolled students. Because we used these data solely for background or contextual information that does not materially affect findings, conclusions, or recommendations, we determined that a data reliability assessment was not necessary.