Appendix

Scope and Methodology

The Joint Legislative Audit Committee (Audit Committee) directed the State Auditor to examine BRN’s oversight of nursing programs. Specifically, we reviewed BRN’s process for approving new nursing programs or programs seeking to expand and its efforts to analyze the nursing workforce in California. The Table lists the objectives that the Audit Committee approved and the methods we used to address them.

Audit Objectives and the Methods Used to Address Them
AUDIT OBJECTIVE		METHOD
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, rules, and regulations.

2	Determine whether BRN is appropriately reviewing and approving nursing programs, including the following: a. Whether BRN’s policies and procedures for approving, denying, deferring, or revoking its approval of nursing programs comply with laws and regulations. b. Whether the factors that BRN uses when considering a request from a school to expand its nursing program are reasonable. c. Whether BRN consistently and objectively applied these factors as a part of its decision-making process for a selection of requests.	Objective 2 asked us to assess whether BRN’s policies and procedures comply with state law. We found that in matters not related to enrollment, BRN’s policies and procedures were in compliance with the Nursing Practice Act and BRN’s regulations. We made no determination as to whether BRN has authority to determine the total number of students a nursing program may enroll or whether any of BRN’s policies and procedures constitute underground regulations in violation of the Administrative Procedures Act as these issues were in litigation during our fieldwork and audit standards prohibit us from interfering with litigation. With respect to these issues, our report simply focuses on the actions BRN has taken in the recent past. Reviewed BRN’s director’s handbook, which describes the information nursing programs must provide to BRN when requesting to expand the program. Identified governing board decisions approving new programs and expanding existing programs from January 2015 through September 2019, and reviewed related governing board meeting minutes and materials. Reviewed five requests for new programs and 10 requests to expand existing programs that the governing board decided between January 2015 and September 2019 to determine if the governing board’s decision making was objective and consistent.

3	Review petitions of regulatory violations related to nursing programs filed against BRN with OAL over the last three years and summarize the outcomes of the complaint process.	Obtained and reviewed OAL’s list of petitions for regulatory violations regarding BRN and summarized outcomes.

4	Determine whether there are adequate conflict-of-interest rules or policies for governing board members, executive management, and nursing education staff who work on the oversight of nursing programs. Further, to the extent possible, identify whether BRN’s staff or governing board members appropriately recused themselves from decisions regarding nursing programs with which they may have had a conflict of interest.	Interviewed key staff at BRN and Consumer Affairs to identify relevant laws, regulations, policies, and documentation related to Consumer Affairs’ conflict-of-interest code and statements of economic interest. Identified and assessed whether Consumer Affairs’ conflict-of-interest code that applies to BRN is sufficient and appropriate. Identified governing board members, executive management, and nursing education staff required to file a Form 700, collected and reviewed each of those Form 700s for 2017 through 2019, and determined whether those individuals had any pertinent economic interests. Reviewed meeting minutes for each governing board meeting from January 2015 through September 2019 to determine whether governing board members recused themselves appropriately if their reported economic interests were the subject of board action.

5	Identify the process BRN uses to evaluate clinical displacement and whether it consistently and objectively uses that process across all nursing programs. For a selection of requests for increased enrollment or new nursing programs, assess the factors BRN evaluated in making its decisions and the resulting clinical displacement.	Interviewed key staff at BRN and determined that BRN does not evaluate clinical placements across the State. We could not assess the clinical displacement that might have resulted from BRN’s enrollment decisions because it does not track this information at that level. Reviewed BRN’s annual school survey and the stakeholder summit report to determine the extent of clinical displacement. Assessed the factors BRN evaluated as part of our review under Objective 2, including when applicable, information about clinical displacement. Reviewed BRN’s database to identify the clinical facility information it has. Determined BRN’s database to be incomplete and unreliable.

6	Determine whether BRN’s oversight of nursing programs is appropriate, including the following: a. Whether BRN is duplicating oversight of nursing programs conducted by other entities, including state and federal entities, as well as nursing school accreditors. b. An assessment of the expertise BRN relies on when it evaluates the curricula of nursing programs.	Compared BRN’s oversight requirements to national accreditation standards and processes. Reviewed National Council documents related to state boards of nursing and national accreditation. Interviewed key nursing education staff about documentation and processes related to their review of nursing programs. Determined that nursing education staff are primarily responsible for evaluating the curricula of nursing programs. Compared the hiring applications for each nursing education staff member hired after December 2014 with California Department of Human Resources’ minimum qualifications for those positions. Assessed the type of oversight nursing education staff perform and available documentation of the various processes related to BRN’s approval of nursing curricula.

7	Determine whether BRN’s analysis of California’s nursing workforce is reasonable and consistent with the scope and breadth of current and future health care workforce needs as identified by similar analyses.	Interviewed key staff at BRN to understand the process BRN uses to develop and publish studies on California’s nursing workforce forecast. Identified recent studies related to the nursing workforce in California. Reviewed key elements of the studies, including their methodologies and conclusions. Compared the methodology and findings of BRN’s nursing workforce forecast to those of other studies.

8	To the extent possible, identify the time spent and resources used by BRN on each of its programs.	Interviewed key staff at BRN and Consumer Affairs to identify and understand BRN’s budgeting practices. We could not identify the time spent and resources used by BRN on each of its programs because BRN is a single payroll reporting unit, which means it budgets and reports expenditures as a single unit. It does not track time and resources by program or organizational units. For example, its expenditures for salaries are recorded as one amount, even though BRN has staff dedicated to different units. Reviewed documentation related to BRN’s budget, including its latest budget augmentation.

9	Review and assess any other issues that are significant to the audit.	Reviewed facility approval forms for 16 nursing programs in five counties in the San Francisco Bay Area and compared the clinical facilities associated with the 16 nursing programs with OSHPD data of registered health care facilities from the same five counties to identify facilities not currently used by the 16 nursing programs. Prior to the completion of this audit, the State Auditor received a whistleblower complaint alleging that BRN executives in the enforcement division intentionally manipulated data and delivered a falsified report to the State Auditor to satisfy a recommendation the State Auditor had made during a 2016 audit of the enforcement division. In response to the complaint, the State Auditor launched an investigation and substantiated that BRN executives violated state law when they carried out a plan to artificially decrease caseloads for BRN investigators before delivering a falsified report to the State Auditor. The plan involved temporarily reassigning some of the BRN investigators’ cases to other employees who should not have had cases assigned to them. The investigation found that within 10 days of the State Auditor reviewing the falsified report and concluding that BRN had fully implemented the recommendation, BRN managers reversed the reassignments, increasing caseloads to their original level. A copy of investigative report I2020-0027, Board of Registered Nursing: Executives Violated State Law When They Falsified Data to Deceive the State Auditor’s Office, can be found at www.auditor.ca.gov. The audit team became aware of the investigation during this audit and re‑evaluated the risk assessment it conducted for the audit to ensure it could rely upon the documentation provided by BRN for this audit report. We determined that the documentation we obtained was reliable.

Source: Analysis of the Audit Committee’s audit request number 2019-120, and information and documentation identified in the column titled Method.

Assessment of Data Reliability

In performing this audit, we relied on electronic data files that we obtained from OAL related to petitions it received and from OSHPD’s website related to health care facilities. The U.S. Government Accountability Office, whose standards we are statutorily obligated to follow, requires us to assess the sufficiency and appropriateness of computer‑processed information we use to support our findings, conclusions, and recommendations. We used the data from OAL to verify that it had received two petitions related to BRN over the last three years. OAL performed for us multiple queries of its system to identify petitions related to BRN, and each query identified the same two petitions; therefore, we determined that the data were sufficiently reliable for our purpose. We also downloaded from OSHPD’s website the list of health care facilities. We used the data to identify clinical facilities that nursing programs are not currently using for clinical placements. We verified that the data included logical information; however, we did not perform completeness testing because the supporting documentation is maintained at the facilities, making such testing impractical. We concluded that the data are of undetermined reliability. Although we recognize that this limitation may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations.