Report 2020-104 Recommendation 8 Responses
Report 2020-104: Calbright College: It Must Take Immediate Corrective Action to Accomplish Its Mission to Provide Underserved Californians With Access to Higher Education (Release Date: May 2021)
Recommendation #8 To: Calbright College
To ensure that it adequately prepares its target student population to obtain positive employment outcomes after graduation, Calbright should, by November 2021, develop and implement a process for selecting and expanding educational programs that will provide value to that population; that process should include the following:
1. Collaboration with employers and industry groups to inform the content of the programs.
2. Consideration of market demand for graduates of such programs.
3. Determination of whether the programs can help its target student population obtain positive employment outcomes including jobs, earning gains, and upward mobility.
4. Available resources for program implementation.
5. An evaluation of student demand for the programs.
Annual Follow-Up Agency Response From October 2023
As noted in previous updates, Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The College had previously (in November 2022) provided a draft version of this SOP, and then received further guidance noting that a final version was necessary in order to complete this recommendation. A final version of the SOP, which includes steps the College takes to review available resources for the program (such as technology infrastructure, staffing, and licensing costs), was provided to the auditor's office in October 2023. This SOP will continue to be updated throughout the College's start-up period in order to reflect evolution in Calbright's development and operations.
- Completion Date: October 2023
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
Calbright provided its updated procedures for selecting educational programs, which now includes that Calbright will identify the costs associated with the educational program. It further includes as criteria for proceeding with a new program that Calbright has sufficient financial resources available for the program.
Annual Follow-Up Agency Response From November 2022
Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The auditor's office has shared that it is satisfied with the information provided in past updates and has made an additional request for Calbright to provide its Standard Operating Procedure (SOP) that outlines the College's process for identifying, assessing, and selecting potential programs. The auditor's office has indicated that this recommendation will be fully implemented after the SOP is provided. The SOP was shared with the auditor's office on November 7, 2022.
- Completion Date: November 2022
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Calbright provided its draft procedures for selecting educational programs. Those procedures include performing market research, identifying employer partnerships, and considering the benefits that a program will provide to Calbright's target student population. However, the procedures are still in draft form, and they do not establish a process for verifying that sufficient financial resources are available to establish the program. We will consider this recommendation to be fully implemented when Calbright finalizes a set of procedures that address all aspects of our recommendation, including verifying that resources are available to launch a program.
- Auditee did not address all aspects of the recommendation
1-Year Agency Response
Calbright has enhanced its human-centered design practice to ensure program selection strategy is directly influenced by adult learners, their unique circumstances, needs, and lived experiences. The College's evolution in this area has led to the development of a workforce-driven equity-centered design approach for program selection which acknowledges that equity requires intent and focus.
The College's decision making leverages labor market insights to assess traditional supply and demand metrics as well as in-demand occupational skills, including workforce analysis to gauge hiring probability of our target population. We utilize data sources such as EMSI to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. In addition, we maintain an Advisory Council for each program area composed of external members who provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. Research on job postings' educational attainment requirements, as well as race equity within incumbent workforce by occupation, further validate that the College's programs will enhance both job prospects and economic mobility of our target population.
We are creating market demand by responding to critical gaps in the workforce that have historically been the responsibility of the employer to address. Our recent partnership with SEIU-UHW to develop a Licensed Vocational Nurse training program is one such example of this new approach. We are working in tandem with the labor organization to address a need felt acutely in every region of the state, expand the training pipeline, and diversify the frontline healthcare provider workforce.
- Completion Date: March 2022
- Response Date: May 2022
California State Auditor's Assessment of 1-Year Status: Pending
Calbright provided documentation containing some information on factors it considers when researching new educational programs. However, it has not yet developed a clear, specific, and documented process describing the criteria that it will use to select new programs or the procedures it will follow.
- Auditee did not substantiate its claim of full implementation
- Auditee did not address all aspects of the recommendation
6-Month Agency Response
Calbright has developed and implemented a process for selecting and expanding educational/career pathways of value. This process begins with determining labor/workforce needs based on employment supply and demand, as well as available Calbright resources to expand programs, and includes quantitative validation as well as engagement with employers, advisory councils, regional workforce development boards, and worker associations to understand urgent needs and anticipate future opportunities.
Calbright maintains an Advisory Council for each program area composed of external members who, through regular meetings, provide expert feedback and collaborate to help identify in-demand jobs and the technical and soft skills they require. To date Calbright has engaged with industry leaders, employers, and advocacy groups including from the technology and healthcare sectors to both inform and validate our program pathways and career supports. To drive stronger outcomes for our students, we have also built partnerships with regional organizations and workforce development boards to align efforts to support the college's focus population.
The College utilizes data sources such as Burning Glass Labor Insights and Emsi to inform program selection, including information on total job postings in California for roles related to specific programs; regional and state-wide demand for these roles; expected job growth within California over the next decade; and whether the median and range of salaries for those positions will foster economic mobility. The College also conducts qualitative research interviews with individuals currently employed in these roles to identify pain points and key skills for their positions, as well as their career goals and aspirations.
- Completion Date: November 2021
- Response Date: November 2021
California State Auditor's Assessment of 6-Month Status: Partially Implemented
Calbright provided a document that it stated describes its process for selecting and expanding programs. The document provides a very high level overview of its process for selecting and launching new programs. It describes in greater detail Calbright's efforts to conduct market research and collaborate with labor market representatives. However, it provides little detail on how it will evaluate the appropriateness of potential programs for its target student population, whether the programs will provide positive employment outcomes for those students, and whether Calbright possesses sufficient resources to launch, maintain, and expand the program.
- Auditee did not substantiate its claim of full implementation
- Auditee did not address all aspects of the recommendation
60-Day Agency Response
Calbright's process for selecting and expanding educational/career pathways starts with determining need based on employment supply and demand, and includes engagement with employers, Advisory Councils, Workforce Boards and worker associations to understand urgent needs and anticipate future opportunities. Calbright maintains an Advisory Council for each program area composed of external members who provide expert feedback. To date Calbright has engaged with a variety of industry and employers from technology and healthcare to both inform and validate our program pathways and career supports. Designing effective programs also requires us to understand employees, their environments and what makes them successful. To drive stronger outcomes for our students we have also built partnerships with regional organizations and workforce boards to align efforts to support the college's focus population.
We also conduct Labor market research to: align academic programs to support employment in high-demand areas; determine total job postings in California for relevant job roles; understand statewide and regional demand; identify top defining and distinguishing skills for job roles (to align with our curriculum); analyze the ratio of job postings to eligible graduates who could fill them; and assess salary data leading to improved economic mobility.
Calbright's process also includes determining need based on areas where our targeted student population does not have access to programming, yet there is a high demand for such a program. We also conduct qualitative interviews to glean first-hand insights into the needs of those currently in the workforce.
We have begun to document this process and are on schedule for full implementation by November 2021. Reflected in the attached documentation are specific examples for how the data and analysis occurs for a given program prior to Calbright Board of Trustees approval.
- Estimated Completion Date: 11/2021
- Response Date: July 2021
California State Auditor's Assessment of 60-Day Status: Pending
All Recommendations in 2020-104
Agency responses received are posted verbatim.