Report 2020-104 Recommendation 10 Responses

Report 2020-104: Calbright College: It Must Take Immediate Corrective Action to Accomplish Its Mission to Provide Underserved Californians With Access to Higher Education (Release Date: May 2021)

Recommendation #10 To: Calbright College

To ensure that it is providing students with the assistance they need to graduate, Calbright should, by November 2021, establish systems to monitor the effectiveness of its student support efforts. Specifically, it should take the following actions:

1. Establish a monitoring system to ensure that it provides each student with the supports it has identified in its student support plan.

2. Conduct an annual survey of enrolled students to assess their satisfaction with its support services and instruction and with their own progress toward their educational goals.

3. Every six months, evaluate the effectiveness of the student support plan, including reviewing data on its provision of support to its students, student progress, and its annual student survey. Following its review, it should adjust the plan as necessary.

4. Include in its annual report to the Board of Governors and the public the results of its annual student survey and the steps it has taken to address student feedback.

Annual Follow-Up Agency Response From October 2023

Calbright has developed and implemented a Student Support Plan, conducted a new annual survey to enrolled students (spring of 2022), reported data from the survey to the College's Board of Trustees (summer of 2022), and will continue to evaluate and refine the Student Support Plan on a six-month cycle, as well as report annual student survey results to the Board each year. The auditor has indicated that the information provided in past updates is satisfactory, and has also requested additional information about how the College uses student progress and engagement information to inform its support activities.

Among other mechanisms, Calbright utilizes heat maps to inform outreach activities. These heat maps categorize student activity and alert the support team to students who have been inactive for distinct lengths of time (such as 10-21 days). Each week, these reports are pulled and students are then called, texted and emailed in order to assist them with persistence. Three weeks after being contacted, data is tracked to see which students re-engaged with their coursework and to evaluate the effectiveness of the outreach. From April 2023 to August 2023, for example, on average 26.7% of students completed a substantive academic activity (a module or quiz, for example) following outreach, while 47.2% did some kind of activity or reading within their coursework.

In addition, and in response to student feedback, Calbright also launched a timeline pilot in the spring of 2023 that created a non-binding structure around students' progress in the College's CRM program. In the first month after enrolling, students offered timelines as part of the pilot completed roughly 10% more program assignments, compared to students in the control group. Nearly four months into using timelines, the pilot group completed roughly 15% more assignments than the control group. The timeline structure will continue to be implemented across programs throughout 2023 and 2024.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Calbright provided documentation showing that it has incorporated some data tracking and managerial oversight to monitor students' receipt of its various support services. The documentation describes evaluating whether certain student support efforts resulted in students making further progress in their programs. It also has reported on three different student surveys, and included information in its milestone report to the Board of Governors on actions it has taken in response to student feedback.

Annual Follow-Up Agency Response From November 2022

Calbright has developed and implemented a Student Support Plan, conducted a new annual survey to enrolled students, reported data from the survey to the College's Board of Trustees, and will continue to evaluate and refine the Student Support Plan on a six-month cycle. The auditor has indicated that the information provided in past updates is satisfactory, and has also requested additional information about how the College uses student progress and engagement information to inform its support activities.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. Over the summer of 2022, Calbright faculty and staff created heat maps that are built utilizing student progress and engagement data within the College's learning platforms. Heat maps allow student-facing members of the team to quickly reference student progress, design personal and group outreach endeavors, and also help identify common hurdles and understand who is moving forward and who may need additional support.

Early indicators show that these heat maps and associated outreach activities are making an impact: During the month of September 2022, counselors reached out to students who had been inactive for 21-30 days and, as a result, 42% of the students logged back in and engaged with their coursework. In October 2022, the re-engagement rate increased to 52%.

Per the auditor's additional request, the Calbright team will provide a demonstration of College's heat maps during a meeting on November 7, 2022.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Calbright demonstrated that it has tools, which it refers to as heat maps, available to monitor students' progress and identify when a student's inactivity suggests that the student may need additional support. However, Calbright lacks formally documented procedures to establish when and how it will use those tools to ensure that students receive the support they should. As we describe in the report, if Calbright does not monitor its staff's adherence to its student support plan, it will lack assurance that it is supporting students in the way it intends. Additionally, a formalized process for monitoring that support will better enable it to evaluate the effectiveness of the support it provides. We will consider this recommendation to have been fully implemented when Calbright demonstrates that it has implemented all four elements of the recommendation, including formalizing a process for monitoring the support it provides to students and evaluating the effectiveness of that support.

1-Year Agency Response

Calbright has developed and implemented a Student Support Plan: a learner-centered approach to providing timely resources and support throughout the student journey. This plan includes resources and support provided by the enrollment, counseling, instructional, and career services teams, as well as coordination between those teams. This plan will be evaluated every six months, and annual student survey results, as well as steps taken to address student feedback, will be shared in the College's annual report to the Board of Trustees.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. The College tracks contacts with prospective students and creates "Cases" at various stages of the application process, which signal Student Support Specialists to reach out to students via phone, text, and email. Instructors and counselors also monitor student progress—through Caseload Explorer and other systems—to target outreach (e.g., to students who are nearly finished with their program, or to those who have not attempted any activity within a certain timeframe). In their one-on-one outreach to students, counselors use Recommended Schedules for Program Completion to help set goals for coursework and make sure students are on track to achieve them. Early indicators show that the student support plan is making an impact: In October 2021, 88% of Calbright students had engaged in their program in the past 180 days. During the first quarter of 2022, 88% had been engaged during the past 120 days, a 33% improvement.

Additionally, Calbright has conducted surveys of enrolled students to assess their satisfaction with their program as well as the College's support services, and plans to administer a revised survey by Q2 2022. This new annual survey will be sent to all students and cover satisfaction with our support services and instruction, with students' own progress toward their educational goals, and with program enhancements.

California State Auditor's Assessment of 1-Year Status: Pending

Calbright has updated its student support plan to include more accountability measures for ensuring that the college identifies when students may need additional support and providing such support. However, the college has not yet provided evidence that it has implemented those measures.

6-Month Agency Response

Calbright has developed and implemented a Student Support Plan: a learner-centered approach to providing timely resources and support throughout the student journey. This plan includes resources and support provided by the enrollment team, counseling team, instructional team, and career services team, as well as coordination between those teams.

Calbright uses a variety of methods to monitor the effectiveness of its student support efforts. The College tracks contacts with prospective students and creates "Cases" at various stages of the application process, which signal Student Support Specialists to reach out to students via phone, text, and email. Instructors and counselors also monitor student progress -- through Caseload Explorer and other systems -- to target outreach (e.g., to students who have not attempted any activity within a certain timeframe). In their one-on-one outreach to students, counselors use Recommended Schedules for Program Completion to help students set goals for their coursework and make sure they're on track to achieve them. The effectiveness of the Student Support Plan will be evaluated at least every six months, and the plan will be adjusted as needed.

Additionally, Calbright has conducted surveys of enrolled students to assess their satisfaction with their program as well as the College's support services, and plans to administer a revised survey by Q2 2022. This new annual survey will be sent to all students and cover satisfaction with our support services and instruction, with students' own progress toward their educational goals, and with upcoming program enhancements. Results will be presented to the College's Board of Trustees and the public.

California State Auditor's Assessment of 6-Month Status: Pending

Calbright provided a document that it stated is its Student Support Plan. The plan provides a timeline for contacting students who have applied but not yet completed the steps to become fully enrolled in the program. It also describes at a high level the support services that it will provide to students during their enrollment in the program. However, the Plan lacks specificity with regard to a monitoring system for ensuring that college identifies when students may need additional support, and for ensuring that the college provides that support. It also does not call for an annual survey of enrolled students.

60-Day Agency Response

Calbright utilizes data to monitor student progress and academic activity as well as inform our support of students towards completion. This data helps inform counselors and instructors on which students may need additional support and guidance. The platforms we utilize for the data are Salesforce Customer Relationship Management (CRM) and Strut Learning Management System (LMS).

Calbright conducts student surveys at various intervals to assess student feedback on instruction, student support services, and technology. These surveys include alumni surveys (program completion), mid-program surveys (mid-program), and end-of-term surveys (end of each six-month term). Additional feedback sources include one-to-one meetings with counselors and instructors, engagement with Calbright's support teams, Career Services interactions, informal webchats, instant messaging channels dedicated to student feedback, and what we call Calbright Cafe meetings.

Cross-functional teams meet weekly to analyze data and evaluate the effectiveness of the student support plan and recommend modifications to consider and enact. Results from surveys are shared across the college and we will be presenting our findings.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2020-104

Agency responses received are posted verbatim.