COMMITMENT • INTEGRITY • LEADERSHIP	View PDF report View Fact Sheet View Selected Abbreviations
California Air Resources Board Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals Report Number: 2020-114

Use the links below to skip to the section you wish to view:

• Chapter 1—CARB Has Not Done Enough to Collect and Analyze Data to Demonstrate Its Programs’ Effectiveness in Reducing GHG Emissions
• Chapter 2—CARB Has Not Adequately Demonstrated Its Programs’ Socioeconomic Benefits
• Other Areas We Reviewed

---

Chapter 1

CARB Has Not Done Enough to Collect and Analyze Data to Demonstrate Its Programs’ Effectiveness in Reducing GHG Emissions

Chapter Summary

The State is at risk of not meeting its GHG reduction goals for 2030, and CARB has acknowledged the need for greater emissions reductions from the transportation sector. Because transportation‑related emissions are the largest single source of GHG emissions in California and because they have increased in recent years, it will be necessary to accelerate the pace of reduction if the State is to achieve its goals. Given the ambitious nature of these goals and the progress still needed to meet them, it may be reasonable for CARB to operate multiple programs that address a shared transportation objective. However, to ensure that it is operating the most effective mix of programs, it is important for CARB to identify the GHG reductions that each individual program achieves.

CARB does not formally account for the incentive programs’ overlap with relevant regulatory programs, nor does it collect and analyze all of the data necessary to isolate the incentive programs’ GHG emissions benefits. CARB has also not done enough to measure or collect data about behavioral changes of the participants in its consumer‑focused incentive programs, further limiting its ability to identify the programs’ individual impacts on emissions.

The State needs more accurate program measurement to guide its GHG investments and increase the chances of meeting its GHG goals. The measurement concerns we identified limit the usefulness of the information CARB reports annually to the Legislature about the GHG reductions from its cap‑and‑trade‑funded incentive programs. Specifically, CARB overstates the programs’ GHG reductions, although it is not clear by how much. These reporting issues also obscure the programs’ true cost‑effectiveness. Finally, CARB must do more to ensure that the emissions reductions it reports are accurate and supported by valid underlying data.

CARB Has Not Identified the GHG Reductions Its Incentive Programs Provide

CARB’s regulatory and incentive programs are intended to work toward shared objectives for reducing GHGs, such as increasing the number of ZEVs on California’s roads. CARB has taken steps to identify the new GHG reductions its regulatory programs are intended to achieve and has generally designed those reductions to be achievable through the regulatory programs alone. However, it has not done enough to demonstrate the amount of GHG reductions it projects and measures for its incentive programs. CARB does not collect certain data or perform measurements that would help it determine those additional impacts, including the extent to which the incentive programs motivate consumers to purchase lower‑emission vehicles that they otherwise would not have purchased. CARB’s lack of measurement of participants’ behavioral changes also limits its ability to project the longer‑term benefits of its incentive programs.

CARB Cannot Distinguish the Individual GHG Benefits of Multiple Programs That Work Toward Shared Objectives

Given the ambitious nature of the State’s goals for GHG reductions, it may be reasonable for CARB to operate multiple programs that work toward a shared transportation objective, such as putting more ZEVs on California’s roads or reducing GHG emissions from heavy‑duty vehicles. However, to ensure that it is operating the most effective mix of programs, it is important for CARB to identify the GHG reductions that each individual program achieves. The short time frame remaining for the State to meet its 2030 GHG goals also underscores the need for the best information possible to guide the use of the State’s limited resources.

We engaged an environmental consultant to help us review CARB’s methods for projecting the GHG emissions reductions it expects its programs to achieve. For the eight regulatory programs and nine of the incentive programs we reviewed, our consultant determined that CARB’s methodologies for estimating emissions reductions were generally reasonable on a program‑by‑program basis. ³  The solicitation for one of our 10 selected incentive programs, the Zero‑Emission Drayage Truck Pilot, was not released until late November 2020, and we therefore did not include it in our review of GHG emissions reductions.However, because of the potential overlap among many of these programs, we—along with our consultant—also evaluated whether CARB accounted for that overlap when designing the programs and when measuring the programs’ actual estimated GHG reductions.

For the eight regulatory programs we reviewed, which CARB approved or modified between 2007 and 2020, CARB generally identified relevant overlap between those proposed regulatory programs and other existing transportation regulatory programs. For example, when it proposed a new regulatory program in 2018 to require transit agencies to purchase only zero‑emission buses in the future, CARB took steps to account for GHG reductions from the existing regulatory programs that set emissions requirements for buses. By doing so, CARB isolated the additional GHG emissions reductions that the new regulation would achieve.

In contrast, although the proposed regulatory programs sometimes acknowledged the existence of incentive programs that share the same objectives, CARB did not formally assess how relevant incentive programs might overlap with the proposed regulatory programs. When we asked CARB why it did not assess these overlaps, key staff and program managers for two regulatory programs explained that they did not consider incentive programs when determining the regulation’s costs. Specifically, CARB does not assume that incentive funds will be available in the future to help manufacturers and consumers offset vehicle costs. Therefore, CARB designed these regulatory programs to achieve the expected reductions without help from incentive programs that target the same types of vehicles. Although reasonable, this approach means that any GHG reductions it claims from incentive programs that work toward the same objectives should be above and beyond what the regulatory programs achieve. Because incentive programs can represent significant costs, it is important that CARB be able to measure those additional GHG reductions to ensure that it is pursuing the most effective approaches to meeting the State’s goals.

However, CARB has not done enough to measure the emissions reductions its incentive programs achieve beyond the reductions brought about by its regulatory programs. We reviewed documentation related to a selection of CARB’s incentive programs and found that CARB generally did not acknowledge the potential overlap with regulatory programs or discuss how it accounted for the overlap in the incentive programs’ design. CARB also does not measure precisely how many additional GHG benefits its incentive programs may actually provide. For example, as we discuss in the Introduction, CARB has a regulatory program that requires car manufacturers to sell ZEVs and it operates incentive programs that provide payments to consumers to encourage them to purchase ZEVs. According to CARB’s 2019 Zero Emission Vehicle Credits report, released in August 2020, car manufacturers were overcomplying with the ZEV regulation. This overcompliance could indicate that CARB’s ZEV incentive programs achieve additional GHG emissions reductions by driving consumer demand and ZEV sales beyond what would be sold under the regulation alone. However, CARB has not determined whether its incentive programs are responsible for the additional sales.

CARB offered two primary reasons for not performing such a determination, but these reasons suggest a lack of coordination in measuring how the respective programs perform. First, CARB does not track the number of ZEVs sold over and above the number required by the regulation because the regulatory program is not designed to measure or require reporting on the number of vehicles sold. Instead, the regulatory program uses a system in which manufacturers earn a certain number of credits per vehicle sold in California, and CARB monitors manufacturers’ credit balances to ensure that they have complied with the regulation. In addition, some ZEVs earn more credits than others. For example, a vehicle with a 350‑mile battery range earns the manufacturer four credits, while a vehicle with a 50‑mile range earns only one credit. Although longer‑range battery‑electric vehicles do not have a greater direct impact on GHGs, because they may be more attractive to consumers, the regulatory program awards more credits for these vehicles to reward manufacturers for producing and selling them.

In contrast, CARB measures the GHG emissions reductions achieved by its incentive programs using the number of vehicles for which CARB provides a rebate. CARB staff explained that although it is feasible to report the credits generated in a year and the number of ZEVs sold, CARB has not done so because of the reporting requirements and how it evaluates compliance with the regulation. Because it has not taken steps to coordinate the performance measurements for regulatory and incentive programs, CARB does not know how the additional ZEVs being sold in the State compare to the number of rebates CARB provides through its incentive programs. Being able to make that comparison would be an important step in analyzing the extent to which CARB’s incentive programs achieve additional emissions reductions.

Second, the manager for CVRP, the largest incentive program, stated that CARB cannot conclusively identify how many ZEVs are on the road because of CVRP, as many factors play a role in a consumer’s decision to purchase one. The manager explained that if the market for ZEVs is increasing, CARB assumes that CVRP is having a positive impact on the market. However, without conducting analyses to validate these assumptions, CARB cannot accurately identify the GHG emissions reductions that its ZEV incentive programs are actually achieving.

We identified similar gaps in CARB’s analysis of how HVIP, an incentive program for low‑ and zero‑emission trucks and buses, relates to CARB’s regulatory program that establishes GHG emissions limits for heavy‑duty trucks and buses. As it does in the ZEV regulatory program, CARB uses a credit system to monitor its regulation of GHG emissions for heavy‑duty vehicles. Manufacturers can earn credits by producing low‑ or zero‑emission heavy‑duty vehicles, such as trucks and buses, and HVIP provides incentives to encourage companies and transit agencies to buy these vehicles. However, unlike their practice for the ZEV program, staff do not track the total number of credits that all manufacturers have generated under the heavy‑duty GHG regulatory program across the entire industry; instead, CARB explained that it assesses each manufacturer individually. The CARB branch chief who oversees compliance with this regulation explained that CARB has been focused on ensuring that individual manufacturers comply with the regulation but acknowledged that it would be possible to calculate the industrywide credit balance. Until it does, CARB does not know whether the industry as a whole has overcomplied with the requirements in a way that might indicate an additional effect from incentives.

Moreover, CARB has not evaluated the extent to which manufacturers are using HVIP‑eligible vehicles to comply with the GHG regulations for heavy‑duty vehicles. Without assessing the relationship between HVIP and the related regulatory program, CARB cannot demonstrate whether HVIP is achieving GHG reductions over and above what the regulatory program would achieve on its own. As with the ZEV programs, a greater level of coordination here is needed. In fact, the branch chief confirmed that CARB collects the data it needs to determine how frequently manufacturers have used vehicles eligible for incentives like HVIP to comply with regulatory requirements, but that the analysis would require coordination across different divisions of CARB.

CARB has also not isolated the GHG reduction benefits of incentive programs that overlap with one another. In addition to sharing the same transportation objective of increasing ZEVs in California, three of the incentive programs we reviewed—CVRP, Clean Cars 4 All, and Financing Assistance—also overlap because qualifying consumers are able to receive financial support from one, two, or all three programs for a single purchased vehicle. Therefore, if CARB does not account for this overlap, it could lead to the double or triple counting of emissions reductions associated with a single vehicle. Despite this risk, our consultant found that CARB’s methodologies for estimating the emissions reductions from the Clean Cars 4 All and CVRP programs do not account for the overlap or specify how CARB avoids simultaneously assigning the emissions reductions from a single vehicle to multiple programs. Because the Clean Cars 4 All and Financing Assistance programs are relatively small compared to CVRP and because not all consumers are eligible for incentives from multiple programs, the effect of the overlap on CARB’s emissions estimates may be relatively small. However, given that the Governor’s budget for fiscal year 2021–22 proposed directing increased funding to existing transportation programs such as Clean Cars 4 All and Financing Assistance, the importance of this measurement issue could significantly increase if CARB does not take steps to address it.

CARB Does Not Know How Frequently Its Incentive Programs Change Purchasing Behavior

Knowing the extent to which its incentive programs change participants’ behavior is crucial for CARB to accurately assess the effectiveness of those programs in achieving additional GHG reductions. Research has shown that it is important to measure behavioral responses to economic incentives such as rebates in order to determine an incentive’s effectiveness. For example, determining how frequently an incentive payment influences consumers to purchase a cleaner vehicle than they otherwise would choose is a way of accounting for individuals who would purchase cleaner vehicles anyway—sometimes called free‑riders, as we discuss in the Introduction. CARB can then apply this information to its calculations of the vehicles’ impacts on GHG emissions to determine the reductions for which the incentive program is responsible.

Conversely, failing to collect data on or measure behavioral effects could lead CARB to overstate the emissions reductions from its incentive programs. Such overstatement will occur if CARB gives an incentive program credit for the GHG reductions from all vehicles for which the program paid incentives, even if some consumers would have purchased some of those vehicles anyway. In such a scenario, the State would not be able to rely fully on the estimated GHG reductions reported for individual programs or on the relative cost‑effectiveness of those reductions across programs. Problems with the reliability of this information would prevent CARB and the Legislature from being able to use it to direct the State’s limited resources away from less effective programs and toward more effective ones.

Despite the importance of knowing whether incentive payments influence individuals’ decisions, CARB has not done enough to measure or collect data about how its consumer‑focused incentive programs have changed participants’ behavior. As Table 1 shows, for five programs we reviewed in which CARB provides an incentive payment or other financial assistance to consumers who purchase a cleaner vehicle, CARB collects information about behavioral changes of participants for only one—CVRP. Specifically, CVRP participants who receive rebates are given a survey that asks whether they would have acquired their clean vehicle without the rebate. Two of the other programs have participant surveys that include questions about participants’ satisfaction with their vehicles, among other topics. However, CARB was unable to adequately explain or provide documentation for its rationale as to why those surveys do not include questions about what led the participant to purchase the vehicle. For another program, the program manager stated that CARB does not have copies of the survey questions that the external program administrators use; however, we found that the sample survey questions that CARB provides to the program administrators do not contain questions about whether participants would have purchased their vehicle without receiving an incentive. The CARB program manager for the last program—FARMER—could not remember why CARB does not require a survey of participants, but a program staff member noted that the air districts that help administer FARMER on CARB’s behalf stated that it was difficult to collect all of the required program information from participants and that the districts expressed hesitation to add more questions.

Table 1

CARB Rarely Collects or Uses Data on Participants’ Behavior for Key Incentive Programs

Program	Is there an existing survey that could collect participant behavioral data?	Is data collected about participant behavioral changes?	Has CARB used the data to adjust the program?
Clean Cars 4 All	Yes	No*	No
CVRP	Yes	Yes	Yes
FARMER	No	No	No
Financing Assistance	Yes	No	No
HVIP	Yes	No	No

Program managers for the three incentive programs with surveys that do not collect behavioral change data expressed their belief that participants would not have purchased their vehicles without the incentives, but they could not demonstrate whether their belief is justified. For example, the manager for HVIP stated that there are not very many purchases of HVIP‑eligible vehicles—zero‑ and low‑emission trucks and buses—without an incentive and that HVIP‑eligible vehicles represent a small part of the total market for heavy‑duty vehicles. The manager’s claim suggests that HVIP is the reason consumers purchased the vehicles and that they would not have done so without the incentive payment. However, the manager also stated that CARB does not have any data to show what proportion of eligible vehicles were purchased with an HVIP voucher or what proportion of total heavy‑duty vehicle purchases in California are for HVIP‑eligible vehicles. When we asked for such data, the manager acknowledged that CARB has only anecdotal information from vehicle manufacturers who indicated that all or a majority of their zero‑ or low‑emission vehicle sales occur because of incentives like HVIP. Further, the manager stated that CARB presumes there are few purchases of HVIP‑eligible vehicles as a proportion of total heavy‑duty vehicles because they are more expensive than conventional vehicles. However, without actual data, improved surveys, or both, CARB cannot confirm its assumptions that HVIP participants would not have purchased their vehicles without the incentive.

CARB is also missing an opportunity to use data to design programs that maximize their effects on consumer behavior and therefore on GHG emissions. As Table 1 shows, CARB has made changes to only one program, CVRP, based on behavioral considerations. The manager for CVRP explained that over the seven years that CARB has collected survey data, it has used the data to inform and support program changes, and provided documentation of one such change—a limit on the price of vehicles for which consumers can receive CVRP rebate—approved in October 2019. Specifically, the external program administrator for CVRP noted that according to the program’s survey data, as the cost of a vehicle increases, the influence of a rebate on the purchasing decision decreases. The CARB program manager told us that CARB considered this survey result when determining the maximum price for eligible vehicles. However, our review of the survey data raised other questions about participant behavior that CARB could not answer. Between 2013 and 2019, 41 percent to 54 percent of respondents stated that they would have purchased their vehicle even without receiving the CVRP rebate. The CVRP manager explained that CARB knows that there are free‑riders in any incentive program. However, the manager also stated that CARB has not reached any conclusions about what these responses indicate for CVRP’s effectiveness and although the response rates can help justify general changes to CVRP, CARB has not specifically used these response rates to make any policy changes to the program.

In 2015 CARB approved an increase to the CVRP rebate for lower‑income consumers; the text box lists the current amounts of both the standard and increased rebates. Afterward, CARB added a question to the CVRP survey to ask consumers who received the increased rebate whether they would have acquired their vehicles without the increased rebate amount. The responses to the survey indicated that about half of the participants who had lower incomes and received higher rebate amounts answered that they would have acquired their clean vehicles even without the extra payment—a higher proportion than responded this way regarding the smaller, standard rebate. This result raises questions about whether the increased amount is effective in changing consumer behavior. However, when we asked CARB what the survey results indicated about the program, the CVRP manager explained that CARB has not used this aspect of the data to make any program changes though the manager went on to say that CARB would likely review it to examine the efficacy of the additional rebate.

CARB may be missing opportunities to use other sources of data to learn more about how effective its programs are in changing behavior. For instance, in addition to CARB’s CVRP rebate, the federal government provides a federal income tax credit for certain electric vehicle purchases. The federal income tax credit can be as much as $7,500 for certain battery electric vehicles, which is more than CARB’s maximum CVRP rebate of $4,500 for the same type of vehicle. Because the federal income tax credit program is limited to a certain total number of qualifying vehicles per manufacturer, the applicability of the federal income tax credit program for these types of electric vehicles will phase out over time. CARB could use the expiration of the federal income tax credit to help it assess the tax credit’s effect on behavior and use that information to evaluate the effectiveness of CVRP. For example, if consumers stop buying vehicles that are no longer eligible for the tax credit, it might mean that incentives cause consumers to purchase vehicles but that CVRP’s current rebate amounts are not a great enough incentive on their own.

In addition, CVRP rebates make up a smaller percentage of ZEV sales in California than they did previously. Figure 4 shows that CVRP rebates have declined in proportion to total ZEV sales in the State over the last six years. Although CARB attributes this trend to an income limit for CVRP eligibility that CARB implemented in 2016 at the direction of the Legislature, it has not analyzed its data or the broader ZEV market to confirm the reason for the trend. There may be value in learning why consumers who purchase ZEVs without rebates decide to do so, but the CVRP manager explained that CARB has not conducted such an analysis because to do so CARB would likely need help from car dealerships, who are generally uncooperative and would not participate unless paid to do so. However, the manager did not describe any formal efforts CARB has made to collect such information. Although potentially challenging, collecting and analyzing data from these additional sources—along with better use of its own survey tools—could help CARB understand the extent to which its programs change consumers’ behavior and thereby help CARB better measure its programs’ actual emissions reductions. Understanding its programs’ current effects on participant behavior might also help CARB make changes in those programs to increase their cost‑effectiveness and have a greater impact on emissions reductions.

Figure 4
CVRP Rebates Account for a Decreasing Share of ZEV Sales in California

Source: Analysis of published CVRP rebate information from CARB’s program administrator and ZEV sales information from the California Energy Commission.

CARB Has Not Assessed Its Programs’ Effect on Other Behavioral Changes That Are Necessary to Achieve GHG Reductions

We also reviewed whether CARB measures its programs’ impacts on other aspects of participant behavior that CARB depends on to achieve GHG reductions. For example, some of CARB’s pilot programs fund low‑ or zero‑emission transportation services, such as car‑sharing programs and vanpools. CARB intends for these programs to reduce GHG emissions by getting people to change their driving habits. CARB has planned to use the data gathered and lessons learned from these programs to identify opportunities to develop expanded programs in the future. A second kind of CARB pilot program provides funding for clean vehicles, such as zero‑emission buses, to transit agencies and other entities to support larger‑scale deployments of ZEVs, thereby accelerating their introduction and market penetration. Ultimately, this second kind of pilot program seeks to encourage more widespread use of these clean vehicles in California. Knowing whether these programs have the intended effects is crucial to determining their effectiveness at reducing GHG emissions, both now and in the future. However, CARB has not done enough to accurately measure the programs’ effects on participant behavior.

For example, CARB operates a car‑sharing pilot program that provides funding to establish car‑sharing fleets in or near disadvantaged communities to offer alternative modes of transportation and encourage the use of clean cars. Through this program, CARB has awarded funds to several program administrators throughout the State to implement various kinds of car‑sharing projects that use clean vehicles. CARB’s emissions reduction projections for the pilot program assume that all of the trips taken in the shared clean vehicles replace trips that would otherwise be taken in a conventional car. However, as our consultant identified, this assumption would overstate the program’s emissions reduction if trips that participants made in clean vehicles instead replaced trips on public transit or by walking. Therefore, CARB should determine if the assumptions in its methodology are supported by participants’ behavior. One way to do so would be to collect information from participants about how they previously made the trips they now take in the shared clean vehicles.

However, it is not clear whether CARB will have appropriate data to reliably measure the program’s behavioral changes. The branch chief who oversees the car‑sharing pilot program and the program’s manager told us that CARB does not have data about changes in program participants’ behavior. CARB requires the program administrators for the individual car-sharing pilots to conduct initial and follow‑up surveys of program participants. However, CARB was not able to provide follow‑up surveys for two of the pilots, explaining that it is working with its program administrators to determine timelines for the follow‑up surveys and that all of the pilots have seen delays. When we reviewed the follow‑up surveys for the four pilots CARB has collected so far, we found that the surveys for only two contained questions that could allow CARB to identify what kind of transportation participants replaced with shared vehicles. Subsequently, in January 2021, CARB provided a recently approved, additional follow‑up survey for one of the two pilots that was previously missing the necessary information. Although the new survey does contain one such question, this pilot began in 2015 and we are concerned that CARB did not better ensure the collection of this information earlier in the program’s existence, which would have allowed CARB to more closely monitor the program and be certain that it has the data it needs to evaluate its assumptions.

Further, it is not clear that CARB will have better information when the car-sharing pilot program has ended because grant agreements do not consistently specify what data the follow‑up surveys must contain. Without ensuring that it has this information and that the information is complete and usable, CARB cannot verify its assumption or determine precisely how effective its car‑sharing pilot program is at reducing GHGs. The branch chief explained that a program evaluation contract CARB signed with the University of California, Berkeley (UC Berkeley) in May 2020 should provide insight into the impacts of pilot programs, including the car‑sharing pilot program, on users’ lives. However, results from this evaluation will likely not be available until 2022.

CARB also does not have sufficient data on the longer‑term behavioral changes that some of its pilot programs for heavy‑duty vehicles intend to achieve. For instance, CARB operates a bus pilot program that provides grant funding to enable transit agencies and other entities to acquire multiple zero‑emission buses and associated infrastructure, such as charging equipment. CARB intends for the pilot program to help the participating agencies address challenges in deploying zero‑emission buses and to encourage those agencies and others to transition more of their fleets to zero‑emission buses, thereby reducing GHGs. To know whether the pilot program is having the intended effect, CARB stated that it measures behavior changes for participants in its bus pilot program by tracking whether participating transit agencies go on to invest in additional clean buses during or after the end of the pilot program. If they do, the program may be responsible for more considerable GHG reductions over the longer term. Achieving such long‑term reductions is a goal of CARB’s pilot programs in general.

Further, the information that CARB uses to measure behavioral changes related to the bus pilot program does not allow it to fully determine whether the program has had its intended effect. Specifically, the information indicates which participating agencies applied for other incentive programs from CARB or other agencies to purchase additional buses. However, that information alone does not allow CARB to conclude that the bus pilot program was the reason for the subsequent vehicle purchases. Without collecting and reviewing data about why participating agencies do or do not choose to acquire more clean buses after participating in the bus pilot program, CARB cannot be certain that the pilot program is responsible for changing agencies’ behavior.

CARB staff also referred us to another current contract with a third party for some of its heavy‑duty vehicle programs, which it expects will help it determine whether its pilot programs are successful, including the bus pilot program. However, the contract’s scope of work does not include any specific requirement to ask participants whether they would have changed their purchasing behavior if they had not participated in the pilot program. The scope of work includes an evaluation of whether the vehicles in the pilot program can replace conventional vehicles and the practicality of widespread adoption of those vehicles. Although CARB entered into this contract in 2017, final reports that include the contractor’s analysis of the technology and participant experiences for the program’s various individual pilots were not yet completed at the time of our review in October 2020.

Better Program Measurement Would Help the State Maximize Its Efforts to Reduce GHG Emissions

CARB’s inability to measure the GHG reductions from its incentive programs diminishes the usefulness of the information the State uses to fight climate change. Currently, CARB’s reporting to the Legislature on the effectiveness of its incentive programs does not accurately estimate the emissions reductions that each of its programs achieves. CARB has responded to other specific questions from the Legislature about how the State can use its programs to pursue longer‑term emissions reductions. However, CARB’s lack of information about how its programs change consumer behavior and its imprecise metrics for defining success have limited the usefulness of its responses to the Legislature.

As we describe in the Introduction, CARB reports annually to the Legislature on the estimated GHG emissions reductions its programs achieve as well as the programs’ costs. For the nine incentive programs our consultant reviewed, CARB generally estimates GHG reductions based on the number and types of vehicles for which it paid incentives during the year. ⁴ The solicitation for one of our 10 selected incentive programs, the Zero‑Emission Drayage Truck Pilot, was not released until November 2020, and we therefore did not include it in our review of GHG emissions reductions. For instance, CARB estimates the GHG reductions from CVRP by calculating the difference between the emissions for the clean vehicles participants purchased and the emissions of an average new vehicle CARB assumes the consumer would otherwise have purchased. CARB then multiplies those reductions by the total number of vehicles the incentive helped pay for. This approach is generally consistent across CARB’s incentive programs that we reviewed.

Although this is reasonable on a per vehicle basis, CARB’s approach leads it to overestimate its incentive programs’ total GHG reductions because it does not account for the effects on industry and consumer behavior of regulatory programs and other factors we have discussed, such as the federal tax credit. Instead, CARB assumes that the emissions reductions from all of the vehicles funded by an incentive program would not otherwise have occurred. By attributing all vehicles to a program without considering these other factors, CARB is overstating the programs’ benefits in its annual reports to the Legislature though it is unclear by how much. One effect of this overstatement is to obscure the programs’ cost‑effectiveness in reducing GHG emissions. Taking the steps we describe in the previous section to account for the incentive programs’ overlap with regulatory programs, as well as to improve efforts to ensure that incentive programs cause consumers to make choices they otherwise would not, would help CARB get closer to identifying—and reporting—the actual emissions reductions its incentive programs achieve.

As the primary report on the status and outcomes of all programs funded by cap‑and‑trade—including the programs that CARB operates—CARB’s annual report must contain accurate estimates of the GHG emissions reductions the funds have achieved. The Legislature has referenced GHG emissions reductions that programs achieve when it has codified programs into state law. For example, effective in 2018 the Legislature established in law the Clean Cars 4 All program, which previously had been a pilot program that CARB operated only in certain parts of California. The Legislature indicated that one of the reasons for expanding the program and making it permanent was the GHG emissions benefits that the pilot program had achieved. The annual reports also contain information that the Legislature has relied on, in part, when considering other changes to state law regarding how cap‑and‑trade funds are spent. Without accurate information in the annual reports—which would make these reports a reliable emissions measurement tool—the Legislature’s ability to make decisions about its investments to help the State achieve its GHG goals may be hampered. Specifically, if the annual reports contain accurate information, these reports can better help the Legislature make decisions about whether to continue funding a given program at its current level, decrease funding and use those resources elsewhere, or significantly increase funding to help the State achieve its GHG emissions goals.

In addition to the annual reports, the Legislature has required that CARB publish information about how certain programs will help CARB meet the State’s goals. For example, the Supplemental Report of the 2018–19 Budget Act required CARB to update its forecast annually to indicate how much CVRP funding will be necessary to meet the State’s objective of five million ZEVs on the road by 2030. However, because CARB stated that it does not have data to calculate how CVRP drives ZEV sales or how other factors may change the future need for CVRP rebates, CARB could not be fully responsive to this question in the analysis it included in its fiscal year 2019–20 funding plan. CARB responded to the Legislature’s request by assuming that five million vehicles would be sold and used historical rebate payment information to calculate the price to the CVRP program. Because it assumed that CVRP would provide rebates for half of the ZEVs sold in California based on its historical data, CARB projected that it would ultimately need to pay out 2.1 million rebates totaling $5.6 billion to get five million ZEVs on the road, although it noted that it would not reach that goal until after 2030, given the current ZEV market. This approach suggests that CVRP payments are merely a by‑product of ZEV sales and does not provide the Legislature information about how to use CVRP funding to actually generate ZEV sales. The CVRP program manager indicated that CARB took the approach it did with the data it had available. However, we believe that if CARB pursued some of the analysis and data we describe in the previous section, it could offer useful information about how to invest in its incentive programs.

The Legislature has also asked CARB about its plans for meeting longer‑term objectives. Effective January 2019, it directed CARB to establish a three‑year investment strategy that includes estimated funding needs for zero‑ and near‑zero emission heavy‑duty vehicles and equipment to meet certain state goals, including lessening the impacts and effects of climate change. The strategy identified several key areas in which CARB’s heavy‑duty incentive programs must be successful in order to meet the State’s goals, including support for the future evolution of cleaner technology. In the strategy, CARB explained that current technology is insufficient to meet California’s long‑term air quality and climate change goals. Therefore, CARB intends its incentive programs to support the development, improvement, and commercialization of cleaner technology until the technology can either be supported with other programs or exist without the support of the programs.

However, CARB has not yet adequately defined the metrics it will use to determine when the incentive programs have successfully accomplished these goals. Instead, CARB’s strategy identifies anecdotal evidence about production costs and observations about the use of technologies as metrics for determining the programs’ success. CARB program managers stated that the metrics do not include quantifications or indicators of success because it is still working to identify and develop what those should be. CARB explained that before the publication of the next strategy document for fiscal year 2021–22, it will continue to work with stakeholders to identify what data are available that could be used to measure whether a technology can transition to a different program or stand on its own. One of the program managers stated that ideally, when the next strategy is published, a few of the metrics will have more details about how CARB plans to quantify success and determine whether a technology is able to transition out of the program. We are concerned that CARB has not yet identified quantifiable metrics to determine when its heavy‑duty programs have succeeded in supporting a new technology. The state law requiring CARB to develop this investment strategy took effect in January 2019 and CARB has identified the need to evaluate the effectiveness of its investments in HVIP since at least 2014. Until it has developed these informal metrics into measurable objectives, CARB will be limited in its ability to know when an incentive program has succeeded in supporting new technology to the point that it can either be supported by a different program or stand on its own.

Lastly, to ensure the effective future use of state resources to fight climate change, CARB must have plans for how it will phase out or adjust incentive programs when those programs have met their goals. However, CARB’s investment strategy for heavy‑duty vehicle programs does not include details for phasing out relevant incentive programs, and staff were not able to provide them when we asked whether CARB had formal projections for when HVIP would no longer be necessary to support a specific kind of technology. Planning for how long CARB’s incentive programs will be needed is part of ensuring the success of its other programs as well. For example, in 2019 CARB established a regulatory program that will require, beginning in January 2029, all new bus purchases by transit agencies in California to be for zero‑emission buses. CARB recognized that a transition of the bus fleet to entirely zero‑emission buses could be challenging for transit agencies and, as a result, identified its commitment to using incentive programs to help with the transition. However, some transit agencies have told CARB that the zero‑emission buses currently in production do not serve all of their needs. Accordingly, the program manager explained to us that HVIP may still be needed to support and develop the market for zero‑emission buses after the regulatory compliance dates have passed. The manager also acknowledged that CARB does not have a projected date for when HVIP incentive payments for zero‑emission buses will no longer be necessary. Although making these projections will likely be challenging for the newest technologies, CARB must work to establish clear metrics for when incentive programs have achieved their goals as well as strategies for how to use those resources thereafter. Without doing so, CARB risks continuing to fund programs that are no longer needed to actively support GHG reduction objectives.

CARB’s Measurement of Emissions Reductions Suffers From Errors and Issues With Its Programs’ Underlying Data

CARB has not done enough to ensure that the reductions in emissions it reports are fully supported by program data. Our review of CARB’s emissions reductions in its most recent annual report found several errors in the calculations CARB reported to the Legislature—some of them significant. In other cases, CARB cannot provide documentation to support the accuracy of the data it uses in its aggregate reports. Finally, CARB’s inability thus far to measure emissions by region creates uncertainty about a statewide transportation program it administers with regional planning organizations, and it undermines the State’s ability to identify and pursue the best regional strategies to ensure that the program is successful in reducing GHGs.

CARB Has Not Done Enough to Ensure the Accuracy of the Data It Reports

The emissions reductions that CARB has included in its annual report to the Legislature have not always been accurate and fully supported by its program data. CARB estimates emissions reductions using the methodologies we describe in the preceding section—generally based on emissions reductions calculated from the number and type of vehicles for which it provided incentives—and the data it receives from the program administrators who handle the daily operations of CARB’s incentive programs. Those data contain the number of incentives paid for low‑ and zero‑emission vehicles and specifics about the vehicles, such as vehicle type, engine specifications, and model year. With assistance from our consultant, we reviewed CARB’s calculations of emissions reductions for the most recent annual report, issued in March 2020. Our review found that CARB’s calculations were generally accurate, based on the methodologies for those programs. However, we found instances in which CARB incorrectly applied its own methodologies and therefore made errors in its calculations. One of these errors had a relatively small impact. Specifically, CARB overstated the GHG reductions by 3 percent for heavy‑duty vehicles with a specific type of engine—one of several types of vehicles supported by the HVIP program.

However, errors affecting other programs had larger impacts. For example, CARB used the wrong time frame for its CVRP and Financing Assistance programs when calculating the emissions reductions for vehicles those programs helped pay for. As a result, CARB underestimated the amounts it reported to the Legislature for those programs’ emissions reductions by more than 40 percent during the second half of the 2019 reporting cycle, which is included in the 2020 annual report. After we raised these errors in the CVRP, Financing Assistance, and HVIP calculations with CARB, program staff informed us that they had already found and corrected internally some of the errors and would work to correct the others. However, we believe these are errors CARB should have identified before publishing the annual report. Further, at the time of our review, CARB had not provided any corrected emissions estimates to the Legislature. Additionally, for the Clean Cars 4 All program, staff were unable to provide documentation to support CARB’s emissions reduction calculations for the entire second half of the 2019 reporting cycle, which is also included in its 2020 annual report. Without retaining this supporting documentation, even for recent years, CARB cannot demonstrate that the program accomplishments included in its annual reports are accurate.

CARB also has not always ensured that the program activity it includes in its annual reports is fully supported by underlying program data. CARB staff aggregate detailed data from the program administrators to summarize a program’s total incentive payments and emissions reductions in its annual reports. For the five programs we reviewed that provide incentive payments or other financial incentives to support consumers’ vehicle purchases, we used a selection of data from the years 2016 through 2020 to compare the programs’ underlying data to the totals given in CARB’s annual reports. We found that the data for two of the programs did not align with the total incentive payments in its annual reports. For CVRP, we found discrepancies between the cap‑and‑trade spending totals in the program data and the totals CARB reported to the Legislature. Although these discrepancies were relatively small—an average difference of 7 percent of the total dollar amount reported by CVRP—they occurred in each of the four reporting cycles we reviewed going back to 2016. Program staff were unable to adequately explain the discrepancies. We did not fully assess the underlying data for the Financing Assistance program because CARB staff informed us that they had already identified inaccuracies in the data that had affected the annual public report. The program manager explained that errors, including incentive payment amounts and omitted data, had led CARB to report inaccurate information over the history of the program, which began in 2015.

Finally, CARB has not taken sufficient steps to ensure the accuracy of the program data it uses in its calculations. Because CARB relies on its program administrators to compile and provide program data, we reviewed how CARB ensures that the data they compile are accurate. Program staff explained that CARB relies on what it refers to as desk reviews to ensure that the data are valid and supported by appropriate documentation. During these reviews, CARB examines the documentation for specific incentive payments to determine whether the vehicle data for those incentive payments are accurate, among other tasks. Significant inaccuracies in the data could affect estimated emissions reductions as well as undermine CARB’s efforts to track the specific types of vehicles its incentives pay for as part of longer‑term efforts to support emerging technology. Even so, CARB has completed these desk reviews for only two of the five programs we reviewed that make such payments to consumers—HVIP and CVRP.

Further, we have concerns about the frequency of the reviews and the evidence CARB maintains to support its conclusions. CARB has not conducted any desk reviews for HVIP since 2014 and has conducted only two reviews for CVRP—in 2014 and 2020. The report from the 2020 CVRP desk review indicates that CARB staff did not identify documentation issues, and CARB staff told us that the issues it identified in its 2014 HVIP review would not have been significant enough to affect emissions reporting for the program. Nonetheless, the documentation CARB maintained from the reviews is not sufficient for us to independently determine whether that is the case. Managers for HVIP and CVRP stated that they also work less formally with the program administrators to review supporting documentation on an ongoing basis. However, given the importance of the program data to CARB’s reporting and broader GHG objectives, more frequent and widespread formal reviews are needed across CARB’s incentive programs.

A Regional GHG Reduction Program Is Failing, but CARB Has Not Collected Data to Determine Why

As we discuss in the Introduction, CARB also provides statewide oversight for the Sustainable Communities program, which is intended to help the State reach its 2030 GHG goals by reducing vehicle travel and therefore GHG emissions. State law requires certain regional planning organizations (governed by local officials) to prepare strategies designed to achieve regional GHG emissions reduction targets. The strategies are aimed at reducing per capita GHG emissions from passenger vehicles, which they do largely by reducing vehicle travel. Pursuant to state law, CARB established specific GHG reduction targets for the 18 distinct regional planning organizations throughout California, which account for nearly all of the population in the State. State law also requires CARB to review the strategies submitted by the regional planning organizations. In approving the strategies, CARB must determine whether the strategies, if implemented, will achieve the GHG reduction targets it set.

Although CARB has updated the Legislature on the progress of the Sustainable Communities program, that update did not contain key information needed to evaluate the program’s success. Beginning in 2017 the Legislature required CARB to assess the progress made by each regional planning organization toward meeting the GHG emissions reduction targets set by CARB and to report that information to the Legislature every four years. In its first and only report thus far, released in November 2018, CARB reported that California is not on track to achieve the program’s GHG reduction targets. However, CARB did not report the progress made by each regional planning organization in meeting the regional targets, as required by state law. CARB explained in the report that it was unable to find a data source to report that information accurately, and it instead reported changes in GHG emissions statewide. In doing so, CARB concluded that passenger vehicle emissions had not declined as significantly as expected and that vehicle travel had actually increased. Accordingly, CARB concluded that the regional planning organizations either were not implementing the strategies or were not achieving the expected results. However, because CARB did not identify GHG emissions changes by region, it could not determine which of these explanations was more responsible for the lack of emissions reductions or whether any regional strategies were actually working. CARB also could not determine which specific types of activities might be successful in reducing GHGs via reductions in vehicle travel.

CARB’s inability to measure GHG emissions by region creates uncertainty about the future success of the Sustainable Communities program. When we asked CARB about its progress in finding a data source for regional GHG emissions, it indicated that it had not found one and does not know whether it will have regional data in time for the 2022 report. As a result, we are concerned that CARB and the State may not be able to identify and pursue the best regional strategies for reducing emissions in time to help the State meet its 2030 GHG goals.

Recommendations

To improve its ability to isolate each of its incentive programs’ additional GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs’ overlap with other programs that share the same objectives. As part of that process, CARB should document how it will account for the overlap to allow the most accurate program measurement possible.

As part of its work to measure both incentive and regulatory programs’ additional GHG reductions, by February 2022 CARB should begin collecting and analyzing the data it needs to assess the extent to which the requirements in its regulatory programs are being exceeded by manufacturers. To the extent applicable, that analysis should focus on the components of the requirements that overlap with CARB’s incentive programs, such as the extent to which manufacturers are complying with regulations for heavy‑duty vehicles via low‑ and zero‑emission vehicles.

To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer‑focused incentive programs, as well as information about the behavioral effects of programs that other entities offer, such as the federal tax credit.

To better assist the State in achieving its GHG goals, CARB should use the information we describe to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer‑term planning documents or reports.

To promote transparency and inform stakeholders, beginning in December 2021, CARB should prepare an annual report for its board and the Legislature on its progress in isolating the GHG emissions reductions attributable to each of its regulatory and incentive programs. As a part of this report, CARB should identify any measurement challenges that persist and highlight any administrative barriers that prevent it from obtaining the information it needs to perform better analysis.

To strengthen the accuracy and integrity of its emissions reduction reporting, CARB should immediately begin retaining all supporting documentation it uses to perform calculations of GHG reductions for its cap‑and‑trade‑funded incentive programs for a period of at least five years. In conjunction with this change, CARB should also document the justification for any instances in which the underlying data it uses to compile its annual reports vary from the information it publishes in those reports.

To better ensure the accuracy of its program data, by August 2021 CARB should develop a formal schedule and procedures for reviewing the supporting documentation maintained by its program administrators. These procedures, which CARB should begin using with the 2022 annual report, should specify a minimum number of records to review in relation to the program’s size, should specify how staff will collect and maintain evidence to support conclusions, and should be standardized across all of CARB’s incentive programs.

To ensure that the State is positioned to assess the status of the Sustainable Communities program, by April 2021 CARB should report to the Legislature whether it will have a usable source for measuring regional GHG emissions in time for the 2022 report. If CARB believes it may not, it should identify any administrative or bureaucratic barriers it faces in accessing data it needs for the estimates and request relevant action by the Legislature to make those data available.

---

Back to top

Chapter 2

CARB Has Not Adequately Demonstrated Its Programs’ Socioeconomic Benefits

Chapter Summary

State law requires CARB to spend a portion of its cap‑and‑trade funds to provide benefits to disadvantaged communities as well as low‑income communities. The history of these requirements demonstrates the Legislature’s desire to target cap‑and‑trade funding in a way that provides benefits for Californians most in need. However, although in recent years CARB has exceeded the minimum requirements in state law for program spending in disadvantaged and low‑income communities, it has significant opportunities to better evaluate and demonstrate its programs’ effectiveness.

CARB operates incentive programs that it intends to have specific, tangible socioeconomic benefits for participants. However, it does not consistently collect data to determine whether the programs actually provide those benefits. CARB has also missed opportunities to use data it has already collected to determine whether participants are receiving the intended benefits. Finally, CARB has been slow to measure the jobs its programs are required to support, and it has not done enough to measure the benefits of the job training activities that its own guidelines require. Given that programs designed to provide socioeconomic benefits may cost more than other CARB programs and do not focus primarily on GHG reductions, it is crucial that CARB effectively define and measure the socioeconomic benefits they provide.

The Legislature Has Shown a Desire to Focus Benefits on Californians Most in Need

Requirements in state law for cap‑and‑trade spending in disadvantaged and low‑income communities have become more specific over time. As discussed in the Introduction, disadvantaged communities are classified based largely on their sensitivity to environmental pollution, while the low‑income community classification is based solely on income statistics. In 2013 state law required CalEPA to identify disadvantaged communities in California. CalEPA performed this identification by zip code based on a range of factors that focused primarily on sensitivity to the physical environment, such as air pollution. At that time, state law required a minimum of 25 percent of all cap‑and‑trade funds to benefit these disadvantaged communities, with 10 percent to be spent on projects located within those communities.

The original approach to identifying disadvantaged communities proved to be too broad. In 2014 CalEPA determined that identifying communities by zip code did not allow for enough precision and changed the identification of disadvantaged communities from zip codes to census tracts—smaller geographical areas determined by the U.S. Census Bureau with assistance from local entities and intended to allow statistical comparisons. This change allowed for a more detailed assessment of each community’s status because there are more than four times as many census tracts as there are zip codes in California, and census tracts are established with the goal of having roughly equivalent populations. Even with this change, some of the cap‑and‑trade funds counted toward the required minimum spending under the state law in effect at that time even if those funds benefited disadvantaged communities but were not spent on projects located within those communities. Under this approach to defining eligible spending, many cap‑and‑trade dollars were not spent within the disadvantaged communities most in need. For example, our review of two of CARB’s largest incentive programs found that in 2016 significant proportions of spending categorized as benefiting disadvantaged communities took place outside those communities’ borders. Specifically, program data indicate that HVIP spent nearly one‑third of its $8.4 million in disadvantaged community spending in areas physically outside of disadvantaged communities; for CVRP it was more than 80 percent, or $34 million, of its total $42 million in reported disadvantaged community spending.

Acknowledging the need to better target cap‑and‑trade spending, in 2016 the Legislature amended state law to focus this cap‑and‑trade spending primarily on projects located within the boundaries of, and benefiting individuals living in, disadvantaged communities. The amended law significantly limited the amount of cap‑and‑trade spending that could take place outside of disadvantaged communities and still be considered as benefiting those communities. Effective January 2017, the law now requires that all cap‑and‑trade funds that count toward the disadvantaged community funding requirement be spent on projects located within the boundaries of the disadvantaged communities.

Also effective in 2017, the Legislature established a cap‑and‑trade spending requirement for a second group of communities, referred to as low‑income communities. Here too the Legislature encouraged a higher degree of precision in focusing the benefits of this cap‑and‑trade funding on Californians who need them most. Specifically, the amended law defines low‑income communities as census tracts where the median household income is below certain statewide thresholds, defined in part as 80 percent of the statewide median income. The law also requires 10 percent of cap‑and‑trade dollars to benefit these newly defined communities.

When establishing the low‑income community funding requirement, the Legislature again demonstrated a desire to ensure that the new benefits went to Californians most in need. Although defining these communities at the census‑tract level was more precise than previous requirements for disadvantaged communities, the Legislature also acknowledged that some of the best GHG reduction strategies are those that benefit low‑income households directly, regardless of where those households are located. Accordingly, the amendment, which took effect in 2017, also gives agencies that receive cap‑and‑trade funds the option to measure and report low‑income community spending based on the actual households that the programs benefit, as opposed to reporting only at the census‑tract level.

In recent years, CARB has exceeded the minimum spending requirements for disadvantaged and low‑income communities. In its 2020 annual report, CARB reported that the State has spent a cumulative 39 percent of its cap‑and‑trade funding in and benefiting disadvantaged communities and 21 percent of funding in and benefiting low‑income communities since August 2017. Further, our review of selected programs found that CARB generally exceeded the separate minimum 25 percent and 10 percent requirements for each of those programs. For example, in 2019, program data show that CARB spent nearly 50 percent of its total cap‑and‑trade funding for HVIP in disadvantaged communities and more than 15 percent in low‑income communities, both well over the required minimum. However, CARB’s reporting could be more precise. Specifically, for the consumer‑based programs we reviewed, CARB reported spending in low‑income communities at the census‑tract level and not in terms of actual households. Reporting at the household level, where possible, would provide more valuable information about programs that serve individuals and families. Further, as we discuss in the following sections, spending in disadvantaged and low‑income communities is generally the only information CARB measures or reports regarding the socioeconomic benefits of its programs, even though its programs have specific intended benefits that focus on individuals and households. Given the Legislature’s desire to focus and track the non‑GHG benefits of cap‑and‑trade‑funded programs, we believe CARB has significant opportunities to better evaluate and demonstrate the programs’ effectiveness, including by reporting at the household level.

CARB Cannot Sufficiently Demonstrate That Its Programs Achieve Socioeconomic and Jobs Benefits

As required by state law and its own guidelines, CARB operates certain transportation incentive programs to achieve socioeconomic benefits for participants. Although CARB uses cap‑and‑trade revenue to operate these programs, the programs place more emphasis on the socioeconomic benefits than on maximizing GHG reductions. As a result, the programs may cost significantly more than other incentive programs relative to the GHG reductions they achieve—underscoring the importance of ensuring that they actually provide the other intended benefits. However, although CARB has identified specific socioeconomic benefits it intends its programs to achieve, it does not consistently collect the data it needs to demonstrate those benefits, and it has been slow to collect required socioeconomic data, such as data related to job creation. Further, CARB has missed opportunities to use data it currently collects to ensure that participants are receiving the intended benefits.

CARB Does Not Evaluate Programs to Determine Whether They Achieve Intended Socioeconomic Benefits

CARB’s programs that are aimed primarily at producing socioeconomic benefits receive cap‑and‑trade funding but do not have GHG reduction as their primary goal. CARB categorizes the programs it develops for disadvantaged and low‑income communities as equity programs. Equity programs that focus on socioeconomic benefits may cost significantly more than CARB programs that focus primarily on reducing GHGs relative to the GHG reductions they achieve because they offer higher incentive payments or require more intensive program administration to provide socioeconomic benefits to participants. For example, CARB’s data indicate that the cost to reduce one ton of GHG emissions through CARB’s Financing Assistance program is seven times the cost of doing so through its CVRP program. Some of the higher costs come either from the larger cash incentives that the programs provide per vehicle to the low‑ or middle‑income Californians who participate or from the more intensive program administration required to achieve the additional socioeconomic benefits the programs are expected to provide. For example, grant agreements we reviewed for the Financing Assistance program allow 14 percent or more of the total grant funding to go towards personnel costs for the program administrators, who work closely with participants, while the CVRP grant agreement limits administrative costs—including but not limited to personnel costs—to 7 percent. Because equity programs are less cost‑effective in terms of emissions reductions, it is critical that CARB demonstrate each program’s value by clearly defining and measuring its socioeconomic benefits.

Although CARB has identified specific socioeconomic benefits that it intends for its equity programs to provide participants, it does not consistently collect data to determine whether the programs actually provide those benefits. As we discuss above, CARB’s program administrators for its Financing Assistance program work closely with participants, providing training on financial literacy with required topics such as credit, bank services, and vehicle expenses to ensure consumer protection and increase the chance of successful loan repayment. For example, in its fiscal year 2017–18 funding plan, CARB stated that it intended for the Financing Assistance program to provide economic benefits to participants such as increased credit scores, the ability to qualify for housing loans, and access to more reliable transportation. However, the metrics CARB identified that it will use to measure progress do not allow CARB to actually measure these benefits. Instead, they include the number of consumers who participate, costs and types of vehicles purchased, and loan repayment rates, none of which allow CARB to determine whether the program achieves the intended benefits. Further, over the five years CARB has operated the Financing Assistance program, it has not collected data that would allow it to report on benefits achieved. For example, despite repeatedly citing participants’ loan repayment rates as a measure of the program’s success in its annual funding plans, CARB does not collect data about or report on these rates. CARB stated it does not have access to this information because the program administrators do not manage the participants’ loans, leading us to question why CARB identifies loan repayment as a metric to measure program success. We expect CARB to identify metrics for which it can collect the data and measure the benefits when designing and requesting funding for the program.

CARB also does not collect data that allows it to measure the socioeconomic benefits of participation in its car‑sharing pilot program. In its annual funding plans, CARB has described improvements in participants’ access to mobility as a metric with which to measure the program’s success. When we asked CARB how it measures this metric, the program manager explained that CARB expects to see participants taking trips that they would not have been able to without the program, such as traveling outside of their neighborhoods for medical appointments. However, CARB does not consistently collect this information from participants. Specifically, although the grant agreements require initial and follow‑up participant surveys be conducted by the program administrators, they do not require program administrators to ask questions about changes in mobility. Accordingly, the surveys do not consistently collect information about such changes. The program manager also confirmed that CARB has not evaluated its car‑sharing pilot program to determine such mobility.

CARB’s failure to collect and measure relevant program data greatly limits its ability to inform the Legislature and other stakeholders about whether the programs are effective. As Figure 5 shows, despite the specific socioeconomic goals CARB identified for its programs, its public reporting on their benefits is generally limited to the broad spending requirements we discuss in the first section of this chapter. In May 2020, CARB entered into a contract with UC Berkeley to, among other things, evaluate the socioeconomic benefits of the car‑sharing program, including increased access to transportation and increased economic opportunity. However, CARB does not expect a final report on the results from the evaluation until 2022, at which point the program will have been operating for five years without an appropriate evaluation of the benefits it provides to participants.

Figure 5
CARB Establishes Specific Socioeconomic Goals but Only Reports on Broad Outcomes

Source: CARB’s funding plans, 2020 annual report to the Legislature, and interviews with CARB staff.

CARB also does not consistently use the data that are available to demonstrate benefits from programs and determine whether the programs are achieving their goals. CARB’s Clean Cars 4 All program provides incentives for lower‑income consumers who scrap their old vehicles and purchase new clean vehicles, ostensibly providing more reliable and less costly transportation. Consequently, CARB states that the intended benefits of its Clean Cars 4 All program include maximizing economic opportunities for participants, reducing their cost of driving, and minimizing work interruptions because of unreliable transportation. However, as Figure 6 shows, CARB has missed clear opportunities to assess whether the Clean Cars 4 All program actually provides these socioeconomic benefits. CARB administers the program through grant agreements with local air districts, which perform the day‑to‑day activities of the program. These grant agreements require the air districts to conduct surveys of program participants. When entering into multiple grant agreements with several air districts, which it did between 2015 and 2020, CARB provided the air districts with identical sets of example surveys. The example surveys include questions that indicate how CARB intended to measure the socioeconomic benefits it expects the program to provide, such as whether participants’ employment opportunities have changed as a result of the program and whether their income has changed. CARB could use the results from these survey questions to evaluate whether the program is providing the socioeconomic benefits it expects. However, although the grant agreements require the air districts to conduct surveys of participants, they do not require the air districts to ask specific questions or use the example surveys. Instead, the grant agreements allow the air districts to design their own survey questions. The program manager explained that CARB does not have copies of the actual surveys the air districts use because it has not asked for them. As a result, CARB does not know what questions the surveys ask.

Figure 6
CARB Has Missed Opportunities to Measure Benefits for Its Clean Cars 4 All Program

Source: CARB’s funding plans, grant agreements, and interviews with CARB staff.

CARB has also missed opportunities to use data it currently has to determine whether participants are receiving the intended benefits of its programs and to monitor the effectiveness of those programs. Although greater financial stability for participants is a shared goal of both the Financing Assistance program and Clean Cars 4 All, CARB does not analyze data it collects on program participants that could help it determine whether the programs are contributing to improved financial stability. For both programs, CARB collects data about participants’ personal financial circumstances, including details about their household income and vehicle loans—such as loan amounts and interest rates. CARB’s manager for Clean Cars 4 All told us that CARB uses the data in these reports to monitor the performance of the air districts, determine whether the districts are meeting statutory obligations in administering the program, and determine funding needs and annual statutory goals. However, that manager also confirmed that CARB has not made any changes to the program based on the data. Similarly, the manager for the Financing Assistance program stated that CARB uses the data in the quarterly reports from its external program administrators to ensure that the program is working but confirmed that CARB has not made any changes to the program based on the data contained in the reports. The manager explained that the programs each have a low volume of data, meaning that it is not useful for analyzing program effectiveness. Given that Financing Assistance is a small‑scale pilot program, a low volume of data for the program is reasonable; however, the scale of the program does not prevent CARB from analyzing the available data and using that analysis to inform changes if necessary.

The program data we reviewed show that interest rates for participants’ loans vary significantly, with some rates as low as 1 percent and others exceeding 25 percent. Given that helping consumers overcome the barrier of obtaining vehicle financing by providing low‑interest loans is a primary purpose of the Financing Assistance program and that CARB intends for Clean Cars 4 All to maximize economic benefits for participants, we expected that CARB would be analyzing the loan information to determine whether the programs are operating as expected and in ways that are beneficial to participants. However, CARB has not formally done so for either program. The Financing Assistance manager stated that CARB expects participants to pay off their loans and avoid vehicle repossession. When we asked whether CARB has formally collected data on vehicle repossessions, CARB reviewed past reports from its two external program administrators and learned that there were five repossessions reported by one and zero by the other. However, CARB has not formally analyzed the information and is accordingly unable to demonstrate whether these numbers indicate that the program is succeeding. These numerous instances of uncollected and unused data prevent CARB from determining whether its programs are providing the specific intended socioeconomic benefits. As a result, it also has not been able to inform stakeholders, such as the Legislature, about the extent to which these programs are succeeding.

CARB Has Been Slow to Measure Job Creation and Has Done Little to Evaluate Job Training

CARB has also done relatively little to evaluate the effects of its programs on required job creation and training. As discussed in the Introduction, state law directs CARB, to the extent feasible, to use cap‑and‑trade funds in a way that fosters job creation by promoting GHG emissions reduction projects carried out by California workers and businesses. In addition, CARB’s own funding guidelines from 2015, the first year it published these guidelines for cap‑and‑trade‑funded programs, direct it to design programs that result in jobs and job training whenever possible. CARB’s 2018 updated funding guidelines expanded on this requirement, specifying that programs’ job creation and training efforts should target disadvantaged and low‑income communities and households. Both sets of guidelines include elements for measuring and reporting on the programs’ impact on jobs. The 2015 guidelines specify that CARB should track and report the number of job recipients and trainees funded by its programs, and the 2018 guidelines refined the required reporting to direct the external entities administering CARB’s programs to use a specific job calculator tool to estimate the number of jobs each program supports more broadly.

Despite clear direction and a stated requirement to estimate the number of jobs its programs support, CARB has failed to do so in a timely or comprehensive way. CARB began collecting data in 2019 for only some of its programs about the numbers of jobs they fund and support, and it still does not do so for other programs. Specifically, of the nine programs we reviewed for which jobs reporting requirements apply, CARB had collected jobs data in its reporting database for only three at the time of our review. Further, although it published estimates for some of its incentive programs in its fiscal year 2019–20 funding plan, as of its most recent annual report to the Legislature in March 2020 CARB had not reported any jobs creation numbers for its individual programs. Instead, CARB reported the aggregate jobs numbers for all cap‑and‑trade programs across multiple agencies, numbers that our review of program data showed were incomplete.

When we asked about its general lack of jobs reporting, CARB staff referenced the difficulty of establishing and implementing the measurement process. Staff also indicated that they have been unable to require some program administrators to follow the jobs reporting requirement because CARB entered into multiyear agreements with those program administrators before developing the reporting requirement. We find these explanations unconvincing, given that CARB has had since 2015 to implement the funding guidelines and according to a literature review, the jobs calculation tool CARB requires its program administrators to use is based on a method that is relatively simple and widely used for such analyses. We also note that CARB could seek to amend its grant agreements or, failing that, use its own tool to determine jobs benefits rather than relying on its program administrators to do so. Finally, CARB explained that staff had collected jobs data for two of the programs we reviewed where those data were missing, but they had neglected to enter them into CARB’s reporting database.

CARB also has requirements for providing job training benefits and reporting on them. As we discuss above, CARB’s funding guidelines require it to foster job training for its cap‑and‑trade‑funded programs and to focus its training efforts on disadvantaged and low‑income communities and households. The guidelines further specify that CARB should report information on employment outcomes from programs that provide job training, including information on the quantity and quality of the job trainings as well as the wages participants are paid and any credentials they receive as a result of the training activities.

Although the jobs training requirements in CARB’s funding guidelines apply whenever possible, we understand that these types of benefits may be more relevant for some programs than for others. Specifically, it might be reasonable for a consumer‑focused rebate program like CVRP not to have an explicit job training component. In contrast, pilot programs focused on industry and demonstrating advanced clean technologies have a natural overlap with job training. For example, as we discuss in Chapter 1, CARB operates a bus pilot program that helps transit agencies and other entities acquire zero‑emission buses and associated infrastructure. The program is focused in disadvantaged communities, and its goal is to place large numbers of zero‑emission buses in service while also providing opportunities for on‑the‑job training for bus operators and maintenance workers to learn the new technology. Among the 10 incentive programs we reviewed, CARB includes a job training element for only four programs. The fact that job training benefits may be relevant to only certain programs underscores the importance of tracking those benefits when they are provided.

When CARB does include job training as a program benefit, it does not always require its program administrators to provide detailed information about that training or how it benefits participants. For example, CARB operates a freight program that provides grant funding to demonstrate advanced emissions‑reducing technologies for moving freight within and through California and to encourage industry’s development of those technologies. CARB’s grant agreements with the program administrators require them to provide job training. The grant agreements include training materials and completed training rosters as required deliverables but do not require the program administrators to provide any information to CARB about the outcomes of the trainings for participants, the quality of trainings, or whether participants come from the communities CARB is trying to serve. For the two pilots we reviewed under this program, one of the program administrators reported to CARB that trainings were conducted and provided CARB with the training manual as evidence that the trainings occurred. However, the program administrator provided no further information, such as wages paid to program participants, the training credentials they received, or where they live. At the time of our reporting, the other program administrator reported it had not yet provided any training as part of the program.

Similarly, one of CARB’s pilot programs provides grant funding to enable transit agencies and other entities to acquire advanced zero‑emission buses and associated infrastructure. We reviewed three of the five bus pilots CARB operates under its bus pilot program and found that the grant agreements for these three pilots include general job training requirements and require the external program administrators to report to CARB the date the training was completed. However, similar to the freight program, the grant agreements for the three bus pilots do not include specific requirements for reporting on the quality of trainings, how the training benefits the participants, or where the participants live.

Notably, we identified one pilot program that has more detailed job training and reporting requirements, but those requirements apparently were included because additional funding was provided. Specifically, CARB and the California Energy Commission (energy commission) are planning to jointly operate a zero‑emission truck pilot program that includes funding from both entities. The solicitation documents for the program indicate that the program administrators will be responsible for developing and implementing a workforce training and development plan as part of the program, as well as for including specific data about the training and those who participate. However, the funds supporting these job trainings and detailed reporting requirements are from the energy commission, not CARB’s cap‑and‑trade funding. In fact, when we asked CARB about the job training for this truck pilot program, it indicated that the program includes these requirements because the energy commission had specific funds to invest in workforce development. Although the energy commission does allocate funds for job training, CARB’s own funding guidelines have, since at least 2015, directed it to include job training as a component of funded projects wherever possible. Therefore, we question why CARB has not included similarly detailed requirements for its own programs to ensure that they meet its own funding guidelines and provide these important benefits.

Recommendations

To ensure that it communicates clearly to the Legislature about the extent to which programs benefit low‑income households as the Legislature intended, by March 2022 CARB should begin reporting its spending in low‑income communities at the household level wherever possible in its annual report to the Legislature.

To better define incentive programs’ impact beyond GHG emissions reductions, by August 2021 CARB should review its incentive programs to ensure that it has clearly designated which programs focus primarily on socioeconomic benefits. As a result of this process, by February 2022 it should ensure that it includes the benefits expected for each program in its funding plan or other public documents, such as its annual report and individual grant agreements.

To better demonstrate the socioeconomic benefits that its incentive programs achieve, by February 2022 CARB should do the following:

• Identify clear and measurable metrics it will use to assess each of the socioeconomic benefits it intends its programs to achieve.
• Develop a process to collect data, or use existing data, to measure and report on each metric.
• In its funding plans and annual reports, CARB should report to the Legislature and its board on the metrics.

To provide transparency to the Legislature and other stakeholders, beginning in 2022 and using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost.

To ensure that the State has reliable information about the extent to which cap‑and‑trade‑funded programs create and support jobs, by August 2021 CARB should begin collecting data on the jobs produced by each of its incentive programs. Where needed, CARB should pursue amendments to its agreements with its program administrators to make reporting this information mandatory. CARB should include an analysis of these jobs data in its annual reports to the Legislature beginning in 2022.

To ensure that its incentive programs promote effective and equitable job training, by August 2021 CARB should develop a process to assess which programs should include a job training element. For those programs it identifies, by February 2022 CARB should direct its staff or its external program administrators to collect and report on the quality of job trainings and outcomes experienced by participants, including who received training, the credentials participants received as a result, any actual or expected wages they received as a result of participating in the training or for developing the relevant expertise, and the number of participants from disadvantaged communities or low‑income communities and households.

Other Areas We Reviewed

To address the audit objectives approved by the Joint Legislative Audit Committee (Audit Committee), we also reviewed the subject areas discussed below. These areas include the risk that regulatory programs may shift GHG emissions outside of California (as opposed to reducing them overall), CARB’s economic analyses for its proposed regulatory programs, and CARB’s tracking of the costs it incurs to administer its transportation programs. Below, we provide the results of our review and present an associated recommendation that we have not already discussed in other sections of this report.

Transportation Programs Generally Do Not Risk Shifting Emissions Elsewhere

The Audit Committee directed us to examine whether the reduction of GHG emissions resulting from a selection of CARB’s transportation programs were the result of shifting emissions rather than eliminating them. Such shifting may occur if regulatory requirements cause economic activity to move out of state. In such a scenario, there may appear to be a reduction in emissions, but the emissions have just moved elsewhere. We engaged our consultant to help us determine the extent to which emissions reductions from CARB’s programs may result from shifting emissions elsewhere rather than eliminating them. Our consultant concluded that the transportation‑related industries regulated by the programs we reviewed are not likely to shift emissions materially. For example, the regulatory programs generally focus on vehicles sold or operating in California, and individuals and businesses cannot easily shift their use of transportation out of state. As such, even when California’s vehicle requirements are more stringent than those in other states, manufacturers cannot avoid the restrictions by making the vehicles elsewhere. Further, for sectors that could experience a shift in activities from California to another location depending on the magnitude of regulatory costs—such as services provided by ports—our consultant identified that CARB has acknowledged these regulatory costs and has adopted certain programs to address them, which may limit the potential shifting of emissions.

CARB Reasonably Projects the Economic Impacts of Its Regulatory Programs, but It Could Do More to Explain the Effects on Consumers

The Audit Committee directed us to identify the cost‑effectiveness of the transportation programs we selected for review. Our consultant evaluated the economic analyses that CARB conducted when it proposed the eight regulatory programs we reviewed. Our consultant concluded that CARB reasonably approached its assessment of the compliance costs and resulting cost‑effectiveness of all eight of these regulatory programs reviewed. However, our consultant also found that CARB may understate costs in some circumstances and that it could expand its consideration of the costs and benefits to consumers. Specifically, CARB often concluded that the regulatory requirements in its programs, such as requiring the sale of ZEVs, represent net savings to consumers. In one specific analysis, CARB concluded that the long‑term cost savings for some vehicles were greater than the additional up‑front cost of those vehicles.

However, given its conclusion that these vehicles save consumers money, CARB could do more to consider all costs and benefits to purchasing a clean vehicle—such as driving range—and if, in fact, such vehicles are more desirable to consumers, explain why it is necessary to require sales of clean vehicles through its regulatory programs. CARB has not fully analyzed these issues and their implications for the regulation’s rationale and design. We recommend in Chapter 1 of this report that CARB develop a process to define, collect, and evaluate information on what factors influence consumers’ purchasing behavior as a key part of measuring the effectiveness of CARB’s incentive programs. Doing so would also allow CARB to better speak to the necessity of its regulatory programs and the effectiveness of particular designs it proposes.

Tracking Administrative Costs Would Allow CARB to Better Assess Its Programs’ Cost‑Effectiveness

The Audit Committee directed us to assess, in part, how CARB considers the effects of a proposed program on its administrative costs. For the eight regulatory programs we reviewed, CARB identified its expected administrative costs for the programs when proposing them. However, CARB did not consistently identify its own administrative costs when proposing funding for the 10 incentive programs we reviewed. Some managers explained that they do not estimate administrative costs because they consider overseeing incentive programs to be a part of CARB’s everyday responsibilities or because CARB did not know how much it would cost to administer the programs. However, it is clear that the addition of new programs for CARB to oversee represents new costs; therefore, CARB should include those costs in its estimates of the expenses to operate the programs once implemented. Our review determined that CARB also does not track actual administrative costs for its individual incentive programs. As a result, although CARB has spent approximately $120 million in cap‑and‑trade funds to administer its programs since 2013, it could not tell us what its administrative costs have been for individual programs. CARB explained that staff may work on more than one program at a time and indicated that it does not yet have a means of tracking staff time or other costs across all programs. However, because these costs are a potentially significant component of the total costs required to operate a given program, CARB should take all reasonable steps to measure them. Moreover, it is a basic and often required task for state agencies to track their costs related to specific programs and activities, and we see no reason why CARB would be unable to do so.

Recommendation

To ensure that it can account for the total costs of its transportation programs, beginning with fiscal year 2021–22 CARB should develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs. After doing so, it should begin including those costs as part of the cost‑effectiveness measurements in its annual reports to the Legislature.

We conducted this performance audit in accordance with generally accepted government auditing standards and under the authority vested in the California State Auditor by Government Code 8543 et seq. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on the audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Respectfully submitted,

ELAINE M. HOWLE, CPA
California State Auditor

February 23, 2021

---

Back to top