COMMITMENT • INTEGRITY • LEADERSHIP	View full PDF report View Selected Abbreviations
California High‑Speed Rail Authority Its Flawed Decision Making and Poor Contract Management Have Contributed to Billions in Cost Overruns and Delays in the System's Construction Report Number: 2018-108

CHAPTER 2

THE AUTHORITY HAS NOT SUCCESSFULLY ENFORCED THE POLICIES IT IMPLEMENTED TO ADDRESS ONGOING DEFICIENCIES WITH ITS CONTRACT MANAGEMENT

Chapter Summary

Although the Authority recently adopted contract management policies to help ensure that it monitors its contracts and controls costs, it lacks an effective organizational structure to implement them. Under its current structure, the Authority's contract managers experience high rates of turnover and receive little oversight. Further, they generally serve as contract managers in addition to their other professional responsibilities. Although the RDP consultants assist in contract management, they may not always have the State's best interests as their primary motivation.

Likely in part as a result of these weaknesses in the Authority's contract management structure, we identified significant problems in its management of its contracts. We reviewed a selection of engineering and other service contracts to assess the Authority's compliance with its policies and procedures most directly relevant to controlling costs and ensuring value: invoice review, deliverables monitoring, and change management. We found that although the contract managers complied to varying degrees with the invoice review procedures, they consistently did not document the receipt or evaluation of contractor deliverables, nor did they independently evaluate the need for contract changes that added cost and time. In fact, the contract managers' lack of documented, independent review prevented us from reaching conclusions about the Authority's effectiveness in assessing the quality, timeliness, or cost of the work performed under these contracts. Without such documentation, the Authority cannot demonstrate that the hundreds of millions of dollars it has spent to date on the selected contracts—including for cost overruns in the form of amendments—has been necessary or appropriate.

Similarly, our review of the Authority's construction contracts found that it has implemented a construction invoicing process capable of significantly limiting the risk that it overpays these contractors for the work they perform. However, the Authority has not provided reliable monitoring of the oversight firms that are responsible for managing this invoicing process. Further, it has only recently developed formal monitoring to evaluate the performance of the oversight firms.

The Authority Has Not Established an Effective Contract Management Structure

As we discuss in the Introduction, the Authority adopted policies related to contract management and oversight in April 2017. However, these policies will only prove effective if Authority staff follow them, and the Authority has yet to create a contract management structure that adequately ensures such adherence. Instead, its contract managers experience high turnover and receive little oversight. Moreover, weaknesses in the Authority's contract management structure have contributed to its reliance on contractors for important functions—such as contract oversight—that state employees should perform.

The Authority’s Contract Managers Experience High Turnover and Receive Little Oversight

The Authority's contract managers' official responsibilities do not always specify contract management duties. As of September 2018, the Authority's 56 contract managers were collectively responsible for 204 contracts with values totaling $5.6 billion. However, according to the contract administration manager, only three of those 56 contract managers serve in contract management roles full‑time. Although records from the Contract Management Support Unit (CMSU) show that each of the eight contract managers responsible for the contracts we reviewed completed the Authority's required contract management training, our review of these individuals' duty statements found that only three specifically mentioned contract management duties.⁴ Further, only one of the individuals mentioned in this chapter carries the job title of contract manager. Although we refer to them all as such for the purposes of their responsibilities, their actual titles include administrator, engineer, and executive.

The former manager of CMSU acknowledged that the Authority has not established a formal practice for selecting contract managers and assigning them to contracts. Perhaps as a result of that fact, the Authority frequently changes the individuals responsible for managing each contract, resulting in high rates of turnover among its contract managers. For the nine contracts we reviewed, CMSU's roster of contract managers shows that five contracts had two or three different contract managers in the past year alone. In fact, the contract managers for three of the contracts changed during the period of our review from March through June 2018. High turnover has not only likely contributed to noncompliance with policies, but it also underscores the need for strong documentation practices. For example, one new contract manager told us she received a transfer form from the previous contract manager indicating that all the required contract documentation was available and up to date. However, she did not sign the form because much of the documentation listed on the form was not actually available.

The responsibility for ensuring that contract managers perform required tasks lies with their direct supervisors, who themselves are not consistently trained in the Authority's contract management policies and procedures. Six supervisors managed the eight contract managers in our review, and half of them had not received training on the Authority's contract management policies and procedures at the time of our review. If they do not understand the Authority's specific requirements for contract management, supervisors may not be able to intervene effectively when contract concerns arise or to provide strong oversight to ensure contract managers adhere to policies and procedures. Additionally, these supervisors also have full‑time responsibilities unrelated to contract management. In fact, some of the supervisors are in executive leadership positions, including the chief executive officer and the chief financial officer (CFO), each of whom oversees multiple contract managers in addition to their responsibilities for large segments of the Authority's operations.

The Authority established CMSU within the Contract Administration Branch in part to oversee compliance with contract management policies and procedures, but its oversight has been weak and inconsistent. The extent of its oversight activities to date has been a fall 2017 review of whether contract managers had filed documentation in the locations and structure that the Authority's policies and procedures required. Although this review consistently found that contract managers had not filed required tracking logs as expected, CMSU did not take any additional steps to determine whether the contract managers were actually using those logs or to verify compliance with any other policy requirements. If CMSU had performed such additional reviews, it could have identified some of the more significant compliance issues we discuss later in this chapter.

In response to our concerns, the CFO asserted that the current set of contract management policies and procedures took significant effort to develop, as did developing and conducting the Authority's contract management training. However, he also acknowledged that the Authority has not yet taken additional steps to ensure compliance. Similarly, the director of the Contract Administration Branch (contracts director) asserted that only a short time has elapsed since the implementation of the Authority's new contract management policies and procedures and that the branch is relatively new. Given that these policies and procedures have been in effect for over a year and that the contract managers signed forms pledging to comply with them, we believe ample time has passed for the Authority to have conducted meaningful oversight. The CFO informed us that the Authority intended for CMSU to conduct contract manager compliance assessments and submit the results of these assessments to the contract managers' supervisors, as the Authority's policies and procedures specify. However, he confirmed that CMSU has not conducted these assessments and thus has not implemented the process of notifying supervisors of any noncompliance.

The CFO further expressed his belief that timing and leadership transitions have contributed to the Authority's general challenges in ensuring its staff comply with its contract management policies and procedures. He stated that since adopting its contract management policies and procedures in April 2017, the Authority has directed its efforts toward creating two new executive positions—the chief deputy director and the chief operating officer—to oversee new offices that would more appropriately include contract management. In October 2018, the Authority adopted a project management plan that includes placing contract management under the direction of these new executive positions. However, the plan is not sufficiently detailed to address shortcomings in the Authority's current contract management related to defining contract managers' formal duties and ensuring enforcement of those duties. We also do not agree that the time spent rearranging the high‑level organization of contract management supplants the responsibility to simultaneously strengthen the existing system. If the Authority believes, as we do, that strong and accountable contract management is key to controlling the system's costs, it must commit to fully implementing and enforcing its contract management policies and procedures.

Weaknesses in the Authority’s Contract Management Structure Have Likely Contributed to Its Overreliance on Contractors

The Authority's inadequate enforcement of its contract management policies and procedures may encourage its reliance on contractors to perform important functions, further hindering its ability to control costs. As we noted in our 2012 audit report (Chapter 2, page 28, paragraph 1), the Authority's organizational structure places large portions of its program planning, construction, and oversight in the hands of the RDP consultants, who may not have the best interests of the State as their primary motivation. Further, as we discuss in the Introduction, the Authority's internal audits concluded that roles and responsibilities for contract managers and RDP consultants were not clearly defined. To address this issue, the Authority's contract management policies and procedures clearly assign contract managers the responsibility for tracking and monitoring all aspects of the contracts they manage. Nonetheless, we observed that the contract managers for the regional planning contracts—which are for preliminary engineering and environmental work in locations where the Authority plans to develop the high‑speed rail system—still often rely on the RDP consultants to provide the oversight for which the contract managers are ultimately responsible. In fact, during our review, the Authority's contract managers for the regional planning contracts directed our contract management questions to the RDP consultants for answers and were generally unable to provide documentation related to contract management that did not originate from the RDP consultants. As a result, the RDP consultants have become the de facto contract management body, working closely with contractors with insufficient Authority oversight.

Further, the Authority has also placed the oversight responsibility for contract management with the RDP consultants, which creates a potential conflict of interest. Specifically, although an Authority employee heads CMSU, the RDP consultants fill its seven positions. When the Authority's contract managers inappropriately rely on the RDP consultants to perform their contract management responsibilities, it may not be reasonable to expect CMSU staff—who are also RDP consultants—to tell state contract managers to stop this practice. Consequentially, CMSU's current composition raises questions about the Authority's ability to use the unit as a tool to prevent the Authority's continued overreliance on the RDP consultants to perform contract management, which we believe should be among CMSU's priorities.

The Authority tasked contractors with duties that state employees could have performed in other instances as well. For example, the Authority's documentation for its $40 million contract for financial advisory services states that contracting for those services is justified because the tasks are of a highly technical nature and equivalent expertise is unavailable within state civil service. However, in December 2016, the Authority's former chief executive officer (CEO) sent the CFO an email in which he expressed concern over high spending rates for the contract, particularly in the areas of accounting support and budgets. In his email, the CEO stated that contract spending should focus on nontraditional areas of work, such as financing analyses and commercial and real estate strategies, and that employing state staff to perform basic budgeting and accounting work would be more appropriate and cost‑effective.

The CFO responded to the CEO's concerns by stating that a core group of state staff provided services for budgeting and accounting, but that the financial advisory contractors were needed to help perform responsibilities that had no precedent in state service, including implementing information technology systems. The CFO reiterated this position to us during our audit. However, our review of the contract's work plans and invoice materials determined that, although some of the contractor's duties included information technology tasks, many of the tasks it reported performing were described as support for general budgeting and accounting activities. Given the fact that the contractor billed the Authority $3.5 million in fiscal year 2016–17 and $1.1 million in fiscal year 2017–18 for the budgeting and accounting portions of the contract, we believe the Authority should have taken steps to ensure and document that its use of contracted resources was necessary and prudent.

The Authority also assigned work related to contract management to outside contractors that may have been more appropriately performed by state employees. For example, the Authority tasked its financial advisory contractor to perform analysis and support for its Contract Administration Branch from July 2016 through June 2018. As part of this work, the contractor developed the contract management policies we reviewed during this audit. The CFO explained that these policies and procedures took significant effort because no equivalent state criteria directly apply to the Authority. However, we do not believe that the Authority is so unique in its contract management needs that state resources could not develop similarly adequate tools. Further, the Authority also tasked the contractor with monitoring and reporting on the status of the RDP consultants' deliverables and with supporting CMSU in its compliance reviews of the Authority's contract managers. As we discuss above, these compliance reviews have been insufficient. We do not believe that any of these tasks are so highly technical or specialized that state employees could not have performed them. Nonetheless, the financial advisory contractor billed the Authority almost $4 million for these tasks over two years.

Finally, just as the Authority staffed CMSU with RDP consultants to oversee contract manager compliance, it also staffed its separate administrative unit for supporting contracts entirely with RDP consultants, who filled all 17 positions as of June 2018. We question why using RDP consultants in place of state employees to perform contract management oversight and support is necessary. The Authority agreed that it should place state employees in these positions in the future.

The Authority Has Not Ensured That Its Contract Managers Actively Manage Expenditures and Deliverables in Compliance With Its Requirements

We found that the contract managers generally complied with the Authority's documentation requirements for reviewing invoices but did not comply with the procedures for documenting timely and thorough review of deliverables. Figure 9 illustrates that although contract managers were often able to provide us documentation to demonstrate their compliance with invoice requirements for using a tracking log and completing an invoice approval checklist, they could not consistently demonstrate how they validated invoiced costs to ensure they were appropriate and allowable. The contract managers were also generally unable to demonstrate their review of deliverables or their efforts to monitor contractor performance. This lack of documented, independent review prevented us from reaching conclusions about the quality, timeliness, or cost of work performed under these contracts. Further, when we did identify references to concerns with contractors' work products, the lack of documentation meant that we were generally unable to determine how contract managers identified or resolved such issues. Without clear documentation that its contract managers ensured deliverables were consistent with requirements before approving payments or that they appropriately monitored contractor performance, the Authority cannot demonstrate that the hundreds of millions of dollars it has spent to date on those contracts—including for cost overruns—has been necessary and appropriate.

Figure 9
Contract Managers Failed to Consistently Document Completion of Tasks Necessary to Control Costs and Ensure Value

Source: Analysis of the Authority's available contract management documentation.

The Contract Managers Could Not Consistently Demonstrate That They Performed Thorough Reviews of Contractors’ Invoices

In order to assess the Authority's compliance with its own policies and procedures, we reviewed nine active contracts overseen by eight contract managers. The contracts we reviewed include planning, engineering, consulting, and construction oversight services and have a total value of $1.3 billion.⁵ The services on these contracts include a combination of discrete deliverables and day‑to‑day work involving on‑site staff support. Our review focused on the Authority's compliance with those contract management policies and procedures most closely linked to controlling costs and ensuring value, including invoice and deliverables review.

The Authority has established procedures requiring a systematic approach to its contract managers' receipt and handling of invoices, appropriate invoice documentation and review, and required approvals for timely processing of payments. For example, Authority contract managers must ensure that contractors bill hourly rates appropriately, that all direct costs are eligible and supported with receipts, and that the totals of each invoice are calculated correctly. In addition, the Authority's procedures require contract managers to complete an approval checklist for each invoice for contracts with values greater than $5 million. These checklists require contract managers to attest to the accuracy of the invoices and the sufficiency of supporting documentation by checking boxes stating that the invoices comply with relevant contract provisions and that all expenses are eligible and supported with receipts. The procedures also require contract managers to keep logs that track invoiced and approved amounts, key dates, and any disputed costs.

When we examined a selection of invoice approval checklists for the nine contracts we reviewed, we found that the contract managers completed checklists indicating that they evaluated each invoice for accuracy. However, the contract managers for only three of the nine contracts could provide documentation to support the checklists' assertions that they ensured invoiced rates and expenses were allowable. For example, one contract manager provided us with a review spreadsheet in which, in addition to monitoring monthly spending against the contract's value, he tabulated each individual billing rate, the hours reported by task, and all direct expenses. The manager of the Contract Administration Support Unit told us that contract managers can request support from this unit to verify that contractors bill hours against allowable rates, but only one of the contract managers in our review used this resource.

We expected to see documentation from contract managers verifying that costs were allowable and accurate because most of the invoices we reviewed were complex; they included many individual pay rates for contractor staff, tens or hundreds of thousands of dollars in direct costs that the contract managers had to verify, and many subcontractor costs. When it relies primarily on the contract managers' attestations on the invoice approval checklists, the Authority risks that those contract managers may not have performed thorough and comprehensive invoice reviews. As a result, the Authority could pay for unauthorized costs. We observed one such instance in which an internal audit that the Authority issued in June 2017 identified that it had paid for more than $1.2 million in questioned and disallowed costs to one of its contractors—costs that either were not supported by adequate documentation or were not in compliance with the contract terms or applicable rules and regulations.

Similarly, although the contract managers we reviewed consistently used required invoice tracking logs, these logs had limited utility. Specifically, the invoice tracking logs serve as a record of the amounts that the contractors bill and the Authority approves each month. When the contract managers enter approved invoice amounts, the tracking logs automatically calculate the remaining contract balance. Although they help ensure that the Authority does not overspend the contract balance, the tracking logs do not demonstrate detailed invoice review any more than the invoice checklists. More importantly, none of the required invoice documents demonstrate whether or how the contract managers performed the critical task of determining that invoiced costs were appropriate for the amount and quality of the contractors' work.

The Authority Lacks Evidence Supporting the Quality, Timeliness, and Cost of Contract Deliverables

Most of the managers for the contracts we reviewed asserted that when they approved invoices, they reviewed narratives summarizing the work performed. However, these narratives are generated by the contractors and do not serve to independently verify that the contractors have, in fact, performed the work to the standards of the contracts. In establishing its 2017 policies, the Authority acknowledged the limitations of this invoice review process by creating a parallel but distinct process through which it requires contract managers to identify, document, track, receive, review, and accept contract deliverables.

The Authority's policies and procedures clearly state that this process—known as deliverables management—is meant to ensure the Authority is able to construct the high‑speed rail system on schedule and within budget. The text box describes the Authority's procedural requirements for managing deliverables and identifies the documentation required to track that process. The fundamental purpose of these tracking activities is ensuring that the contractors' deliverables are consistent with quality, timeliness, and cost requirements.

Despite the specificity of the Authority's requirements related to deliverables, the contract managers we reviewed did not comply with the required actions. None of the contract managers for the nine contracts we reviewed used the standardized deliverables tracking log—or any other document—to independently track the status and review of contract deliverables. Further, contract managers did not document their formal reviews of deliverables; instead, they described various alternative methods for these reviews. Some contract managers claimed to monitor contracted work simply by observing and working closely with contracted staff on a daily basis. Others relied on the RDP consultants to approve contractor‑generated progress reports and individual deliverables, after which the contract managers would approve payment. Still others stated they relied on other Authority staff and subject matter experts to evaluate and approve individual deliverables, although those delegated evaluations were also generally not documented. Though we recognize that subject matter experts play an important role in assessing the quality of work, their expertise does not supplant the responsibilities of the contract managers.

Finally, instead of using the Authority's required templates for tracking the receipt and sufficiency of deliverables, contract managers for four of the nine contracts we reviewed provided documents that they use to verify the extent to which deliverables were timely and adequately completed. However, we noted that the contractors prepared these documents. In fact, for nearly all the contracts we reviewed, the only documented source of information regarding the timeliness and status of deliverables came from the contractors themselves. As a result, we were generally unable to determine how the Authority independently ensured it received the deliverables for which it paid and that they were of the quality that it required.

The Authority Paid for Deliverables Without Performing Proper Review

Our review of nine contracts also found that the contract managers rarely had documentation for their acceptance of deliverables. In fact, the contract managers for seven of the nine contracts did not have any acceptance notices on file. Authority policies and procedures require contract managers to review deliverables for compliance with contract requirements and the Authority's acceptance criteria. The contract managers must then officially accept the deliverables in writing. This documented communication marks the point in time when the Authority takes care, custody, and control of the deliverable.

Despite this requirement, when we asked the contract manager for an environmental contract about the missing acceptance notices, he stated that the contract had only two real deliverables, one of which is the environmental impact report. He stated that he planned to evaluate that report upon its completion and that he would issue the acceptance notice at that time. However, the contract documentation makes clear that the contractor is responsible for many individual work products as part of developing this larger report. When we asked about these work products, the contract manager stated that instead of reviewing deliverables as the contractor submits them, he plans to later issue a memorandum to summarize all deliverables. We received similar responses from other contract managers, who stated that acceptance notices were not applicable because the contracts they managed did not include what they considered to be formal deliverables. However, we disagree with this assertion; for each of these contracts, we identified evidence of discrete work products—or deliverables—that the Authority policies and procedures would require contract managers to evaluate and formally accept.

In fact, only two of the nine contracts had acceptance notices on file that documented the timing of deliverable submission and review, and this documentation was minimal and used inconsistently. When we requested acceptance notices for one of the two contracts—a four‑year, $40 million contract for financial advising services—the contract manager provided nine notices, all of which he signed and dated June 11, 2018—after the date of our request on June 4. Further, none of the notices contained any comments in the template fields that contract managers can use to record the details of their reviews. When we subsequently followed up to ask about additional deliverables for which the contract manager had not provided acceptance notices, he produced another three notices, each of which he signed on July 9, 2018—again, after our request. In some instances, the contractor had submitted the deliverables in question as early as October 2017. Because the Authority pays the contractor for its work on a monthly basis, even if the contract manager had performed a detailed review in June and July 2018 and had found issues with the deliverables, the Authority had already long since paid the contractor for the work.

When we asked the contract manager about his signing the nine notices on the same day, he asserted that the process was for the contractor to submit the acceptance notices to him at the end of the fiscal year, rather than with the deliverables. However, as we state above, he provided the original set of acceptance notices before the end of the fiscal year, and soon after we requested them. Further, because the acceptance notices are a part of the Authority's policies and procedures, we would expect the contract manager to initiate the process, not the contractor. When we expressed concern about the contract manager's approach, he replied that he believed the process was sufficient because he would complete the notices before the contract was closed. However, if the Authority waits until the contract's end to review deliverables, it will have already substantially paid for these deliverables. When we asked about any other evidence to demonstrate his review of deliverables, the contract manager provided emails intended to demonstrate his review. Our review of those emails found that the contract manager was not included on many of them, some of which were between contractors only. The contract manager told us that going forward, he plans to complete an acceptance notice contemporaneously with receipt of each deliverable. To the extent that the contract manager also performs and documents his comprehensive review of each deliverable at this time, we agree it is appropriate for him to complete the process by signing and issuing acceptance notices.

We also identified significant concerns with the Authority's collection and evaluation of deliverables for its $666 million RDP consulting contract, which includes tasks in 30 subject matter areas. In late 2017, the contract manager tasked a contractor—from a firm other than the RDP consultants—with conducting an assessment of the deliverables and corresponding acceptance notices that the Authority had on file dating back to the start of the RDP contract in July 2015. To conduct this assessment, the contractor compared the RDP consultants' self‑reported information about the deliverables it had submitted to the Authority to the deliverables the Authority actually had on file. The review determined that for the work plans that were active when the Authority's contract management policies and procedures went into effect, the Authority was missing formal acceptance notices for 70 of the 80 deliverables that the RDP consultants reported as complete. Further, the Authority did not have 145 of the 184 deliverables that the RDP consultants had reported as having completed since the beginning of the contract. These missing deliverables ranged from engineering documents to software updates to white papers and other strategic documents. Nonetheless, from July 2015 through December 2017, the Authority paid the RDP consultants over $200 million for the tasks that included these deliverables.

When we asked about the Authority's efforts to follow up on the missing and unapproved deliverables, the contract manager told us that because of funding constraints, the non‑RDP contractor was no longer working on the deliverables issue. However, he stated that he had requested help from other Authority staff in identifying and recovering the deliverables. Nonetheless, as of July 2018, he had not yet received any new information. Without documentation of formal review and approval, the Authority cannot demonstrate that it received the quantity and quality of work for which it paid the RDP consultants.

Moreover, the Authority's tracking and evaluation of this contract's deliverables has continued to be an issue. In July 2018, the contract manager provided us a list of deliverables that the RDP consultants had reported submitting, which he originally represented as an updated tracking log. However, the contract manager later stated that because of the aforementioned funding constraints and a lack of support staff, he had not had the opportunity to verify the submission and timing of the deliverables on this list. He asserted that subject matter experts, who are state employees, are involved in developing the monthly status reports that the RDP consultants submit to the Authority, which include the status of deliverables. However, he acknowledged at the time that he had not reached out to subject matter experts to collect the deliverables that the RDP consultants had reported as complete and therefore had not yet completed deliverable reviews in order to issue acceptance notices. Despite having confirmed this lack of acceptance notices on multiple occasions, the contract manager informed us in October 2018 that he did in fact have completed acceptance notices for some of the RDP consultants' deliverables. He then provided 77 signed acceptance notices dated as far back as January 2018.

These acceptance notices do not alleviate the need for detailed deliverable tracking and review documentation. The acceptance notices on their own do not allow the contract manager to determine whether the RDP consultants' work is generally on schedule. Along with the acceptance notices, the contract manager also provided a log his staff began compiling in August 2018 to track the status of deliverable acceptance. If used going forward, the log will help the contract manager more proactively track the status of all deliverables. However, even though the contract manager asserted that the acceptance notices are the formal deliverable review documents, neither the acceptance notices that he provided nor the log contain detail about how the contract manager determined that deliverables met contract requirements.

Given the large dollar amount associated with this contract, it is crucial that the Authority improve its practices for tracking the status of deliverables and reviewing them for quality.

Because of the Authority’s Failure to Track Deliverables, Determining How It Resolved Quality Issues Is Difficult, If Not Impossible

The contract managers' insufficient tracking of deliverables also means that when the contract documentation alluded to issues with contractors' work products, we generally could not determine how the Authority identified the issues or confirm that they were resolved appropriately. When contractor deliverables are unsatisfactory, a contract manager must not only identify the unsatisfactory work, but also ensure the Authority does not pay for the hours spent to fix it. For example, in September 2017, a regional planning contractor in Southern California had to revise its design for a train station access road because it adversely affected a historical bridge. The Authority considered the hours spent revising the design as repeated work for which it had already paid. Although the contract documentation indicated that the Authority contract manager formally disputed the invoice that included charges for this work, it provided no further information. When we asked for details about the dispute, the contract manager stated that the RDP consultants were responsible for identifying the issue and determining the total amount that the Authority should not pay. However, neither the contract manager nor the RDP consultants documented how they ensured that they accounted—and therefore avoided paying—for the contractor's repeated work.

In another example from January 2018, an RDP consultant expressed concern to his RDP supervisor about the quality and timeliness of a contractor's deliverables for a Northern California environmental planning project. When we asked how the issue was resolved, the RDP consultant stated that he did not dispute the related invoice because the contractor did not claim the deliverables for payment because of their deficiencies. To demonstrate this, the consultant provided an invoice from the contractor that showed hours worked for which the contractor did not bill the Authority. However, because of the lack of detail in the RDP consultant's tracking documentation and the fact that the contract manager had no additional documentation, we could not substantiate that the hours on the invoice accounted for all of the repeated work the contractor performed.

Both of these examples also demonstrate the Authority's overreliance on the RDP consultants to provide oversight of certain contracts. In the first example regarding the train station access road, the available documentation about the origin of the issue was limited to communications among the RDP consultants, and we identified no evidence that the contract manager took steps to independently evaluate the situation. Similarly, in the second example, the contract manager for the environmental planning contract asserted that because his background is engineering, he relies on the RDP consultants to recommend when the Authority should dispute invoices for environmental work. However, as of April 2018, the contract manager was still not aware of the issues the RDP consultants discovered in January 2018. Instead, the contract manager believed that the contractor had not had to repeat work on the contract.

Despite these and other quality concerns, the contract manager for only one of the nine contracts we reviewed requested that a contractor submit a recovery plan—a document Authority policy directs contract managers to request if deliverables have fallen behind, do not meet contract requirements, or may require repeated work. Specifically, the contract manager for the RDP consulting contract requested a recovery plan in December 2017 for a single deliverable for the development of cost management software. This deliverable had delays spanning multiple years. The contract manager communicated his expectations for the recovery plan in a formal letter to the RDP consultants, stating that he expected the plan to include a schedule with detailed implementation activities, a list of all remaining project scope items and necessary resources, and a detailed mitigation strategy should the RDP consultants miss any milestones. The contract manager's request was consistent with Authority contract management policies and procedures. However, because this example is the Authority's only use of a recovery plan for any of the nine contracts we reviewed, we are concerned that the Authority may have missed other opportunities to address untimely or unsatisfactory deliverables.

In fact, the RDP consultants missed deadlines for other deliverables for this same contract, suggesting problems may exist that the Authority has not actively tried to mitigate. The contract's current work plan was originally scheduled to be eight months, with the RDP consultants completing all deliverables by the end of February 2018. However, as of January 2018, the RDP consultants had reported submitting only 10 of 81 deliverables, despite spending nearly $70 million of the work plan's $90 million budget. According to the contract manager, the Authority extended this work plan twice, for a total of seven months, because the RDP consultants had not yet completed the deliverables. When it extended the work plan, the Authority also assigned additional deliverables and added funds to the work plan, now valued at $157 million.

Although the contract manager asserted that the Authority expects the RDP consultants to complete outstanding deliverables with no additional resources, the Authority added dollar amounts with the extensions that do not appear proportional to the additional deliverables it assigned. For example, the most recent extension added nearly $30 million but just eight new deliverables. As of October 2018, the RDP consultants reported to us that they had submitted 101 of the 111 deliverables due by the work plan's revised September 30, 2018 deadline. However, as we explain in the previous section, the contract manager provided acceptance notices for only 77 deliverables and has only recently begun proactively tracking the timeliness of the RDP consultants' work. Overall, the work plan's shifting deadlines and large dollar increases make us question why the Authority did not initiate the formal corrective actions that its policies and procedures indicate it should.

Given the types of challenges some of the contracts we reviewed have presented, we found the lack of formal intervention by the Authority concerning. Most of the contract managers, and even the RDP consultants who identified the quality issues that resulted in repeated work, insisted that there had been no need to establish recovery plans. Moreover, we identified evidence of formal invoice disputes for only two of the nine contracts we reviewed, and the disputed items generally had to do with incorrect billing rates or unsubstantiated costs for invoiced expenses, rather than deliverable quality. Some of the other contract managers explained the lack of formal intervention by asserting that when quality issues arose, they resolved them informally with the contractors. Although Authority policies allow for the informal resolution of issues as an alternative to formal documented disputes, the lack of documented deliverable review and tracking by contract managers, along with the deliverable delays we discussed above, creates the risk that the Authority is not detecting or resolving issues with contractor performance. Further, without the contract management documentation its policies and procedures require, the Authority cannot demonstrate that the hundreds of millions of dollars it has spent to date on these contracts—including for cost overruns—has been necessary and appropriate.

Although the Authority Has Amended Many of Its Contracts, Contract Managers Have Not Sufficiently Documented Attempts to Control Costs or the Reasons for Overruns

The Authority frequently amends its contracts to add additional time or funds. Of the nine contracts totaling more than $1.3 billion in our review, the Authority used amendments to increase the value of six by a total of $183 million and to extend the contract terms for five of those six. In addition, it amended two other contracts to reduce their value by nearly $40 million when reassigning tasks and funds to other contractors. Only one contract that we reviewed, for financial advisory services, has no amendments. As Table 3 shows, the amendments increasing the contracts' value represent significant additional costs. These amendments more than doubled the value of two contracts and increased the value of three others by more than 40 percent. Amendments have thus unquestionably contributed to the high‑speed rail system's cost overruns.

Table 3
The Authority Has Added Large Amendments to Many of the Contracts We Reviewed

CONTRACTOR/SERVICE	TOTAL VALUE OF AMENDMENTS	NUMBER OF AMENDMENTS	ORIGINAL CONTRACT VALUE	CURRENT CONTRACT VALUE	TOTAL TIME EXTENSION	ORIGINAL CONTRACT TERM
Caltrans State Road 99 construction	$64,200,000	3	$225,900,000	$290,100,000	—	February 2013– June 2020
Wong+Harris, Joint Venture Construction oversight firm	35,500,000	3	34,209,000	69,709,000	12 months	May 2013– December 2018
T.Y. Lin International Bakersfield to Palmdale regional consultant*	26,558,000	3	46,100,000	72,658,000	24 months	February 2014– January 2019
Parsons Transportation Group Central Valley Wye regional consultant*	25,640,000	6	55,000,000	80,640,000	63 months	December 2008– June 2014
Nossaman, LLP Legal services†	18,500,000	10	500,000	19,000,000	114 months	January 2009– June 2011
HNTB Corporation Construction oversight firm*	12,800,000	1	30,064,000	42,864,000	1 month	January 2016– November 2020
KPMG, LLP Financial advisors	—	0	400,000,000	40,000,000	—	June 2016– June 2020
WSP USA, Inc. RDP consultant‡	(33,630,000)	3	700,000,000	666,370,000	—	July 2015– June 2020
Arcadis US, Inc. Construction oversight firm‡	(6,000,000)	1	71,885,000	65,885,000	—	November 2014– April 2019
Totals	$143,568,000	30	$1,203,658,000	$1,347,226,000

The Authority designed its contract management policies and procedures, as well as its associated tracking requirements, to ensure that it identifies the need for changes in a timely manner and that it appropriately ensures those changes' justification before adopting them as amendments. For example, when an involved party identifies a potential need for a change, such as a change in the scope of the contract or in required deliverables, the policies and procedures require the contract manager to include the change in the change tracking log; document who identified the need for it; describe the issue or potential change; and identify its proposed cost, its impact to the contract schedule, and the relevant dates in the process. The policies and procedures also require the contract manager to assess the potential change for merit and discuss this determination with an appropriate supervisor for approval. The Authority may also provide amendment details in a staff report to the board, which during the period of our review had to approve any amendments to existing engineering and architectural contracts with values that exceed $5 million. The Authority presented all of the proposed amendments we reviewed to the board.

Despite these requirements, the contract managers for the contracts we reviewed could not consistently provide documentation demonstrating the Authority's independent evaluations of potential amendments. Since April 2017, when the Authority established its new policies and procedures, it has approved 13 amendments for eight of the nine contracts in our review. We reviewed three of these amendments, each for a different contract. However, when we tried to identify the details related to the amendments' necessity and size, we found that none of the contract managers had maintained change tracking logs. Further, when we asked for supporting documents with the information we expected to find in the tracking logs, the contract managers frequently provided the documentation that Authority staff had used to present the recommended amendments to the board. When we requested documentation showing how the Authority determined these three amendments had merit as the board materials attested, contract managers for two of the contracts had to request this documentation from the RDP consultants or from the contractors themselves. The Authority documented its analysis of the third amendment we reviewed, but that analysis was incomplete.

An example involving a $3 million amendment with a one‑year extension demonstrates why relying on contractor‑provided evidence is problematic. When we asked about this amendment, the contract manager first provided us with the formal high‑level amendment request that the former contract manager signed. According to Authority procedures, a contract manager should sign this form only after determining that the proposed amendment has merit. The contract manager also provided the staff report that the Authority presented to the board when the board considered the amendment. The staff report asserted that a number of unforeseen changes had impacted the scope, budget, and overall schedule of the contract's work, precipitating the need for the amendment. When we asked for evidence supporting the claims that the board documents made regarding the amendment's merit, the current contract manager provided additional documentation that the Authority received from an RDP consultant working with the contractor. This documentation showed that the contractor was the source of information regarding the amendment's necessity and accompanying costs, and it included no evidence that the Authority independently verified the contractor's claims. In fact, the content in the Authority's report to the board came directly from the contractor's amendment request. When it does not independently verify and document the need for contract amendments, the Authority risks authorizing additional funds for unnecessary or unwarranted changes.

Contract managers were also unable to demonstrate if and how they independently determined that the approved amendment costs were appropriate and justified. When we reviewed another amendment, we found that the contract manager had originally drafted an amendment request for $26.7 million based on the contractor's estimates for the cost of land acquisitions and utility relocations. The contractor later determined that it required an additional $2.5 million, causing the contract manager to increase this amendment proposal to $29.2 million, which the board ultimately approved. When we questioned how the Authority had evaluated the need to add the $2.5 million or the appropriateness of the $29.2 million total cost, the contract manager stated that the contractor had increased the amount of additional funds it claimed to need after further consideration and analysis. However, the contract manager was unable to provide documentation showing the Authority had independently determined the accuracy of the amendment's costs either before or after the contractor increased them.

For the third amendment we reviewed, the Authority documented a justification for why it required the amendment, but its analysis was incomplete. Specifically, when one of the oversight firms that manage the Authority's construction contracts was projected to deplete its contract funds nine months ahead of schedule, Authority staff requested that the board approve a third amendment for $28.5 million to retain the oversight firm's services for an additional year. The amendment documents the Authority presented to the board indicate that the Authority considered seeking a new vendor to replace the oversight firm, but it decided to recommend the amendment instead. The Authority's documents assert that the need for the amendment stemmed from its assigning work to the oversight firm that was outside of the original contract because of changes to the construction contract. However, the contract manager could not tell us how much money went to the oversight firm for performing out‑of‑scope work and acknowledged that he had not documented the oversight firm's adequate performance, as we discuss in more detail later. The board expressed concerns about substantial cost increases, stating that the Authority generally had some issues in the past with oversight of construction management, and it directed Authority staff to return in 90 days to update it on the oversight firm's progress in fulfilling the objectives of the contract. However, it also unanimously approved the amendment. When staff reported back, they presented the board with a template they planned to use to evaluate the three oversight firms' performance in the future.

Our analysis of the amendment documents indicates that future cost overruns may be likely for this contract. In the documents, Authority staff calculated that retaining the existing oversight firm was more cost‑effective because of its familiarity and experience with the project. In reaching this conclusion, the staff compared the amendment amount to what they indicated was the remaining value of the work still to be completed for the corresponding construction contract. However, when making this comparison, the Authority failed to account for a significant amount of expected future construction costs. The amendment documents, dated March 2018, projected that $545 million of additional construction work would remain as of July 2018. However, a status report from July 2018 shows $676 million remaining, for a total contract value of $1.5 billion. Further, cost projections from December 2017, three months before the amendment discussion, indicated that the construction contract would ultimately cost more than $2 billion due to expected changes. The Authority's failure to account for this expected work suggests that it may need additional amendments for the oversight firm's contract in the future. If the Authority continues to account for expected construction cost increases only after those increases take effect, it must develop a process for tracking the actual impacts of those increases on its oversight firms to ensure their spending is reasonable and to reduce the likelihood that the contracts run out of funds prematurely.

Although the Authority’s Construction Contracts Include Potentially Effective Methods for Verifying Progress and Costs, Those Methods Require Improved Oversight

The Authority has separate management structures to oversee its three active construction projects. As we discuss in Chapter 1, the Authority has assigned an oversight firm to manage—under its direction—each of the three current construction contracts. In addition, the Authority has assigned a contract manager to be its authorized representative for each construction contract and to also manage the pertinent oversight firm's contract. Because managing its construction contracts is distinctly different than managing its other contracts in terms of how it measures and pays for work, the Authority has developed requirements specific to this process. Specifically, the Authority's policy requires oversight firms to provide primary, day‑to‑day oversight of the construction contractors' progress in meeting contract requirements.

Because the current construction contracts include provisions that require the contractors to base their invoices on their progress instead of on the costs they incur, the total amount each construction contractor can earn is the total value of the contract—inclusive of contract increases due to change orders—regardless of the contractor's actual costs to perform the work. The Authority tasks the oversight firms with evaluating and verifying the construction contractors' claimed progress as part of reviewing the contractors' monthly invoices. When an oversight firm has verified a construction contractor's progress, it communicates that fact to the Authority's contract manager, who then approves payment based on the agreed‑upon value of the work as identified in the invoice materials.

The Authority's construction invoicing approach has the potential to significantly limit the risk of it overpaying the construction contractors, but the extent to which the Authority can rely on it as a cost control tool depends on how well the Authority oversees the work of the oversight firms. When we reviewed the invoicing processes for the three construction contracts, we determined that the invoicing documents contained the required documentation to allow the oversight firms to evaluate construction progress. However, as a matter of practice, the Authority's contract managers stated that they rely on formal review and approval from the oversight firms without reviewing additional information to independently verify the accuracy of the invoices. Determining whether the oversight firms have effectively performed their roles would require the Authority to actively monitor the oversight firms and to document that monitoring. However, the contract managers are only now beginning to institute this type of active monitoring. In early 2018, in response to board concerns, the contract managers for the three oversight firms developed a performance evaluation template for the oversight firms, as we discuss above. Authority staff presented the preliminary performance evaluation template to the board in June 2018, but this presentation did not include actual evaluation data. Instead, the contract managers stated that they expected to conduct formal evaluations using the template on a quarterly basis moving forward.

The Authority has not yet established any formal methodology or procedures for using the performance evaluation template. For example, the template asks contract managers to rank the oversight firms' performances on a scale from one (poor) to five (excellent) across a range of categories, but it does not make clear how contract managers are to determine the scores. The fact that three contract managers will use the template while monitoring different oversight firms creates the risk that their evaluations will be inconsistent. Although the contract managers have discussed setting specific criteria for determining how they will assign values, they have not yet done so. Further, the contracts director acknowledged that as of September 2018, the Authority is still evaluating the initial methodology and that it plans to monitor monthly trends shown through the reports. Because the Authority's oversight structure for construction contracts relies heavily on the oversight firms, it is crucial that the Authority provide a clear methodology for consistently evaluating the oversight firms' performance.

The Authority's construction contract management structure also does not mitigate the risks of cost overruns by the oversight firms. In fact, two of the three oversight firms expended their funds ahead of schedule and needed substantial contract amendments to continue performing their oversight duties. When we asked the pertinent contract managers about the reasons for these cost overruns, they referred to changes that the Authority had made to the construction contracts that the firms oversee. These change orders, which we discuss in Chapter 1, added significant costs and time to the construction contracts and thus—according to the contract managers—caused significant additional work and expense for the oversight firms. However, the Authority did not amend the oversight firms' scopes of work and funding contemporaneously with these changes. Rather, as we discuss in the previous section, the Authority amended one of the oversight firms' contracts to add $28.5 million when it was projected to deplete its contract funds nine months ahead of schedule.

Recommendations

To improve its contract management, increase accountability, and justify the significant amount it pays for contracted services, the Authority should take the following steps by May 2019:

• Prioritize contract management efforts and reduce the frequency with which contract management responsibilities shift among Authority staff by establishing a formal process for hiring and assigning full‑time, experienced contract managers. These contract managers should have duty statements reflecting their contract oversight responsibilities, and they should report to supervisors who understand those responsibilities and have extensive knowledge about the contracts' deliverables. In addition, those supervisors' duty statements should clearly lay out their responsibility for addressing any contract manager noncompliance with the Authority's contract management policies and procedures, whether reported by CMSU or identified by another means.
• Require CMSU to establish a schedule to monitor individual contract manager compliance and report annually the results of this monitoring to Authority executive leadership. To help ensure the integrity of its oversight role, CMSU should be composed of state staff in place of RDP consultants.
• Hold contract managers accountable for performing the duties that the Authority's policies assign to them. Specifically, CMSU and, to the extent necessary, contract managers' supervisors should require and review evidence from contract managers demonstrating their approval of deliverables, detection and resolution of contractor performance issues, and assessment of contract amendments for merit. The Authority should not accept observations and reports from its contractors or the RDP consultants in place of this evidence.

To prevent the inappropriate use of contractors to perform state functions, the Authority should develop procedures by May 2019 for evaluating whether new and existing administrative duties should be assigned to contractors or to state employees.

To ensure that contract managers' invoice reviews are complete and that invoiced costs are allowable under contract terms, the Authority should amend its applicable procedures by May 2019 to require contract managers to document their review of invoiced rates and expenses.

To ensure the consistency and effectiveness of its efforts to monitor the performance of the oversight firms with which it contracts, the Authority should develop a formal methodology by May 2019 for using the performance evaluation tool it has implemented. This methodology should include procedures for assessing the sufficiency of the oversight firms' review and approval of invoices for construction contracts.

To ensure that the oversight firms' spending is reasonable, the Authority should develop a formal process by May 2019 for tracking any out‑of‑scope work that the oversight firms perform. To reduce the likelihood that its contracts with the oversight firms run out of funds prematurely as a result of this additional work, the Authority should also develop a formal process for amending the oversight firms' contracts contemporaneously to change orders that significantly extend the timelines or increase the scope of work of the construction contracts that oversight firms oversee.

---

Footnotes

⁴ One contract manager was responsible for two of the nine contracts we reviewed. Go back to text

⁵ We also reviewed the Authority's oversight of its three construction contracts, which have a combined current value of $3.1 billion. We discuss the management of these contracts in the following sections of this report. Go back to text

Back to top