Report 2018-108

California State Transportation Agency

October 22, 2018

Elaine M. Howle, California State Auditor
California State Auditor's Office
621 Capitol Mall, Suite 1200
Sacramento, CA 95814

Dear Ms. Howle:

Attached please find a response from the California High-Speed Rail Authority (Authority) to your draft audit report (#2018-108) issued October 15, 2018. Thank you for allowing the Authority and the California State Transportation Agency (CalSTA) the opportunity to respond to the report.

From the outset, CalSTA and the Authority’s new Chief Executive Officer and executive management team welcomed your audit and now appreciates your identification of opportunities for improvement. Accordingly, the Authority concurs with all recommendations in the report and either already has taken steps toward implementing them or has established timelines to do so.

If you need additional information regarding the Authority’s response, please do not hesitate to contact Michael Tritz, CalSTA Deputy Secretary for Audits and Performance Improvement, at (916) 324-7517.

Sincerely,

BRIAN C. ANNIS
Secretary

Attachment

cc: Brian P. Kelly, Chief Executive Officer, California High-Speed Rail Authority

California High-Speed Rail Authority

October 22, 2018

Brian Annis, Secretary
California State Transportation Agency
915 Capitol Mall, Suite 350-B
Sacramento, CA 95814

Dear Secretary Annis:

The California High-Speed Rail Authority (Authority) appreciates the opportunity to respond to the California State Auditor (CSA) draft audit report issued on October 15, 2018. The CSA conducted this audit at the request of the Joint Legislative Audit Committee. The audit’s scope addressed the efficiency and efficacy of the policies and practices employed by the Authority. We concur with and will work to implement the CSA’s recommendations as an integral part of the Authority’s commitment to excellence and continuous improvement. We are pleased to report that we have begun implementing these and other corrective actions to remedy the issues identified.

The Authority is making progress on delivering the California high-speed rail system. The Authority’s challenge has been to evolve its organization and improve its program management processes, while concurrently delivering this complex mega-program. We are working to meet this challenge through a continuous improvement process by which we methodically and regularly identify and apply lessons learned. Through this ongoing process, we are taking systematic steps to expedite the Authority’s transition to a more rigorous program management and delivery organization. This evolution included establishing new governance structures in mid-2017 -- that we further strengthened in 2018 -- to more effectively manage the program through highly structured configuration management and change processes.

To advance and expedite this organizational transition, the Board of Directors recruited new leadership in 2017. The Board appointed me as the Authority’s Chief Executive Officer and my tenure began in February 2018. I was joined immediately thereafter by two newly appointed senior executives I selected to support me in leading this organization – a new Chief Operating Officer and a Chief Deputy Director. Those individuals brought specific skill sets in construction and program delivery and in administrative management. Together, we and the rest of the senior executive team take the audit’s recommendations seriously.

My senior executive team has reviewed the audit and is working with staff to address the CSA’s recommendations. This will include conducting root cause analyses and implementing additional corrective actions beyond the audit’s recommendations, if necessary. We regard this as an opportunity to further solidify the systemic improvements that the Authority has continued to establish and that we have strengthened over the last year.

Also, the CSA’s audit coincides with the Authority’s development of three key foundational governance documents that are essential to our ongoing process of continuous improvement. First, in May 2018, the Board of Directors adopted the 2018 Business Plan, which lays out the Authority’s implementation strategy for delivering California high-speed rail. Second, in June, the Board adopted our 2018 Program Baseline, which is an essential management document that outlines the scope, schedule and budget for the Authority Program’s early delivery objectives. Third, in October 2018 we finalized our Project Management Plan (PMP), which clarifies our integrated organizational framework and presents a governance structure stressing program management and delivery.

As detailed below, we are actively addressing the audit’s recommendations through implementation and other remedial steps. As it has done in the past, the Authority will move swiftly to fully implement the recommendations of the State Auditor that bear on the efficacy of program delivery. We hope and trust that our rapid implementation of these recommendations will bolster confidence in the Authority’s commitment to active management and continuous improvement of this most important program.

The CSA recommendations and the Authority’s responses (in bold) are as follows:

Chapter 1:
The Authority’s Decision to Begin Construction Before Completing Proper Planning Led to Cost Overruns and Delays

1.	To ensure that the change orders it approves are necessary and that their costs are appropriate, the Authority should adhere to the guidance and estimates the oversight firms provide to it. If the Authority chooses to deviate from the oversight firms’ recommendations, it should clearly document why it made those deviations. Response: The Authority concurs with this recommendation. In 2017, the Authority initiated a governance process to assess the construction, financial, legal and other program perspectives for all changes. The Board of Directors adopted the 2018 Business Plan that included the new governance (see page 63 of the Plan) and adopted the program Baseline in June 2018. The Authority then updated and formalized this process in the Program Management Plan (PMP). The current process includes: a Program Delivery Committee, which is a management committee that holds the functional groups accountable for program delivery and evaluates all pending and potential change orders; and a Business Oversight Committee, which acts as a change control committee and must approve all change orders before they go to the Executive Committee and, when needed, the Board of Directors (Board) for approval to execute. Each governance committee has an approved charter that outlines its purpose and decision-making authority. In accordance with the CSA recommendation, the Authority will revise the charters to require documentation when a governance committee overrules a Project Construction Management (PCM) firm’s recommendation. This will be accomplished through a Business Case, which is the document for requesting a proposed change order or a change to the Baseline. The Business Case is required to provide a summary and justification of the recommended actions/changes and includes signatures from the relevant functional, legal, construction and program teams. For construction change orders, the accompanying Business Case will document the PCM recommendation and cost estimate with an explanation of any differences. Planned completion date: June 2019
2.	Before executing its next construction contract, the Authority should establish formal prerequisites for beginning construction to prevent avoidable cost overruns and project delays. At a minimum, these prerequisites should identify specific benchmarks related to land acquisition, utility agreements and relocations, and agreements with external stakeholders, including impacted local governments and other railroad operators. Response: The Authority concurs with this recommendation and has placed significant focus on this issue. The 2018 Business Plan identifies this as a critical lesson learned and it continues to be a point of emphasis (see Chapter 4, Lessons Learned and Managing Risk, page 53 of the Plan). Key among the lessons learned was that the Authority’s decision to award design-build contracts before acquiring right of way and completing agreements with utilities, local governments and railroads meant there were many unknowns that created risks of delays and higher costs. The same chapter also describes the governance and management procedures initiated in 2017 to strengthen decision making through a highly structured process, which our response to Recommendation 1 outlines briefly. 1 More specifically, the Business Oversight Committee considers benchmarks prior to progressing to procurement or to the next phase of project delivery. For example, the Business Oversight Committee would consider the progress of pre-construction activities, including right-of-way acquisition, prior to approving procurement activities to select a construction contractor. 2 Further, delivering projects in accordance with the Program Baseline, which was adopted by the Board of Directors in June 2018, also ensures that certain predecessor tasks (or prerequisites) are sufficiently advanced prior to beginning construction, as the Baseline incorporates the lessons learned outlined in the 2018 Business Plan. The Authority continues to develop Baseline project work plans composed of discretely defined tasks. These tasks are linked together based on project delivery sequencing, which establishes what tasks must be completed prior to beginning other tasks (i.e., predecessor tasks and successor tasks). Planned completion dates: Project Work Plans (with benchmarks) - Draft: January 2019 Project Work Plans (with benchmarks) - Final: February 2019
3.	To better position itself to complete the three Central Valley projects by the December 2022 federal grant deadline, the Authority should improve its monitoring and evaluation of the oversight firms’ risk assessment processes and should take steps to ensure that these processes are consistent across the three projects by May 2019. Response: The Authority concurs with this recommendation. The Authority recognizes the risks associated with complying with the deadline established by the American Recovery and Reinvestment Act of 2009 and is in the process of reorganizing its risk team overseeing the Central Valley projects. It is hiring additional risk management personnel and will prioritize both risk management and mitigation in a way that is consistent across the three construction projects. Further, to verify that the risk assessment processes are consistent across the three projects, the Authority will continue to enforce all policies and procedures related to PCM oversight, revise the PCM manual to be more explicit on risk management, and will publish a program estimate-to-complete and risk management manual. Planned completion date: April 2019
4.	To enable policymakers and the public to track the Authority’s progress toward meeting the federal grant deadline of December 2022, the Authority should, by January 2019, begin providing quarterly updates to the Legislature detailing the progress of the three Central Valley construction projects using an earned value model that compares construction progress to the projected total completion cost and date. The Authority should base these updates on the most current cost estimates available. Response: The Authority concurs with this recommendation. The Authority recently developed a Program Delivery Status Report (PDSR) that it produces on a monthly basis. The PDSR includes comprehensive status information within the Authority’s three main areas of project delivery: 1) right-of-way procurement, third party agreements, and environmental clearance; 2) engineering/design and construction; and 3) rail infrastructure. This PDSR and associated procedures will be codified within the Program Controls Manual to be published. The PDSR provides detailed information on the progress of the three Central Valley construction projects within the Infrastructure Delivery section, including cost variance and schedule performance index (SPI). Moving forward, the Authority will use the most current cost information and an earned value model to refine the cost variance and SPI for each of the three Central Valley construction projects. On a monthly basis, the cost variance and SPI information will be used to estimate the projected total completion cost and date for each of the Central Valley construction projects. This information will be included in the PDSR. By January 2019, the Authority will use information from the PDSR – including earned value, cost variance and SPI – to develop and provide quarterly updates to the Legislature. The updates will include detailed information on the progress of the three Central Valley construction projects. This information will be used to actively manage the construction projects to ensure that the 2022 federal grant deadline is met. Planned completion dates: PDSR Quarterly Update - Draft: December 2018 PDSR Quarterly Update - Final: January 2019 PDSR Manual – May 2019
5.	To ensure that it is adequately prepared if it is unable to meet the federal grant deadline of December 2022, the Authority should, by May 2019, develop a contingency plan for responding to such a scenario. Response: The Authority concurs with this recommendation. The Authority intends to meet the federal grant deadline and, to achieve that, we continuously monitor and assess the program through the Program Delivery Committee and the Business Oversight Committee. As part of this monitoring process we routinely update the individual project risk registers in coordination with the Federal Railroad Administration (FRA) on a quarterly basis. The Program Delivery Committee and the Business Oversight Committee use the Project Update Report, Business Plan and Baseline in their monitoring activities. We will continue to assess different contingency plans/options through each of these activities and will prepare a contingency plan in accordance with the CSA recommendation, which will be updated on an annual basis. Planned completion date: May 2019.

Chapter 2:
The Authority Has Not Successfully Enforced the Policies It Implemented to Address Ongoing Deficiencies With Its Contract Management

1.	To improve its contract management, increase accountability, and justify the significant amount it pays for contracted services, the Authority should take the following steps by May 2019:. Prioritize contract management efforts and reduce the frequency with which contract management responsibilities shift among Authority staff by establishing a formal process for hiring and assigning full-time, experienced contract managers. These contract managers should have duty statements reflecting their contract oversight responsibilities and they should report to supervisors who understand those responsibilities and have extensive knowledge about the contracts’ deliverables. In addition, those supervisors’ duty statements should clearly lay out their responsibility for addressing any contract manager noncompliance with the Authority’s contract management policies and procedures, whether reported by Contract Management Support Unit (CMSU) or identified by another means. Response: The Authority concurs with the recommendation. The Authority will create a formal process for hiring and assigning full-time experienced contract managers to reduce the frequency with which contract management responsibilities shift among Authority staff. This will include emphasizing contract management experience/skills as well as desirable contract manager qualifications. In addition, all new advertised positions that require contract management will specify the skills required for a contract manager within the duty statement. In addition, all existing duty statements will be reviewed and modified to reflect contract management/oversight responsibilities for all contract managers and their supervisors. Contract manager supervisors’ duty statements will also address their responsibility to hold their contract management staff accountable for compliance with the Authority's contract management policies and procedures. While only contract managers are currently required to be trained in contract management, contract manager supervisors will also be required to attend contract management training to ensure that the contract managers they supervise are adhering to the Authority’s policies and procedures. The Authority will also create a separate contract management training specifically for supervisors. Planned completion date: May 2019. Require CMSU to establish a schedule to monitor individual contract manager compliance and report annually the results of this monitoring to Authority executive leadership. To help ensure the integrity of its oversight role, CMSU should be composed of state staff in place of RDP consultants. Response: The Authority concurs with the recommendation. A schedule to monitor contract managers’ compliance has been created and assessments of contract manager performance are to begin no later than November 2018, to comply with the Authority’s PROC-FIS-038, Contract Compliance Procedure. The assessments will be electronically tracked for resolution status and an executive report will be prepared no less than annually. 3 In addition, the Authority concurs that CMSU should be composed of state staff. The Authority will prepare a Budget Change Proposal for the 2020/21 fiscal year to request staff augmentation to remove contracted Rail Delivery Partner (RDP) consultants and replace with state employees. Planned completion date: The contract manager assessment schedule has been developed and the Budget Change Proposal concept will be submitted to the Authority Executive Committee for consideration in May 2019. Hold contract managers accountable for performing the duties that the Authority’s policies assign to them. Specifically, CMSU and, to the extent necessary, contract managers’ supervisors should require and review evidence from contract managers demonstrating their approval of deliverables, detection and resolution of contractor performance issues, and assessment of contract amendments for merit. The Authority should not accept observations and reports from its contractors or RDP consultants in place of this evidence. Response: The Authority concurs with the recommendation. The Authority’s contract compliance policy POLI-FIS-038 established requirements for performing assessments and reports on contract manager compliance with the Authority’s policies and procedures. The Authority will begin performing assessments of contract managers no later than November 2018. We will begin with the contracts identified in this audit and continue until all contracts have been fully assessed. These assessments will ensure that contract managers are following the Authority's policies and procedures, which are the foundation of a well-managed contract. The assessments will also provide documented evidence that contract managers, not RDP consultants, are properly approving deliverables/invoices, resolving contractor disputes or performance issues appropriately, and justifying contract amendments with verifiable documentation in all contracts. The assessments will not only provide supervisors/management with documentation demonstrating accountability (conformance/nonconformance), but also contain recommendations for best practices and opportunities for improvement. Once an assessment is issued, it will be tracked to closure/resolution. The process also maintains the tracking mechanism to verify the implementation of the corrective action. This plan provides a framework for review of the contract managers’ performance by contract manager and by contract, providing a quantitative assessment of contract manager performance. 4 Planned completion date: The Authority will begin performing assessments of contract managers no later than November 2018 and anticipates that all contract manager assessments will be completed by or before November 2020 and be ongoing thereafter.
2.	To prevent the inappropriate use of contractors to perform state functions, the Authority should develop procedures by May 2019 for evaluating whether new or existing administrative duties should be assigned to contractors or to state employees. Response: The Authority concurs with this recommendation. The Authority’s Administration Office will develop procedures for evaluating whether new or existing administrative duties should be assigned to contractors or to state employees. Planned completion date: May 2019
3.	To ensure the completeness of contract managers’ invoice reviews and that invoiced costs are allowable under contract terms, the Authority should amend its applicable procedures by May 2019 to require contract managers to document their review of invoiced rates and expenses. Response: The Authority concurs with this recommendation. As part of a larger plan to revise, field test, and finalize the contract management procedures, the Authority will revise its invoice review procedures (FIS-PROC-033) to require contract managers to document their review of invoiced rates and expenses. Planned completion date: May 2019
4.	To ensure the consistency and effectiveness of its efforts to monitor the performance of the oversight firms with which it contracts, the Authority should develop a formal methodology by May 2019 for using the performance evaluation tool it has implemented. This methodology should include procedures for assessing the sufficiency of the oversight firms’ review and approval for construction contracts. 5 Response: The Authority concurs with this recommendation. The Authority has developed and implemented monthly performance-based evaluations for its construction oversight firms that are updated by each contract manager. A formal methodology will be developed for the performance evaluations. Contract Performance Monitoring and Reporting Policy POLI-FIS-034 and Procedure PROC-FIS-034 will be revised to document this formal methodology. The Authority’s contract manager and quality team will evaluate the PCM’s procedure for compliance with section 3.9 Contractor’s Monthly Payments of the Project and Construction Management Manual (For Design-Build Contracts) on all construction packages; specifically, for the purpose of assessing construction progress to validate invoice submissions and earned value from the design-builder. The Authority will amend its Quality Manual to reflect this process and the frequency of the assessments. Any non-conformance will be formally documented. Planned completion date: May 2019
5.	To ensure oversights firms’ spending is reasonable, the Authority should develop a formal process by May 2019, for tracking any out-of-scope work the oversight firms perform. To reduce the likelihood that its contracts with these oversight firms run out of funds prematurely as a result of this additional work, the Authority should also develop a formal process for amending these oversight firm’s contracts contemporaneous to change orders that significantly extend timelines or increase the scope of work of the construction contracts that they oversee. 6 Response: The Authority concurs with this recommendation. The June 2018 Program Baseline, approved by the Board of Directors, details the scope, schedule and budget for all work on the high-speed rail program, including PCM firms’ contract budgets and schedule. Any PCM contract amendment requires approval from the Business Oversight Committee. As outlined in the response to Recommendation 1, the Business Oversight Committee requires a Business Case with subject matter expert signatures before considering any contract change. Additionally, the Authority will revise the PCM manual to reflect their responsibility to identify, manage and request amendments for out-of-scope work. The PCM contract managers will separately be responsible for tracking and identifying any potential out-of-scope work and the necessity to amend contracts at each PCM work plan revision and approval. The Program Delivery Committee is responsible for governance oversight of contracts and will consider the impact on PCM contracts by programmatic trends (such as schedule delays) or changes approved by the Business Oversight Committee (such as a change to a design-build contract). A key component of Program Delivery Committee meetings will focus on upcoming key milestones and decision-points, including when or if contracts need amending. Planned completion date: May 2019

Chapter 3:
The Authority Can Improve the Quality and Transparency of Its Monitoring and Reporting for Key Goals

1.	To help improve the effectiveness of its sustainability policy, the Authority should revise the policy by May 2019 to more clearly differentiate between construction and operation phases of the high-speed rail system. Further, it should ensure that each objective in each section of the policy is associated with quantifiable metrics for evaluating implementation. 7 Response: The Authority concurs with this recommendation. The Authority will revise its Sustainability Policy to more clearly differentiate between construction and operations. Since the end of audit fieldwork, the Authority has revised the implementation plan to match each existing policy objective with a quantitative metric, posted here: http://www.hsr.ca.gov/docs/programs/green_practices/sustainability/Sustainability_implementation_plan_SUMMARY_Oct2018.pdf. Planned completion date: The Policy will be updated by May 2019.
2.	To allow it to evaluate the sustainability of the high-speed rail system’s construction, the Authority should perform and document a review of its compliance with its existing quality controls related to ensuring the validity and completeness of contractor-reported data by May 2019. The Authority should also establish a formal process to perform such reviews periodically. 8 Response: The Authority concurs with this recommendation. The Authority improved its environmental and sustainability data gathering and analysis system in July 2017. As an additional improvement, the Authority will develop a quality assurance process to perform periodic reviews of its compliance with the quality controls related to validity and completeness of contractor-reported data. The updated system has a data field validation feature, which is a quality control measure that involves a cross check at the data entry level, next level review and confirmation of data by PCMs, and a final data quality review and acceptance by the Authority. This process is enforced by the system and is ongoing and continuous. The recommended periodic reviews will verify that the above-described controls are functioning as expected. Planned completion date: May 2019
3.	To help ensure it meets its sustainability goals, the Authority should comprehensively compare the three construction project’s performance to their construction contractors’ original baseline estimates on a quarterly basis by May 2019. 9 Response: The Authority concurs with this recommendation. Since 2015, the Authority has collected and organized data which it uses to analyze construction activities related to air quality and greenhouse gas emissions. The Authority then compares that to the contractually identified baseline estimates; this comparison is done on a quarterly basis. The Authority compares the contractor’s required performance for waste, absolute targets for recycling of concrete and steel and a percentage target for remaining nonhazardous waste, as recycling records are submitted. Other sustainability performance tracking, related to fuel usage, water usage, and recycling, had previously been analyzed and compared to baseline estimates on an annual basis. The Authority will begin comparison of these performance areas on a quarterly basis and will adjust the analysis to include the relationship to construction progress. Planned completion date: May 2019
4.	To help ensure that its contractors proposed environmental impacts are reasonable and to measure the progress of its sustainable construction efforts over time, the Authority should, by November 2019, identify and track standardized measure – such as project miles – that will allow it to compare construction impacts across the high-speed rail system’s different construction projects. Response: The Authority concurs with this recommendation. Currently, the Authority has a model that assesses program environmental impacts using standardized metrics, including tons of CO2e, kgCO2e/kg, kgCO2e/gallon, kgCO2e/kWh, normalized by miles and construction typology. The Authority is in the process of updating this model to incorporate refined project lengths and infrastructure typologies adopted in the Baseline and to incorporate relevant, validated data (e.g., tons of CO2e and environmental product declarations for actual materials installed) tracked on each construction package. This updated model will allow comparison across construction projects and support establishing targets for future construction contracts. Planned completion date: November 2019
5.	To increase the transparency of its reporting, the Authority should, by May 2019, expand its quarterly small business, DBVE, and DBE utilization reporting to account for the total value of all its contracts and to identify the reasons it has exempted specific contracts. Response: The Authority concurs with this recommendation. In an effort to increase transparency, the Authority will post the total value of all current contracts within the Small Business Program section of the Authority’s website. The Authority will also post the total value of any contracts that require small business, DVBE and/or disadvantaged business, which will appropriately reflect the Authority’s utilization percentages per state and federal guidelines. While the Authority reports utilization on a quarterly basis for federally funded contracts, the state process, administered by the Department of General Services, only requires an annual report from all state agencies/departments. However, the Authority will create an internal process to closely align the state timeline to the federal quarterly process. In addition, a policy will be created that will clearly specify what contracts are exempt from small business, DVBE and/or disadvantaged business according to state and/or federal regulations, policies, and guidelines. Planned completion date: May 2019

Again, we appreciate the opportunity to provide a response to our plans to implement the California State Auditor’s recommendations. If you have any questions, please contact Paula Rivera, Chief Auditor, at paula.rivera@hsr.ca.gov or (916) 403-2679

Sincerely,

JD Hedges
for Brian P. Kelly
Chief Executive Officer

Comments

CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE CALIFORNIA HIGH‑SPEED RAIL AUTHORITY

To provide clarity and perspective, we are commenting on the Authority's response to the audit. The numbers below correspond to the numbers we have placed in the margin of its response.

Although the Authority states that the Business Oversight Committee (BOC) considers benchmarks prior to progressing to procurement or the next phase of project delivery, this committee, which the Authority established in September 2017, has not yet overseen the procurement of a construction contractor. As Figure 6 shows, the Authority executed the most recent of its three construction contracts—for Project 4—in 2016. Therefore, any oversight the committee provides regarding key preconstruction activities is still prospective.

As we discuss here, the Authority's recently released comprehensive schedule delineates that early work tasks, such as land acquisition, should begin before the design and construction phase. However, the schedule does not establish specific benchmarks the Authority must achieve before procuring a construction contractor. Therefore, to avoid cost overruns and delays from moving to the construction phase too soon, as our recommendation specifies, the Authority should establish formal prerequisites for beginning construction and these prerequisites should identify specific benchmarks related to land acquisition, utility agreements and relocations, and agreements with external stakeholders.

As we state here, CMSU's oversight of contract management policies and procedures has been weak and inconsistent. We also explain here that placing oversight responsibility with RDP consultants creates a potential conflict of interest. However, under the Authority's proposed approach, it would not have funding for state employees to staff CMSU until at least July 2020. Given the issues we found and the amount of public funds at stake, we believe the Authority should move faster to secure professional state staff to perform CMSU's crucial oversight duties.

We state here that as of September 2018, the Authority's 56 contract managers were collectively responsible for 204 contracts. Although we recognize the amount of work necessary to hold each contract manager accountable for each of their assigned contracts, we do not believe the State can afford for the Authority to take two years to complete its assessments of contract managers, as its response indicates. Additionally, we encourage the Authority to focus its initial efforts not only on the contracts identified in this audit but also its other largest and highest risk contracts.

The Authority's response does not specify whether its methodology for evaluating its PCMs—which we refer to in our report as construction oversight firms (oversight firms)—will include procedures for assessing the sufficiency of the oversight firms' reviews and approvals of invoices for construction contracts. As we state in our recommendation, such procedures are important to ensure consistency and the effectiveness of the Authority's efforts to monitor the performance of the oversight firms with which it contracts. Therefore, the Authority should include these procedures in its methodology.

The Authority states that any oversight firm contract amendment requires approval by the BOC. However, we identified concerns with an oversight firm contract amendment that the Authority approved in March 2018. Specifically, as we describe here, the Authority's documents assert that the amendment stemmed from it assigning work to the oversight firm that was outside the original contract. However, the contract manager was unable to tell us the amount of funding that went to the oversight firm for this out‑of‑scope work and acknowledged that he had not documented the oversight firm's adequate performance. Therefore, until the Authority implements our recommendation to track any out‑of‑scope work that oversight firms perform, the Authority, including its BOC, will not know whether the oversight firms' spending rates are reasonable or if amendments are appropriate.

We appreciate that the Authority is taking steps to match existing sustainability policy objectives with quantitative metrics in its implementation plan. However, our recommendation is that the Authority first revise its sustainability policy to more clearly differentiate between construction and operation phases of the system, a process we would expect to generate new policy goals and objectives. At that point, the Authority should reevaluate its implementation plan to ensure that the plan contains appropriate metrics for those new policy objectives.

Although the Authority asserts that it improved its environmental and sustainability data gathering and analysis system in July 2017, the Authority's sustainability director confirmed during our audit that the Authority was still in the process of collecting and reviewing sustainability data for calendar year 2017 and that the transition to this system was still ongoing. Additionally, the Authority's response describes quality control measures for the new data system, but these measures are not fundamentally different from those that were in place under the previous system. We describe our concerns with the accuracy of the data here.

The Authority's assertion that it has completed these comparisons since 2015 is inconsistent with the evidence it provided during our audit. During the audit, the Authority's sustainability director confirmed for us that the Authority was beginning the process of comparing actual sustainability impacts to contractor estimates, but that the Authority did not expect to have preliminary findings until the end of 2018. The Authority also provided documentation showing that some comparisons were underway. However, as we state here, our expert observed that this documentation does not allow the Authority to effectively project whether contractors will meet or exceed their estimates because it does not account for actual construction progress to date. We also state here we also state that documentation did not include equivalent comparisons related to the environmental impacts from waste or water use. Therefore, we stand by our recommendation that the Authority should comprehensively compare the three construction projects' performances to their original baseline estimates on a quarterly basis, and it should begin doing so no later than May 2019.