Report 2016-125.2 All Recommendation Responses

Report 2016-125.2: The University of California Office of the President: Increasing Costs and Scheduling Delays Have Hampered the UCPath Project and Originally Anticipated Savings Are Unlikely to Materialize (Release Date: August 2017)

Recommendation #1 To: University of California Board of Regents

To ensure that they are able to exercise necessary oversight for the university's significant IT projects, the regents should develop status reporting standards for the Office of the President and all university locations to follow by December 2017. Such reporting standards should apply to all university IT projects with more than a specified cost and, at a minimum, should establish the frequency with which the Office of the President and all university locations must report to the regents. Such updates should occur at least three times per calendar year and coincide with regents' meetings to allow for oral discussion.

6-Month Agency Response

The Board of Regents approved a new policy related to significant IT project reporting for all projects budgeted or estimated above $5 million.

Policy 5103: (http://regents.universityofcalifornia.edu/governance/policies/5103.html )

The first triannual reporting was made to the Regents at the March 2018 meeting.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The University will develop a policy for large IT projects of $25M or greater that will establish (1) the frequency with which the Office of the President and all university locations report project status to the regents, (2) the types of disclosures the Office of the President and all university locations must present about each IT project to the regents, and (3) the types of significant project risks the Office of the President and all university locations must disclose and steps being taken to mitigate the risks on significant IT projects. Regental approval for the new policy is targeted for November 2017.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #2 To: University of California Board of Regents

To ensure that they are able to exercise necessary oversight for the university's significant IT projects, the regents should develop status reporting standards for the Office of the President and all university locations to follow by December 2017. Such reporting standards should apply to all university IT projects with more than a specified cost and, at a minimum, should establish the types of disclosures the Office of the President and all university locations must present about each IT project including, but not limited to, changes in scope, projected cost, and schedule.

6-Month Agency Response

The Board of Regents approved a new policy related to significant IT project reporting for all projects budgeted or estimated above $5 million.

Policy 5103: (http://regents.universityofcalifornia.edu/governance/policies/5103.html )

The policy establishes the types of disclosures the Office of the President and all university locations must present about each IT project including, changes in scope, projected cost, and schedule. Projects budgeted or estimated above $25 million have an additional reporting requirement, established and agreed by all UC locations and approved by the Regents, to provide more detailed status on the project including specific sections on status and changes in scope, projected cost, risk and schedule.

The first triannual reporting was made to the Regents at the March 2018 meeting. The next report/dashboard will be provided to the Regents at the July 2018 meeting.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The University will draft a policy for large IT projects of $25M or greater that will establish (1) the frequency with which the Office of the President and all university locations report project status to the regents, (2) the types of disclosures the Office of the President and all university locations must present about each IT project to the regents, and (3) the types of significant project risks the Office of the President and all university locations must disclose and steps being taken to mitigate the risks on significant IT projects. Regental approval for the new policy is targeted for November 2017.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #3 To: University of California Board of Regents

To ensure that they are able to exercise necessary oversight for the university's significant IT projects, the regents should develop status reporting standards for the Office of the President and all university locations to follow by December 2017. Such reporting standards should apply to all university IT projects with more than a specified cost and, at a minimum, should establish the types of significant project risks the Office of the President and all university locations must disclose. The updates should also describe the actions the Office of the President and all university locations are taking to mitigate the risks and the potential effects of those risks on a project's cost, schedule, and scope.

6-Month Agency Response

The Board of Regents approved a new policy related to significant IT project reporting for all projects budgeted or estimated above $5 million.

Policy 5103: (http://regents.universityofcalifornia.edu/governance/policies/5103.html )

The policy establishes the types of disclosures the Office of the President and all university locations must present about each IT project including, changes in scope, projected cost, and schedule. Projects budgeted or estimated above $25 million have an additional reporting requirement, established and agreed by all UC locations and approved by the Regents, to provide more detailed status on the project including specific sections on status and changes in scope, projected cost, risk and schedule. They are also asked to outline the specific actions in response to changes in scope, projected cost, risk or schedule.

The first triannual reporting was made to the Regents at the March 2018 meeting. The next report/dashboard will be provided to the Regents at the July 2018 meeting.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The University will draft a policy for large IT projects of $25M or greater that will establish (1) the frequency with which the Office of the President and all university locations report project status to the regents, (2) the types of disclosures the Office of the President and all university locations must present about each IT project to the regents, and (3) the types of significant project risks the Office of the President and all university locations must disclose and steps being taken to mitigate the risks on significant IT projects. Regental approval for the new policy is targeted for November 2017.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: University of California

To ensure that it fully reports the cost of the IT projects, the Office of the President should develop the cost reporting guidelines by December 2017 for UCPath and other significant IT projects across all university locations. These cost guidelines should identify cost categories at both the Office of the President and university locations to ensure that the estimates capture and communicate all development and implementation costs. In addition, the Office of the President should produce cost reports to share with stakeholders at least quarterly.

6-Month Agency Response

The University has obtained approval from the Board of Regents for a new policy related to significant IT project reporting. The first quarterly reporting was made to the Regents at the March 2018 meeting.

As part of the reporting, The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.5, there are guidelines established, examples provided, and templates provided for how locations should estimate their project budgets communicate all development and implementation costs into the project reporting framework. The material is complete and posted to http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

We have worked with the current projects with budgets greater than $25m and confirm that they are on a cadence to review project cost reports with their executive stakeholder committees quarterly. In addition and in coordination with the IT Project Handbook, we have communicated the requirement for all large projects above $5m to start reviewing project cost summaries quarterly and will be assessing compliance with that requirement over the next year. We also will have all project managers from the different university locations at a Symposium in May 2018 to share best practice for project development and reporting.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The University will develop cost reporting guidelines using internal best practices and review of the State's IT manual. Once finalized we will publish as part of a new IT project "handbook" to be distributed to locations, centrally posted on our UC-IT portal & referenced in UC-IT staff and manager meetings.

California State Auditor's Assessment of 60-Day Status: No Action Taken

The Office of the President has not yet developed the recommended cost reporting guidelines or cost reports.

Recommendation #5 To: University of California

To ensure that it consistently follows best practices related to project management, the Office of the President should develop and implement guidelines for IT project development by June 2018. The guidelines should apply to all IT projects undertaken by any university location with a cost estimate of at least $5 million or more and should include a means to assess schedules for reasonableness, and requirements for the creation and maintenance of an integrated schedule and resource plan for each project.

1-Year Agency Response

The Office of the President, working closely with its partners at the UC locations, has completed, posted & trained IT professionals on the guidelines for IT project development - a.k.a., IT Project Handbook - for all IT projects undertaken by any university location with a cost estimate of at least $5 million. This recommendation is covered in Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES and includes examples and templates.

This work was completed and posted to a central location for all IT professionals to access in May 2018.

All locations are required to implement these guidelines. A training symposium was held at UC Berkeley in May 2018. Out of the symposium has come an IT project management community of interest, which has created a virtual collaboration space (Slack channel) and organized a second gathering of the community for UC Davis in August 2018. We expect the evolution and further dissemination of the IT Project Handbook will be an ongoing topic for the community.

http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES there will be guidelines established, examples provided and starter templates for assessing schedules for reasonableness, creating an integrated project schedule, maintaining an integrated project schedule and managing a project resource plan. The material is 50% complete, and we expect to meet our June, 2018 deadline.

California State Auditor's Assessment of 6-Month Status: Pending

The Office of the President is currently drafting this chapter of its IT Project Handbook. As it does so, we encourage the Office of the President to keep in mind two elements of our recommendation which are: specifying that the guidelines apply to all IT projects with a cost estimate of at least $5 million and requiring university locations to implement the guidelines, not merely review them. These two steps are key to the Office of the President ensuring that all significant IT projects are subject to the IT project development guidelines.

60-Day Agency Response

In consultation with the UC Regents, the University will use a threshold of $25M budget as the definition for significant IT projects for which new guidelines for IT project development will apply. We will use UCPath, campus and California State practices as guidance to develop a high-level IT project methodology for projects to follow describing the methodology, guidelines and practices.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Office of the President has consulted with the regents and decided on a project level threshold, but it has not developed and implemented the recommended guidelines for IT project development. In addition, we have a concern that the Office of the President is intending to apply a threshold that is too high, $25 million rather than the $5 million that we recommended, which could result in many significant IT projects not being subject to the IT project development guidelines.

Recommendation #6 To: University of California

To ensure that it consistently follows best practices related to project management, the Office of the President should develop and implement guidelines for IT project development by June 2018. The guidelines should apply to all IT projects undertaken by any university location with a cost estimate of at least $5 million or more and should include requirements for rigorous change management processes that establish a means of assessing the implications of changes to a project's scope, cost, and schedule.

1-Year Agency Response

The Office of the President, working closely with its partners at the UC locations, has completed, posted & trained IT professionals on the guidelines for IT project development - a.k.a., IT Project Handbook - for all IT projects undertaken by any university location with a cost estimate of at least $5 million. This recommendation is covered in Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.7. There are examples and templates provided.

This work was completed and posted to a central location for all IT professionals to access in May 2018.

All locations are required to implement these guidelines. A training symposium was held at UC Berkeley in May 2018. Out of the symposium has come an IT project management community of interest, which has created a virtual collaboration space (Slack channel) and organized a second gathering of the community for UC Davis in August 2018. We expect the evolution and further dissemination of the IT Project Handbook will be an ongoing topic for the community.

http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.7, there will be guidelines established, examples provided and starter templates for should include requirements for managing the requirements of a project, establishing rigorous change management processes for assessing and managing the implications of changes to a project's scope, cost, and schedule. The material is 75% complete, and we expect to meet our June, 2018 deadline.

California State Auditor's Assessment of 6-Month Status: Pending

The Office of the President is currently drafting this chapter of its IT Project Handbook. As it does so, we encourage the Office of the President to keep in mind two elements of our recommendation which are: specifying that the guidelines apply to all IT projects with a cost estimate of at least $5 million and to require university locations to implement the guidelines, not merely review them. These two steps are key to the Office of the President ensuring that all significant IT projects are subject to the IT project development guidelines.

60-Day Agency Response

In consultation with the UC Regents, the University will use a threshold of $25M budget as the definition for significant IT projects for which new guidelines for IT project development will apply. The University will develop change management process guidelines using internal best practices and review of the State's IT manual. Once finalized we will publish as part of the new guidelines for IT project development to be distributed to locations, centrally posted on our UC-IT portal and referenced in UC-IT staff and manager meetings.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Office of the President has consulted with the regents and decided on a project level threshold, but it has not developed and implemented the recommended guidelines for IT project development. In addition, we are concerned that the Office of the President is intending to apply a threshold that is too high, $25 million rather than the $5 million that we recommended, which could result in many significant IT projects not being subject to the IT project development guidelines.

Recommendation #7 To: University of California

To ensure that it consistently follows best practices related to project management, the Office of the President should develop and implement guidelines for IT project development by June 2018. The guidelines should apply to all IT projects undertaken by any university location with a cost estimate of at least $5 million or more and should include procedurally sound requirements for identifying, reviewing, and resolving risks to a project.

1-Year Agency Response

The Office of the President, working closely with its partners at the UC locations, has completed, posted & trained IT professionals on the guidelines for IT project development - a.k.a., IT Project Handbook - for all IT projects undertaken by any university location with a cost estimate of at least $5 million. This recommendation is covered in Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.6. There are examples and templates provided.

This work was completed and posted to a central location for all IT professionals to access in May 2018. A training symposium was held at UC Berkeley in May 2018. Out of the symposium has come an IT project management community of interest, which has created a virtual collaboration space (Slack channel) and organized a second gathering of the community for UC Davis in August 2018. We expect the evolution and further dissemination of the IT Project Handbook will be an ongoing topic for the community.

http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.6, there will be guidelines established, best practices identified, examples provided and starter templates for identifying, reviewing, and resolving risks to a large IT project. The material is 95% complete, and we expect to meet our June, 2018 deadline.

California State Auditor's Assessment of 6-Month Status: Pending

The Office of the President is currently drafting this chapter of its IT Project Handbook. As it does so, we encourage the Office of the President to keep in mind two elements of our recommendation which are: specifying that the guidelines apply to all IT projects with a cost estimate of at least $5 million and to require university locations to implement the guidelines, not merely review them. These two steps are key to the Office of the President ensuring that all significant IT projects are subject to the IT project development guidelines.

60-Day Agency Response

In consultation with the UC Regents, the University will use a threshold of $25M budget as the definition for significant IT projects for which new guidelines for IT project development will apply. The University will develop risk management process guidelines using internal best practices and review of the State's IT manual. Once finalized we will publish as part of the new guidelines for IT project development to be distributed to locations, centrally posted on our UC-IT portal and referenced in UC-IT staff & manager meetings.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Office of the President has consulted with the regents and decided on a project level threshold, but it has not developed and implemented the recommended guidelines for IT project development. In addition, we are concerned that the Office of the President is intending to apply a threshold that is too high, $25 million rather than the $5 million that we recommended, which could result in many significant IT projects not being subject to the IT project development guidelines.

Recommendation #8 To: University of California

To ensure that it consistently follows best practices related to project management, the Office of the President should develop and implement guidelines for IT project development by June 2018. The guidelines should apply to all IT projects undertaken by any university location with a cost estimate of at least $5 million or more and should include IV&V to oversee the technical aspects of project development.

1-Year Agency Response

The Office of the President, working closely with its partners at the UC locations, has completed, posted & trained IT professionals on the guidelines for IT project development - a.k.a., IT Project Handbook - for all IT projects undertaken by any university location with a cost estimate of at least $5 million. This recommendation is covered in Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.8. There are examples and templates provided.

This work was completed and posted to a central location for all IT professionals to access in May 2018.

All locations are required to implement these guidelines. A training symposium was held at UC Berkeley in May 2018. Out of the symposium has come an IT project management community of interest, which has created a virtual collaboration space (Slack channel) and organized a second gathering of the community for UC Davis in August 2018. We expect the evolution and further dissemination of the IT Project Handbook will be an ongoing topic for the community.

http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Although the Office of the President does not require IV&V on IT projects with a cost estimate of at least $5 million, it requires university locations to carefully consider its use as a risk mitigation methodology. Thus, this provision substantially implements our recommendation.

6-Month Agency Response

The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 3 entitled PROJECT DEVELOPMENT GUIDELINES, and specifically in Section 3.8, there are guidelines established, examples provided and starter templates for identifying situations and recommending approaches to apply IV&V to oversee the technical aspects of project development and manage IT project risk. The material is complete and posted at http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 6-Month Status: Partially Implemented

The Office of the President has drafted this chapter of its IT Project Handbook and made it available to the university locations. However, the Office of the President must still act on two elements of the recommendation which are: specifying that the guidelines apply to all IT projects with a cost estimate of at least $5 million and requiring the university locations to implement the guidelines, not merely review them. These two steps are key to the Office of the President ensuring that all significant IT projects are subject to the IT project development guidelines.

60-Day Agency Response

In consultation with the UC Regents, the University will use a threshold of $25M budget as the definition for significant IT projects for which new guidelines for IT project development will apply. It is the University's current and proposed continued practice to use diligent governance as a way to identify and bring in IV&V third-parties for specific topics and situations as appropriate or necessary. We will include in new guidelines for IT project development factors to consider in determining the need for IV&V on an IT project.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Office of the President has consulted with the regents and decided on a project level threshold, but it has not developed and implemented the recommended guidelines for IT project development. In addition, we are concerned that the Office of the President is intending to apply a threshold that is too high, $25 million rather than the $5 million that we recommended, which could result in many significant IT projects not being subject to the IT project development guidelines.

Recommendation #9 To: University of California

The Office of the President should require that all university locations follow best practices by ensuring that each location creates a deliverable expectations document for each IT contract similar to the documents the State's management framework describes. The Office of the President should establish this requirement by December 2017. The deliverable expectations document should, at a minimum, identify the deliverables for each milestone and define the scope, content, entrance criteria, acceptance criteria, and development schedule for each deliverable.

1-Year Agency Response

The Office of the President, working closely with its partners at the UC locations, has completed, posted & trained IT professionals on the guidelines for IT project development - a.k.a., IT Project Handbook - for all IT projects undertaken by any university location with a cost estimate of at least $5 million. This recommendation is covered in Chapter 4 entitled IT CONTRACT DELIVERABLE EXPECTATIONS, and there are guidelines established, examples provided and starter templates for establishing a process and creating a deliverable expectations document for each IT contract.

This work was completed and posted to a central location for all IT professionals to access in May 2018.

All locations are required to implement these guidelines. A training symposium was held at UC Berkeley in May 2018. Out of the symposium has come an IT project management community of interest, which has created a virtual collaboration space (Slack channel) and organized a second gathering of the community for UC Davis in August 2018. We expect the evolution and further dissemination of the IT Project Handbook will be an ongoing topic for the community.

http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Office of the President, working closely with its partners at the UC locations, has initiated the development of guidelines for IT project development - a.k.a., IT Project Handbook - all IT projects undertaken by any university location with a cost estimate of at least $5 million.

An outline has been agreed to by all parties. The chapters of the IT Project Handbook are bring written and leveraging industry best practices, guidelines published by the State of California, and leading practices from our UC locations.

Under the Chapter 4 entitled IT CONTRACT DELIVERABLE EXPECTATIONS, there are guidelines established, examples provided and starter templates for establishing a process and creating a deliverable expectations document for each IT contract similar that identifies the deliverables for each milestone and the required entrance criteria, acceptance criteria, and development schedule for each deliverable - as a prerequisite to approve vendor payment. The material is complete and posted at http://www.ucop.edu/information-technology-services/initiatives/itlc/it-project-reporting-and-guidelines.html

California State Auditor's Assessment of 6-Month Status: Partially Implemented

The Office of the President has drafted this chapter of its IT Project Handbook and made it available to the university locations. However, the Office of the President must still act on two elements of the recommendation which are: specifying that the guidelines apply to all IT projects with a cost estimate of at least $5 million and requiring the university locations to implement the guidelines, not merely review them. These two steps are key to the Office of the President ensuring that all significant IT projects are subject to the IT project development guidelines.

60-Day Agency Response

In consultation with the UC Regents, the University will use a threshold of $25M budget as the definition for significant IT projects for which new guidelines for IT project development will apply. The University will utilize existing "deliverable expectations documents" from existing significant IT projects to develop a guideline outlining what should be in the document and how it should incorporated into the UC-vendor contract template. Once finalized we will publish as part of the new guidelines for IT project development to be distributed to locations, centrally posted on our UC-IT portal and referenced in UC-IT staff and manager meetings.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Office of the President has consulted with the regents and decided on a project level threshold, but it has not developed and implemented the recommended guidelines for IT project development. In addition, we are concerned that the Office of the President is intending to apply a threshold that is too high, $25 million rather than the $5 million that we recommended, which could result in many significant IT projects not being subject to the IT project development guidelines.

All Recommendations in 2016-125.2

Agency responses received are posted verbatim.