California State Auditor Report Number : 2015-121

California Department of Veterans Affairs
The State Paid Nearly $28 Million for a Flawed System That Fails to Meet the Needs of Its Veterans Homes

Introduction

Background

The mission of the California Department of Veterans Affairs (CalVet) is to serve California’s veterans and their families. Further, the mission of CalVet’s veterans homes is to provide the aged and disabled veterans residing in the State’s eight veterans homes (residents) with rehabilitative, residential, and medical care and services in a homelike environment. The eight veterans homes are located in Yountville, Barstow, Chula Vista, Ventura, Lancaster, West Los Angeles, Fresno, and Redding. As of the end of 2015, they had the budgeted capacity to care for more than 2,300 veterans. Figure 1 shows the timeline of the opening of each of the homes.

Figure 1
Location of California Veterans Homes and Timeline of Opening Dates

 

 

CalVet’s veterans homes provide different levels of care to their residents, including domiciliary care, residential care for the elderly, intermediate nursing care, and skilled nursing care. In addition, CalVet provides on‑site outpatient clinics at all of its homes as a service to its residents. These clinics provide comprehensive, multidisciplinary assessments as well as ongoing primary care to address residents’ routine medical needs. Table 1 shows the levels of care available at each of the homes.

 

Table 1
Levels of Care Available at the California Veterans Homes

Level of Care	Locations of the Veterans Homes
	Yountville	Barstow	Chula Vista	Lancaster	Ventura	West Los Angeles	Fresno	Redding
Domiciliary care:
Minimum care and supervision for members who are able to perform all the activities of daily living.	✓	✓	✓
Residential care for the elderly:
Provides assistance and supervision to members in activities of daily living.	✓		✓	✓	✓	✓	✓	✓
Intermediate nursing care:
Provides skilled nursing supervision and supportive care to members on less than a continuous basis.	✓	✓
Skilled nursing care:
Provides skilled nursing and supportive care to members on an extended basis, including 24-hour inpatient care, with medical, nursing, dietary, pharmaceutical, and rehabilitation services, and an activity program.	✓	✓	✓			✓	✓	✓

 

History and Evolution of the Enterprise-Wide Veterans Home Information System

In late 2006, to support the health care information needs of its planned five new veterans homes and its three existing homes, CalVet sought approval for a new information system that could be used across its eight homes as well as in its headquarters. Former state policy required agencies wishing to launch an information technology (IT) project to prepare a feasibility study report (FSR) that describes the relative need for, cost of, and benefits to be derived from the proposed IT investment. The FSR for the proposed Enterprise‑Wide Veterans Home Information System (system) indicated that the new system would replace an existing implementation of Meditech, a commercial software product that CalVet had used since 1996 to support acute care, ambulatory care, and long‑term care operations at its homes. The FSR defined acute care services as hospital‑like services oriented toward clinically complex patients who need a high level of care; ambulatory care as care provided on an outpatient basis, similar to that provided by a physician’s office; and long‑term care as care delivered over a longer period of time and that includes skilled nursing home, rehabilitation, assisted living, and other services. The FSR stated that the primary reason for the limited use of Meditech was that the software lacked specific functionality for supporting long‑term care operations.

In its FSR, CalVet further indicated that considerable technical progress had occurred since the Meditech system was originally installed and that systems were available that could provide coverage across all care settings that CalVet offered. CalVet indicated that newer systems provided improved efficiency, better clinical documentation, and automated medication orders with edits to prevent common medication errors.

The proposed new system was intended to be a catalyst to transition CalVet from semi‑autonomous homes to an eight‑home integrated system of care. As shown in the text box, in its FSR CalVet described the key business opportunities to be realized from the new system.

According to the FSR, CalVet’s implementation of a new system presented an opportunity to take an affirmative step toward its vision of ensuring that veterans receive consistent and integrated care in any CalVet health care facility in California. It indicated that with the Meditech system, when a veteran transferred between facilities, staff at the new facility were unable to access the veteran’s medical history electronically. This lack of access to care information across facilities hampered clinicians’ ability to provide optimal care and led to increased costs, among other effects. For example, CalVet noted that patients’ tests might need to be repeated when information was not available. The new system was also intended to eliminate the need for cumbersome workarounds and unnecessary paperwork, to automate medication orders, to reduce the potential for medication errors, and to improve the quality of care veterans receive. The FSR estimated the cost of the system at nearly $34 million and gave a projected completion date of December 2010.

In January 2007, the California Department of Finance (Finance) approved CalVet’s FSR for the system. By the time CalVet issued its first special project report (SPR) for approval in August 2008, the project was already experiencing significant delays.¹ The FSR stated that CalVet would release the vendor request for proposal (RFP) in September 2007, but delays occurred in the review, validation, and updates to the original system requirements. Further, CalVet added two additional homes (Fresno and Redding) to the scope of the project. These changes, in part, required the creation of the first SPR. Additionally, CalVet took more than six months longer than planned to contract for independent project oversight (IPO) and independent verification and validation (IV&V) services. The SPR indicated that the new release date for the RFP would be November 2008, a delay of 14 months; however, the RFP was not actually released until February 2009, a further delay of three months.

Although the RFP was issued, it was canceled in June 2009, as described in the second SPR that CalVet submitted, because only one vendor submitted a proposal and that proposal did not address all of the required system functionality. A request for information (RFI) was issued in July 2009 to a broader vendor pool to determine their ability to meet CalVet’s system requirements. The outcome of the RFI indicated that none of the vendors could meet all of CalVet’s mandatory requirements; as a result, CalVet revised the RFP. For instance, according to the second SPR, the revised RFP listed significantly fewer mandatory requirements and contained a new “highly desirable” category of requirements that were not mandatory. The second RFP was released in November 2009. In December 2010, CalVet awarded the project to Solutions West Consulting, LLC (Solutions West; later Brekken Technology, Inc.) as its system contractor. The project was contained in two contracts. The first contract and two subsequent contract amendments of work included the purchase of system hardware and software, the installation and configuration of the system, and the training of CalVet staff on how to use the new system. The second contract provided for the post‑implementation maintenance and support services for the system by the system contractor. CalVet’s third and final SPR, in January 2013, adjusted the projected cost of the system to $36.7 million.

CalVet’s headquarters and its homes in Barstow, West Los Angeles, Lancaster, and Ventura started using selected modules of the system during the system’s pilot implementation in May 2012. The home in Chula Vista implemented and began using the system in June 2013. The final three homes—in Yountville, Fresno, and Redding—implemented and began using modules of the system in November 2013. The eight homes’ use of the system modules currently varies. According to the manager of the project management office (PMO manager) of CalVet’s Information Services Division (ISD), the system was not fully implemented and used at all of the homes because of certain functionality problems staff experienced with the system.

Although both CalVet and the California Department of Technology (Technology Department)—the department that now has responsibility for oversight of state IT projects—had identified project delays during the development and early implementation of the system, according to CalVet’s June 2015 response to questions from the Legislature regarding the project, new leadership in CalVet’s ISD reported critical project problems to the Technology Department in December 2013.² According to the final project manager for the system, CalVet identified these problems in late 2013 through a review of outstanding unpaid invoices, many of which lacked sufficient documentation to support the work performed. The June 2015 response document stated that the critical project problems CalVet identified related to project oversight, project management, contract management, contract deliverables, and the viability of the system contractor. The response document noted that to resolve these deficiencies, CalVet effectively suspended the project in December 2013 and, after a 12‑month period for reevaluation and remediation, terminated the implementation contract with the system contractor and a system support contract with the software vendor in December 2014. The response document also stated that CalVet and the Technology Department mutually agreed to close out the system project in January 2015. CalVet agreed to pay the system contractor $350,000 to settle the implementation contract. By the time CalVet reached its settlement agreement with the system contractor, it had spent $26.2 million, including the settlement costs. These costs had increased to $27.9 million when CalVet completed its post‑implementation evaluation report in June 2015. Figure 2 shows key milestones during the course of the project.

Figure 2
Enterprise-Wide Veterans Home Information System Project Timeline

Changing Roles and Responsibilities for Project Approval and Oversight

Both the Technology Department and the Department of General Services (General Services) have had certain responsibilities for overseeing the State’s IT project procurements, and as shown in Figure 3 on the following page, each entity’s roles have changed since January 2007 when Finance approved CalVet’s FSR for the system. Figure 3 also identifies the roles and responsibilities held by oversight agencies at key milestones during the development and implementation of CalVet’s system project. Effective January 2008, a change in state law transferred the authority to approve, suspend, or terminate IT projects from Finance to the Office of the Chief Information Officer—the predecessor entity to the Technology Department. In December 2010, when CalVet executed the contract to develop and implement the system, General Services had responsibility to review state agencies’ IT RFPs. Then state law effective in 2011 required General Services and the Technology Department to review all IT RFPs. Subsequent legislation operative in July 2013 transferred General Services’ share of this authority as well as General Services’ authority over IT project procurement to the Technology Department. This shift in responsibilities from General Services to the Technology Department, and the increase in the Technology Department’s responsibilities, reflects the Legislature’s conclusion that the unique aspects of IT projects and their importance to state programs warrant a separate acquisition authority.

Figure 3
Information Technology Roles and Responsibilities by Key Milestone for the Enterprise-Wide Veterans Home Information System Project

State policy requires departments to implement independent oversight for all reportable projects. The Technology Department, which assumed IT project oversight responsibility under state law effective in 2008, generally provides the IPO services while state entities undertaking the IT projects are responsible to contract for the IV&V services. IPO services provide an independent review and analysis of project management practices to determine if the project is being well managed. IV&V services provide a client with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and they also provide an early warning of process and technical discrepancies, issues, and problems that might not otherwise be detected until late in testing or implementation For its system, CalVet contracted with a single outside consultant (oversight contractor) to provide both IPO and IV&V services.

Scope and Methodology

The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to conduct an audit of the development and implementation of CalVet’s Enterprise‑Wide Veterans Home Information System. Table 2 outlines the audit committee’s audit objectives and our methods for addressing them.

 

Table 2
Audit Objectives and the Methods Used to Address Them

AUDIT OBJECTIVE		METHOD
1	Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, rules, and regulations. Reviewed relevant state policies regarding procurement and information technology (IT) projects.
2	Evaluate the level of oversight the California Department of Veterans Affairs (CalVet) management exercised regarding development and implementation of the Enterprise‑Wide Veterans Home Information System (system), including whether the executive office was involved in making key decisions and ensuring that the system met the needs of the veterans homes.	Identified requirements for oversight of development and implementation of IT projects. Obtained and reviewed documentation related to CalVet’s project management and other required plans to determine compliance with the California Project Management Methodology. Obtained and reviewed executive steering committee (steering committee) meeting minutes to determine the extent of oversight by CalVet’s management, and involvement and decision making by its executive office related to its system project. Obtained and reviewed independent project oversight (IPO) and independent validation and verification (IV&V) reports prepared by CalVet’s oversight contractor to determine whether IPO and IV&V services were provided over the course of the system project and that they adhered to state policy and contractual requirements. Our IT expert identified requirements traceability monitoring as an important function of IV&V. Because he did not see evidence that this function was performed by CalVet’s oversight contractor, we obtained and reviewed invoices approved for payment and requested that CalVet provide us with evidence of an accepted deliverable for proof that this work was performed. Interviewed key staff at CalVet to understand its oversight role.
3	Assess whether CalVet followed laws, rules, regulations, policies, and best practices when selecting vendors for the system, including, to the extent possible, those prohibiting a conflict of interest during the selection process.	Identified relevant laws, regulations, and policies regarding the selection of the contractor to implement CalVet’s system. Obtained documentation to confirm CalVet’s request for proposals (RFP) was properly advertised in the California State Contracts Register. Requested and reviewed the proposals CalVet received in response to its RFP for implementation of its system and documentation for the evaluations of the proposals. Requested and reviewed CalVet’s Statement of Incompatible Activities (SIA) and Conflict‑of‑Interest Code. Requested and reviewed the California Fair Political Practices Commission’s Statements of Economic Interests (Form 700) for applicable staff that participated in the selection of the system contractor. The forms we obtained did not identify any reported conflicts. Requested and reviewed CalVet’s certified SIAs for members of the committee selecting the contractor implementing the system. The forms we obtained did not identify any reported incompatible activities. Interviewed key individuals at CalVet to determine whether any additional methods existed at CalVet to determine potential conflicts of interest.
4	Determine the estimated and actual implementation costs and timeline for the system as well as the number of and reasons for change orders and contract amendments.	Reviewed documentation to determine the approved project cost estimates. Obtained California State Accounting and Reporting System (CalSTARS) accounting records to identify the project’s total cost, as reported in CalSTARS. - Estimated and actual implementation costs are located in Table A.1 in Appendix A.  Reviewed project documentation to determine the estimated project implementation timelines and the actual implementation timeline. - Estimated and actual implementation timelines are located in Table A.2 in Appendix A.  Reviewed the two contract amendments to determine the reason for each. - The first amendment increased the contract costs to include unanticipated tasks—as permitted by the contract—and made minor changes or clarifications to the scope of work. - The second amendment extended the contract by one year, updated the statement of work to reflect the system contractor’s name change, and updated the contact information for the project site coordinators. Reviewed the list of change requests and determined that there were 304 approved change requests. We reviewed a total of 64 change requests to determine the reason for the changes including all change requests categorized as a significant change, a portion of those categorized as a minor change, and a portion of those that were uncategorized. We identified five main reasons for the change requests we reviewed: - Contract language update or clarification. - Deliverable or requirement change. - Requirement start or due date change. - System enhancement or software change. - Business process change.
5	Determine whether the original project requirements, as defined by the scope of work, were delivered during implementation of the system project.	Interviewed key staff at CalVet to determine whether the system contractor met all contract requirements. Compared the original statement of work to each amended statement of work to determine what changes were made to the project requirements.   Reviewed change requests to determine how those changes affected the original requirements. Reviewed selected contract deliverables documentation (that is, deliverable expectation documents and deliverable acceptance documents) for the 33 deliverables to assess whether the system contractor met the project requirements, as defined by the statement of work. Reviewed CalVet’s post‑implementation evaluation report and steering committee minutes to determine the cause for any gaps between the contract requirements and the deliverables documentation.
6	Evaluate the steps CalVet took when project variances were identified. To the extent possible, determine whether CalVet could have identified problems with the system earlier.	Reviewed CalVet’s executive summary reports, project status reports, special project reports, and IPO and IV&V reports to determine when CalVet identified project variances and what steps Cal Vet took to address those variances. Interviewed key staff at CalVet. Provided the IPO and IV&V reports to our IT expert for his professional opinion as to whether CalVet should have identified problems with the system earlier.
7	Review the role of the California Department of Technology (Technology Department) in this project and evaluate whether it fulfilled its roles and responsibilities.	Reviewed relevant laws and policies to identify the Technology Department’s oversight responsibilities throughout the life of the system project. Interviewed key individuals at the Technology Department to determine the Technology Department’s role throughout the life of the system project. Requested and reviewed documentation from the Technology Department to determine the extent of oversight it provided to CalVet and whether it fulfilled its oversight responsibilities.
8	Determine the steps CalVet has taken to improve the functionality of the system and what affect any lack in functionality has had on quality of care and cost and efficiency of the homes’ operations. In addition, determine whether CalVet has documented lessons learned for use in future phases of system implementation.	Interviewed key project management at CalVet headquarters and the homes and reviewed available documentation to gain an understanding of the following: - The system’s lack of functionality. - The impact the lack of functionality has on the homes’ quality of care and cost and efficiency. - Steps CalVet has taken to address the lack of functionality. Reviewed documentation related to lessons learned to determine what changes, if any, CalVet made as a result of the lessons learned.
9	Identify the level of system functionality and use within CalVet’s veterans homes and administrative offices and evaluate CalVet’s efforts to train staff and otherwise address resistance to using the system.	Interviewed key staff at CalVet and the eight veterans homes. Obtained a list of system functionality implemented and used at headquarters and each veterans home. We asked staff at headquarters and each home to verify their use of system components. Reviewed training documentation for a selection of training course evaluations from CalVet and found generally positive comments about the training. We followed up with staff at the homes who mostly expressed positive comments about the training. Reviewed CalVet’s organizational change management strategy to address resistance to the system brought about by organizational change. We followed up with staff at the homes and found that, in general, staff indicated that there was buy in to the system.
10	Review and assess any other issues that are significant to the audit.	Interviewed key staff at the Technology Department to understand its facilitation of the contract disputes between CalVet and the system contractor. Obtained and reviewed documentation to assess whether the Technology Department’s actions as the dispute facilitator were in the best interests of the State.

 

Assessment of Data Reliability

In performing the audit, we obtained financial records from the California State Accounting and Reporting System (CalSTARS) for CalVet’s expenditures related to its system. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer‑processed information that we use to support our findings, conclusions, and recommendations. Specifically, we used expenditure reports from CalSTARS from July 2007 to June 2015. For each fiscal year, we used these data to report CalVet’s total expenditures for its system implementation. However, we did not conduct accuracy or completeness testing on these data because, in accordance with its records retention policy, CalVet did not maintain source documents for many of its historical transactions that we would need to test. As a result, any testing we performed would be incomplete. Further, CalVet stores the source documents it does maintain at locations throughout the State, making this testing cost‑prohibitive. Thus, we determined that the State’s CalSTARS data were of undetermined reliability for the purposes of this audit. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations. To gain some assurance of the accuracy of CalVet’s system transactions recorded in CalSTARS, we tested a selection of expenditures and hours worked for fiscal years 2013–14 and 2014–15, and we determined that CalVet accurately recorded the transactions we reviewed.

 

---

Footnote

¹ An SPR provides a summary of proposed changes to the original project cost, schedule, or scope. It is generally required when the project cost or total financial program benefits deviate or are anticipated to deviate from the original by 10 percent or more, or when a major change occurs in project requirements or methodology. Go back to text

² Until July 2013 the Department of Technology was known as the California Technology Agency, and before that, it was the Office of the Chief Information Officer. Go back to text

Back to top