Appendix A

Comparison of the California Department of Veterans Affairs’ Estimated and Actual Costs for the Enterprise‑Wide Veterans Home Information System

The Joint Legislative Audit Committee asked us to determine the estimated and actual costs, as well as the estimated and actual timelines, for the California Department of Veterans Affairs’ (CalVet) Enterprise‑Wide Veterans Home Information System (system) project. Table A.1 on the following page presents CalVet’s estimated and actual costs for implementing this system.

As shown in Table A.1, CalVet initially estimated the costs to implement the system in its feasibility study report (FSR) at just under $34 million. CalVet increased its estimate as the project progressed. However, the actual costs of implementation, as of the completion of its post‑implementation evaluation report in June 2015, were lower than its estimate, at roughly $27.9 million. Although the cost of the project was nearly $9 million less than the last approved special project report (SPR), CalVet implemented much less functionality than it initially planned for and that was approved in its FSR and subsequent SPRs. Specifically, as discussed, because of the functionality problems that staff encountered during implementation of the system at the first few veterans homes, CalVet indicated that staff either did not receive training or received training on a minimal number of components for the ADL Clinical module functionalities at the Fresno, Redding, and Yountville homes; the ADL Clinical module is a main module of the system through which the homes manage electronic health records. According to their administrators, two of the homes continue to use paper processes to document resident medical information, while a third home uses the previous system, Meditech. Therefore, CalVet spent $27.9 million for implementation of a system that has not been fully implemented at all of the homes, and it did not fulfill one of its primary goals for the project—a fully integrated system of care in which the system used at any home could seamlessly operate with the other homes and in which veterans would receive consistent care throughout the homes—and therefore, does not meet the needs of the veterans homes.

Table A.1
Estimated and Actual Costs for Implementation of the California Department of Veterans Affairs’ Enterprise-Wide Veterans Home Information System
Project Document	Document Date	Estimated Costs	Estimated Completion Date
Feasibility Study Report	December 2006	$33,982,315	December 2010
Special Project Report #1	August 2008	35,469,331	June 2012
Special Project Report #2	November 2010	37,311,098	October 2013
Special Project Report #3	October 2012	36,744,638	April 2014
		Actual Costs	Actual Completion Date
California State Accounting and Reporting System Reports (CalSTARS)	NA	27,914,733*	June 2015^†

Sources: Documents referenced in the table column titled Project Document and the respective approval letters, if applicable, for these documents.

NA = Not applicable.

* Costs recorded in CalSTARS are from July 1, 2007, through June 30, 2015. Costs include CalVet’s and the system contractor’s settlement agreement amount of $350,000.

^† In June 2015 the California Department of Veterans Affairs (CalVet) submitted its post-implementation evaluation report to the California Department of Technology in support of its request to consider the project complete and to terminate project reporting.

Comparison of CalVet’s Estimated and Actual Timelines to Implement the System

Table A.2 presents CalVet’s estimated and actual dates for completing major project milestones to implement the system. CalVet initially estimated the project schedule in the FSR and updated the estimated dates in the subsequent SPRs. CalVet extended the overall project schedule in each of the SPRs. Specifically, although CalVet originally expected to complete the system (final acceptance) by December 2010, the system was still not complete in January 2015, when, as we described in the Audit Rsesults, CalVet and the California Department of Technology agreed to close the project. Several factors contributed to project delays. For instance, CalVet’s initial request for proposal (RFP) was delayed because of changes that were made to the project scope, such as the decision to implement the system at the Fresno and Redding homes. In addition, CalVet issued a second RFP with updated system requirements after its initial RFP failed to generate any responsive proposals. CalVet also delayed the implementation of the system at the Chula Vista home to provide additional support to some of the pilot homes that were struggling with certain complex functionalities of the system. Finally, CalVet postponed the implementation of the system at the Fresno and Redding homes because of delays in the opening of those homes.

Table A.2
California Department of Veterans Affairs’ Enterprise-Wide Veterans Home Information System Timeline Comparison
	Estimated Completion Dates
Major Milestones	Feasibility Study Report December 2006	Special Project Report #1 August 2008	Special Project Report #2 November 2010	Special Project Report #3 October 2012	Actual Completion Dates
Solution procurement	March 08	June 09	December 10	January 201	December 2010*
User acceptance testing	March 09	April 10	September 11	June 12	June 2013*
Implementation at headquarters	NI	NI	January 2012†	August 2012†	May 2012*
Implementation at the Veterans Homes:
Barstow	September 10	March 12	January 2012†	August 2012†	May 2012*
West Los Angeles	April 10		August 2012
Lancaster	April 09	June 2010†	May 2012
Ventura	April 09		July 2012
Chula Vista	July 2010	January 2012	March 12	March 13	June 2013‡
Redding	NI	April 11	July 13	November 2013	November 2013§
Fresno	NI	July 11	September 13	November 2013
Yountville	December 10	June 12	April 13	November 2014
Final acceptance	December 10	June 12	October 2013	November 2014	January 2015ll
Post-implementation evaluation report (PIER)	NI	June 13	October 2014	November 2014	June 15

Source: California Department of Veterans Affairs’ (CalVet) feasibility study report, special project reports (SPR), the PIER, Enterprise-Wide Veterans Home Information System Implementation Contract (implementation contract), and Enterprise Wide Veterans Home Information System deliverable acceptance documents.

I = Not indicated.

* Dates differ from those presented in CalVet’s PIER. The date for solution procurement differs because CalVet used the effective date of its implementation contract, as noted in SPR 3, and we used the date that the Department of General Services approved the contract. The date for user acceptance testing differs because CalVet used the date the system contractor submitted the corresponding deliverable, as noted in SPR 3, and we used the date that CalVet accepted the deliverable. The date for the implementation differs because CalVet was unable to provide supporting documentation for the dates it used. Therefore, we relied on the date that CalVet accepted the deliverable.

^† In SPR 1, CalVet determined it would implement the system in the homes located in West Los Angeles, Lancaster, and Ventura as a pilot implementation to validate the system before statewide roll out. In SPR 2, the pilot implementation was reduced to include CalVet headquarters and the home located in Barstow. In SPR 3, the pilot implementation was expanded to include headquarters and the homes located in Barstow, West Los Angeles, Lancaster, and Ventura.

^‡ Because CalVet did not have a deliverable acceptance document for the implementation in the home in Chula Vista, we relied on the dates CalVet listed in its PIER.

^§ According to the PIER, the implementation at the homes located in Fresno, Redding, and Yountville was not fully completed. Specifically, while the financial modules were implemented, the clinical modules—by which the homes manage electronic health records—were never implemented at these homes.

^ll In January 2015, CalVet and the California Department of Technology mutually agreed to close the Enterprise-Wide Veterans Home Information System project.

Appendix B

An Expanded View of Enterprise‑Wide Veterans Home Information System Use by the California Department of Veterans Affairs’ Veterans Homes

The Joint Legislative Audit Committee asked us to determine the level of Enterprise‑Wide Veterans Home Information System (system) use within the California Department of Veterans Affairs’ (CalVet) veterans homes. Table 3 on page 23 provides an overview of when CalVet implemented the system at its headquarters and each of the eight veterans homes as well as the extent of the system modules in use at each of those locations. Table B provides an expanded view of those data.

Table B
Enterprise-Wide Veterans Home Information System Module Use by Location and Date of Implementation
Components	Headquarters, May 2012	Barstow, May 2012	Chula Vista, June 2013	Fresno, November 2013	Lancaster, May 2012	Redding, November 2013	Ventura, May 2012	West Los Angeles, May 2012	Yountville, November 2013
ADL Clinical Software designed to address critical functions of providing long-term care by capturing and managing patient information.
1. Rehabilitation (Rehab) Touch	NA	F	F	NI	NA	F	NA	F	F
2. Rehab Editor	NA	F	F	NI	NA	NI	NA	F	F
3. Care Plans	NA	F	F	NI	F	NI	F	F	NI
4. Virtual Body	NA	P	P	NI	F	NI	P	N	NI
5. Certified Nursing Assistant (CNA)	NA	F	N	NI	F	NI	F	F	NI
6. CNA Reports	NA	F	N	NI	F	NI	P	P	NI
7. CNA Assignments	NA	F	P	NI	F	NI	F	F	NI
8. Progress Notes	NA	F	F	NI	F	NI	F	F	NI
9. Shift Reports	NA	N	N	NI	F	NI	F	N	NI
10. Assessments	NA	P	P	NI	F	NI	F	P	NI
11. Rehab Assessment	NA	F	N	NI	NA	NI	NA	F	NI
12. Vitals	NA	F	N	NI	F	NI	F	F	NI
13. Order Entry	NA	P	N	NI	N	NI	N	N	NI
14. Electronic Medication Administration Record (eMAR)/ Electronic Treatment Authorization Request (eTAR)	NA	N	N	NI	N	NI	N	N	NI
15. Minimum Data Set (MDS)	NA	F	F	F	NA	F	NA	F	F
16. Clinical Dashboards	N	P	N	NI	P	NI	P	N	NI
17. Quality Assurance	NA	NA	NA	NI	NA	NI	NA	NA	NI
18. Infection Control	NA	N	F	NI	F	NI	F	P	NI
19. Allergy Management	NA	F	F	NI	P	NI	F	F	NI
20. Diagnosis Management	NA	F	F	NI	F	NI	F	F	P
21. Facility Reports— Clinical/Order Reports	F	P	P	NI	P	NI	F	P	NI
22. Precautions	NA	F	F	NI	N	NI	F	P	NI
23. Forms	N	F	P	NI	P	NI	F	P	NI
ADL Financial Software used to support third-party billing, Medi-Cal, Medicare, accounts receivable, transaction history, withdrawals and charges; making, tracking, and controlling financial events.
24. Admissions, Discharge, Transfers (ADT) Registration	NA	F	F	F	F	F	F	F	F
25. ADT History	F	F	F	F	P	F	P	F	F
26. Census Actions	NA	F	F	F	P	F	F	F	F
27. Census Control	NA	F	F	F	P	F	F	F	F
28. ADT Corrections	NA	F	F	F	N	F	F	F	F
29. Trust Funds	NA	F	F	F	F	F	F	F	F
30. Cash Receipts	NA	F	F	F	P	F	F	F	F
31. Cash Receipts— Veterans Affairs (VA) Per Diem	P	P	P	P	P	F	P	P	F
32. Online Receipts	NA	F	F	F	F	F	F	N	F
33. Batch Receipts	NA	F	F	F	F	F	F	F	F
34. Rating Calculation	NA	F	F	F	F	F	F	F	F
35. Paysources	F	F	F	P	P	P	F	P	F
36. Medicare	NA	F	F	P	P	P	F	P	F
37. Insurances	F	F	F	F	N	F	F	F	F
38. Rehab Module	NA	F	F	NI	N	NI	N	N	F
39. Ancillary Charges	NA	F	F	P	N	P	F	P	F
40. Professional	F	F	F	N	N	N	N	N	F
41. Wait List	NA	F	F	F	P	F	F	F	F
42. Billing	F	F	F	N	P	N	F	N	F
43. Pre-Billing	P	P	P	F	N	F	F	P	P
44. Off Hours Admissions by Nurses	NA	F	F	F	F	F	F	F	F
45. Forms	NA	F	F	F	F	F	F	F	NI
46. Dashboards	F	P	P	F	N	P	N	P	NI
47. Immunizations	NA	F	P	NI	F	NI	F	F	NI
48. VA Per Diem Charges	P	F	F	F	P	F	F	F	F
49. VA Per Diem Cash Receipts	F	N	N	N	N	F	N	N	F
50. Veterans	P	F	F	F	P	F	F	F	F
51. Ledgers	F	F	F	F	F	F	F	F	F
52. Medical Diagnosis	F	F	F	F	F	F	F	F	F
53. Reports	F	F	F	F	F	F	F	F	F
54. MDS Data	P	F	F	P	NA	P	NA	P	F
55. Recurring Charges	N	N	N	N	N	N	N	N	N
56. Errors Tracking	P	F	F	F	F	F	F	F	F
57. TAR Authorizations- Medi-Cal TARs	P	F	F	N	NA	N	NA	N	P
58. Expected Income and Budget	F	N	N	N	N	N	N	N	N
59. End Period Process	F	F	F	F	F	F	F	F	F
Dynamics Financial accounting and business management software that automates creation and management of accounting data and workflow. Used to augment ADL Financial to support financial management. The specific component used is for purchasing and inventory.
60. Purchasing and Inventory	N	F	P	F	F	F	F	F	F
GeriMenu Software used for resident meal planning to help facilities run their nutrition departments more efficiently.
61. Resident Meal Planning	NA	P	P	F	N	NI	N	F	P
Documentum An enterprise content management system used to house and manage electronic documents.
62. Document Management	N	F	N	NI	N	NI	F	F	NI
Framework LTC Pharmacy management software designed for long-term care and institutional facilities.
63. Pharmacy Management	NA	F	F	NI	N	NI	F	F	F

Source: California Department of Veterans Affairs’ (CalVet) system use chart provided by the system final project manager, and confirmed or updated by staff at headquarters and the eight homes, draft system architecture design document, and the CBORD Group Inc. Gerimenu website, and system contractor’s response to the request for proposals.

F = The veterans home indicated it fully used this module.
P = The veterans home indicated it partially used this module.
N = Although CalVet implemented the module, headquarters or the veterans home indicated it did not use it.
NI=CalVet did not implement the module at the veterans home.
NA= Although these modules were available for use, staff at headquarters or the homes indicated the components marked NA were generally not applicable for their operations.