Introduction

Background

Government websites are a popular means by which members of the public obtain information and avail themselves of government services. In a 2010 report on the use of online government services and information, the Pew Research Center stated that in 2009 nearly half of American Internet users searched online to identify services provided by a government agency. Specific to California, the Public Policy Institute of California reported in 2013 that 47 percent of Californians said they use the Internet to access government resources, an increase of 4 percent from 2008. Finally, data featured on the California state government home page showed that in January 2015 there were millions of unique visits to California government websites, such as those for the California Department of Motor Vehicles and the Employment Development Department.

However, not all persons interact with and experience the web in the same way. To ensure that persons with disabilities can access web content in a manner comparable to those without disabilities, the federal government and a nongovernmental organization have developed web accessibility standards that address the needs of users who may have one or more of a range of disabilities. For example, the standards encompass the needs of users with visual impairments, hearing impairments, and physical disabilities. These impairments affect many Californians. The results of the 2013 American Community Survey, an ongoing survey conducted by the U.S. Census Bureau, indicate that approximately 4 million Californians live with a disability. Further, the American Foundation for the Blind reported that in 2013 about 789,000 Californians suffered from vision loss. The National Institutes of Health estimates that 17 percent of American adults report some form of hearing loss, which translates to almost 5 million Californians.

California’s Requirements for Accessible Websites

Since January 2003 state law has required all state governmental entities to comply with Section 508 of the federal Rehabilitation Act of 1973, as amended (Section 508), and its implementing regulations. Section 508 regulations mandate that electronic and information technology (EIT) that federal agencies develop, procure, maintain, or use must be accessible to persons with disabilities and requires that persons with disabilities have access to and use of data that is comparable to the access and use by those without disabilities. Federal regulations establish the specific technical requirements (Section 508 standards) that EIT must meet in order to ensure that it is accessible to persons with disabilities. Section 508 standards are similar to, but distinct from, the physical accessibility standards established by the federal Americans with Disabilities Act of 1990, as amended (ADA). Along with other, more general, requirements for overall accessibility, ADA regulations require all public agencies to communicate as effectively with people with disabilities as they do with others; Section 508 standards focus specifically on the accessibility of particular categories of electronic products, such as software, websites, telecommunications and multimedia products, and some physical products, such as stand‑alone terminals. This audit is focused on the compliance of web-based products developed or procured by four state departments with Section 508 and related state web accessibility standards, which we refer to jointly as state web accessibility standards.

In addition to adopting the Section 508 standards in 2003, California further added new standards for accessible websites in June 2006. To address a goal of making government services more accessible to citizens, the former state chief information officer convened the California State Portal Steering Committee (steering committee) in 2005 to guide the development of a new state web portal.³ As part of that effort, the steering committee created the Information Organization, Usability, Currency, and Accessibility Working Group (IOUCA) in 2006 to develop policy and best-practice recommendations to ensure that state websites are accessible, usable, and understandable. In its recommendations, the IOUCA advised that California should go beyond the Section 508 standards and also adopt accessibility standards developed by the World Wide Web Consortium (W3C), an international nongovernmental organization. These standards are known as the Web Content Accessibility Guidelines (WCAG). In July 2006 the steering committee adopted the IOUCA recommendations and made the first version of the WCAG standards—WCAG 1.0—a requirement for California departments and agencies reporting to the governor and the state chief information officer. Figure 1 shows the timing of the release of various web accessibility standards as well as California’s adoption of some of those standards.

The Section 508 standards were issued by the Architectural and Transportation Barriers Compliance Board (Access Board), a federal agency that, according to its website, promotes equality for people with disabilities through leadership in accessible design and the development of accessibility guidelines and standards for the built environment as well as for information technology. When it issued the Section 508 standards, the Access Board indicated that some of the standards could be met by complying with specific WCAG 1.0 standards. However, the Access Board determined that some WCAG 1.0 standards would be difficult to enforce or were not clearly defined and it, therefore, did not adopt equivalent federal standards. Nevertheless, in recommending that California adopt the WCAG 1.0 standards, the IOUCA determined that the WCAG 1.0 guidelines were important to provide a greater level of usability for the State’s citizens. When the W3C later published the WCAG 2.0 standards, it stated that it designed the updated guidelines to build on the WCAG 1.0 standards, apply to future technology, and be testable with a combination of automated testing and human evaluation. Additionally, the WCAG 2.0 standards encompass Section 508 standards that WCAG 1.0 did not previously address. Table 1 shown below provides examples of some accessibility standards to demonstrate the relationship between the Section 508 standards and WCAG 1.0 and 2.0 standards.

Figure 1

Timeline of the Adoption of Website Accessibility Standards by the Federal Government and California

Sources: 1998 amendment to the Federal Rehabilitation Act of 1973; California Statutes of 2002, Chapter 1102; California Office of the State Chief Information Officer, Information Technology Policy Letter 10-10; Information Organization, Usability, Currency, and Accessibility Working Group, Recommendation on Accessibility Standards for California State Web Pages; W3C guidelines and website.

* The Information Organization, Usability, Currency, and Accessibility Working Group.

^† At the time the policy letter was written, the department was known as the Office of the State Chief Information Officer. Later it was reestablished as the California Technology Agency, and in 2013 it became the California Department of Technology.

California’s approach to implementing web accessibility standards generally mirrors the federal approach. At the federal level, individual federal departments and agencies are responsible for ensuring that their websites meet accessibility requirements. State law also requires individual state government entities to comply with applicable accessibility standards and does not identify a department specifically responsible for statewide oversight or enforcement of the accessibility standards. A separate state law assigns general responsibility for information technology oversight and enforcement of most state departments to the California Department of Technology (CalTech). However, this statute does not expressly state that CalTech is responsible for ensuring that all state-owned EIT is accessible to persons with disabilities. Nevertheless, despite not having explicit responsibility for ensuring Section 508 compliance at state departments, CalTech published a policy letter in July 2010 that reinforced the IOUCA recommendations for accessible websites and has devoted a chapter of the Statewide Information Management Manual (SIMM) to information technology accessibility, highlighting the legal requirements for state governmental entities as well as resources that departments can access to help ensure compliance.⁴

Table 1
Comparison of Selected Accessibility Standards and Guidelines
Standard	Web Content Accessibility Guidelines (WCAG) 1.0 (nongovernmental standards issued in 1999)	Section 508 Standards (federal standards issued in 2000, effective 2001)	WCAG 2.0 (nongovernmental standards issued in 2008)
Websites use the clearest and simplest language appropriate	Yes	No*	Yes^†
Users may skip certain repetitive material	No	Yes	Yes
Web pages have titles that describe their topic or purpose	No	No	Yes

Sources: California State Auditor’s analysis of the federal Architectural and Transportation Barriers Compliance Board’s (Access Board) Preamble to the Section 508 Standards; World Wide Web Consortium’s Web Accessibility Initiative and WCAG 1.0 and 2.0; and information presented at the 2009 Annual International Technology and Persons with Disabilities Conference.

* The Access Board determined that although this WCAG 1.0 standard is a worthwhile guideline, it is difficult to enforce because requiring the simplest language can be very subjective.

^† WCAG 2.0 specifically guides web content developers to define words or phrases used in an unusual or restricted way, including jargon, and establishes the lower secondary education level as the maximum reading level required for users.

Federal regulations and best-practice guidance documents go beyond technical accessibility standards for websites to provide direction regarding the procurement of accessible EIT and how departments should address complaints that EIT is inaccessible. Federal regulations require departments that procure EIT products to conduct market research and, as part of that research, to evaluate the availability of products that meet federal accessibility standards. Federal and state guidance also recommends that departments test EIT products for compliance during development and as part of ongoing maintenance. Further, the SIMM states that departments should have processes in place for collecting and addressing complaints about inaccessible EIT. Finally, at both the federal and state levels there is guidance concerning responsibilities departments should meet to ensure the accessibility of their EIT.

How Persons With Disabilities Use the Web

People with disabilities access and navigate the web in different ways, depending on their individual needs and preferences. Some common approaches for interacting with the web include assistive technologies and adaptive strategies. Assistive technologies are software or hardware that people with disabilities use to improve interaction with the web. These include screen readers that read aloud web pages for people who cannot read text, screen magnifiers for people with some types of low vision, and voice recognition software and selection switches for people who cannot use a keyboard or mouse. Adaptive strategies are techniques that people with disabilities use to improve interaction with the web, such as increasing text size, reducing mouse speed, or turning on captions. Figure 2 shown below displays techniques persons with disabilities may use to interact with websites and also shows elements of a website that, if properly coded, will assist persons with certain disabilities in navigating and using that website.⁵

According to the W3C, several different components of web development and web interaction must work together in order for web content to be accessible to persons with disabilities. These components include the content of a website, such as the text, the images, and the code that defines the structure of the site; the web browsers and media players that users employ; assistive technologies; and the knowledge and experience of the user. In addition, the W3C highlights the importance of the website developers and the tools they use to author content and validate that the content they create is accessible.⁶

Figure 2

How Persons With Disabilities Generally Use the Web

Sources: California State Auditor’s analysis of reports issued by the World Wide Web Consortium, Web Accessibility Initiative; Web Content Accessibility Guidelines 1.0; Section 1194 of Title 36 of the Code of Federal Regulations; the Web Style Guide, 3rd Edition; and an interview with our web accessibility consultant (consultant).

Note: Our consultant did not perform an accessibility analysis of the State of California website shown above; it is being used for demonstration purposes only.

Departments and Web Products Selected for This Audit

Our review included four state departments and web-based products that they procured or developed. Table 2 shows the departments and the associated web products that we selected for review. To define a web product at each department, we identified a service that each department offers online and reviewed department web pages that support those services. Further, at each department we reviewed the compliance of additional web pages on which accessibility-related information was located when they were present at the time of our review. Specifically, we reviewed each department’s page describing how to submit a complaint related to website accessibility. As a result, our review did not include a comprehensive examination of any department’s entire web presence.

Table 2
Profiles of Web-Based Services We Reviewed
Department	Web-Based Service	Key Component of the Web-Based Service*	Was the Key Component Procured or Developed In-House?	Year Implemented
California Community Colleges (Community Colleges)	Apply to college	OpenCCCApply	Procured from vendor^†	2012
California Department of Human Resources (CalHR)	Establish eligibility for state employment	Careers in California Government Portal	Developed in-house	2012
Covered California	Apply for health insurance	California Healthcare Eligibility, Enrollment, and Retention System	Procured from vendor^†	2013
State of California Franchise Tax Board (Franchise Tax Board)	File state tax return	CalFile	Developed in-house	2003

Source: California State Auditor’s analysis of websites operated by Community Colleges, Covered California, CalHR, and Franchise Tax Board; procurement information obtained from Community Colleges and Covered California; and interviews with staff at all four departments.

* In addition to reviewing the compliance of the key component associated with the web-based service each department offers, we reviewed the compliance of additional web pages at each department. These included other pages that were in support of the web-based service and the department’s accessibility information page.

^† According to officials at these departments, the key component was procured from a vendor but was not a commercial off-the-shelf product. Instead, vendors built custom products for these departments.

Some departments developed the product we selected for review using their internal staff, while others, specifically Covered California and California Community Colleges (Community Colleges), procured their web-based products from outside vendors. Community Colleges’ Chancellor’s Office (Chancellor’s Office) has an agreement with one of its college districts, the Butte-Glenn Community College District (Butte), wherein Butte receives grant funding from the Chancellor’s Office to operate the California Community Colleges Technology Center (technology center). Through the technology center, Butte procured the OpenCCCApply website from a third-party vendor. According to the assistant project director for Covered California’s California Healthcare Eligibility, Enrollment, and Retention System project, Covered California procured the product from an outside vendor that still maintains the system for Covered California. Conversely, according to a supervisor of the Internet and taxpayer folder section, the State of California Franchise Tax Board developed its online tax filing system—CalFile—internally using staff developers. Finally, the supervisor in the web development and support unit at the California Department of Human Resources (CalHR) stated that staff who built the Careers in California Government web portal and related applications were CalHR staff or former employees at the California State Personnel Board. The site and the staff responsible for it were transferred to CalHR in the governor’s reorganization in July 2012.

Scope and Methodology

The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to review the State’s compliance with applicable laws, regulations, policies, and best practices for making EIT accessible to persons with disabilities. Table 3 lists the objectives that the audit committee approved and the methods used to address those objectives. As already described, our fieldwork concerned four state departments and web products developed or procured by those departments.

Table 3
Audit Objectives and the Methods Used to Address Them
Audit Objective	Method
1. Review and evaluate the laws, rules, and regulations significant to the audit objectives.	Reviewed relevant laws, regulations, and other background materials applicable to web accessibility.
2. Determine the roles and responsibilities within state government for ensuring compliance with electronic and information technology (EIT) accessibility requirements by determining the roles and responsibilities that a selection of departments have for ensuring compliance with Section 508 standards and related laws, regulations, policies, and best practices when EIT is developed or procured.	Reviewed state laws and policies to identify existing roles and responsibilities for statewide Section 508 compliance. Interviewed staff at the California Department of Technology (CalTech) to determine its role in statewide compliance. At each of the four departments we reviewed under Objective 5, identified key individuals responsible for ensuring Section 508 compliance at the department and interviewed them regarding their responsibilities.
3. Review and assess the effectiveness of any guidance, policies, or protocols the State uses to ensure, monitor, and enforce compliance with EIT accessibility requirements. Determine whether such guidance, policies, or protocols are consistent with best practices, if any, at the federal, state, or local level for providing accessible EIT.	Identified best practices at the federal and state levels for providing accessible web-based EIT and compared those best practices to state policy and guidance. We identified no best practices at the local level. Determined that state guidance was generally consistent with the best practices that we identified. Determined whether the accessibility standards included in state policy were the most up-to-date standards issued by the World Wide Web Consortium (W3C). Interviewed staff at each department we reviewed under Objective 5 to identify the sources of web accessibility guidance they use.
4. Determine whether the State provides personnel involved in procuring or developing EIT adequate training and whether the State has a process for investigating complaints related to noncompliance with Section 508.	Interviewed staff at each department we reviewed under Objective 5 and reviewed training documents to determine whether department staff received training related to Section 508. Interviewed staff at CalTech and the California Department of Rehabilitation to determine whether they provide web accessibility training and, if so, identified the type of accessibility training. Determined that the State has no centralized entity responsible for addressing web accessibility complaints and reviewed the complaint process for each department under Objective 5(f).
5. Identify EIT products (web pages and web-based services) administered by a selection of state agencies and select a number of EIT products to review. To the extent possible, the products selected should include both EIT developed, hosted, and/or administered by third-party contractors as well as EIT developed and/or hosted by state agencies; EIT developed or procured within the last two years as well as those from prior years; and EIT offered by both large and small state agencies. For each of the EIT products selected, determine whether:	Identified key services offered online by state departments and considered the web traffic on related websites and the size of the department offering each service to choose four services for review. We selected applications for community college, applications for health care insurance, application and exam processes for state employment, and filing state income tax returns as our key web-based services. The corresponding departments were California Community Colleges (Community Colleges), Covered California, the California Department of Human Resources (CalHR), and State of California Franchise Tax Board (Franchise Tax Board), respectively. For each online service, we identified the web product that was most closely associated with the service and reviewed that product as described under subobjectives (a) through (f).
a. The product complies with Section 508 standards and any other relevant state guidance, such as the State Administrative Manual or the Statewide Information Management Manual, or related laws, regulations, policies, and best practices.	Retained a web accessibility consultant (consultant), Deque Systems, Inc., to determine whether each web product selected under Objective 5 complied with key criteria and to assess the severity of errors against those criteria. Covered California is not subject to the policies issued by the state chief information officer and as such would not be required to follow Web Content Accessibility Guidelines (WCAG) 1.0 as were other departments we reviewed. We assessed Covered California’s California Healthcare Eligibility, Enrollment, and Retention System (CalHEERS) against Section 508 standards and WCAG 2.0. This is because Covered California’s contract with the vendor that developed and maintains CalHEERS required that the vendor provide a product that complied with accessibility standards issued by the W3C’s Web Accessibility Initiative. At the time Covered California entered into this agreement, the most up-to-date version of those standards was WCAG 2.0. Asked the consultant to review key pages in support of each web product, such as the accessibility page at all departments that featured such a page. At every department except Covered California, our consultant conducted its entire review on the live version of each department’s website that is available to the public. At Covered California, this was not possible because the live version of the application for health insurance includes a step that uses personal information to verify the applicant’s identity. As a result, the consultant conducted the review primarily in a test version of the application that, according to Covered California, contains the most up-to-date version of the department’s website. Our consultant conducted its review using the following web browsers and assistive technologies: the most recent versions of the Mozilla Firefox web browser and the NonVisual Desktop Access screen reader available at the time of testing, the consultant’s proprietary testing software, and supplemental testing with the Job Access With Speech screen reader versions 15 and 16 and the VoiceOver feature of the Apple Safari web browser. Interviewed staff at each department to obtain their perspective on the results of our consultant’s review.
b. The product’s accessibility options are clearly identified and readily available for persons with disabilities.	Reviewed the accessibility options discussed on each department’s website. Interviewed staff at departments that did not list accessibility options to determine why there were no options listed on those websites. Conducted research to determine which web browsers are the most popular to determine which web browser–specific information state departments should refer users who are seeking additional information about accessibility options.
c. The administering entity complied with appropriate procurement laws and processes applicable to Section 508 standards if it contracted with a vendor to develop or procure the EIT product. In addition, determine whether the contracts incorporate specific Section 508 requirements.	Interviewed department staff and reviewed relevant documentation to determine whether the web products we selected for review were procured. Determined that only Covered California and Community Colleges procured their web products. Reviewed solicitation and contract documents for procured web products to determine whether the departments complied with procurement laws and best practices applicable to Section 508 standards, including whether the documents incorporated specific Section 508 requirements for the web product. Determined that neither Covered California nor Community Colleges included specific Section 508 requirements in their final contracts. Further, neither department conducted market research related to accessibility. However, we concluded that the effect of these omissions is likely minimal because each department included language in its final contract that requires the vendor to provide a product that complies with Section 508 standards.
d. The administering entity appropriately tested the EIT product while the product was being developed, as it was implemented, and throughout the life of the product to ensure compliance with Section 508 standards and requirements.	Interviewed staff at each department and reviewed documentation to determine how frequently the departments tested the products selected under Objective 5 for Section 508 compliance during the products’ development and maintenance. Also determined what methods each department used to test its product. Compared each department’s testing approach to best-practice guidance for testing web products’ compliance with web accessibility standards and requirements.
e. The administering entity properly oversees and/or enforces compliance with the accessibility requirements and ensures corrective action is taken quickly when noncompliance is noted.	At Community Colleges, Covered California, and Franchise Tax Board, we used data from the databases described in the “Methods Used to Assess Data Reliability” table to select accessibility-related defects for review. We selected those items for review using keywords related to web accessibility because none of these departments tracked accessibility-related defects separately from the other defects in their databases. We reviewed between 20 and 30 issues at each of these departments, depending upon the results of our keyword search. Determined when the department discovered the defect, whether the defect had been fixed, and if so, how long the department took to fix those defects. Interviewed staff at these three departments about defects that were not fixed and those that were not fixed before key releases of their web product. At CalHR, the quality of record keeping regarding the department’s testing efforts was not complete enough to allow us to reliably review the results of the department’s testing and remediation of issues found during testing. As a result, we did not review a selection of defects found during testing at this department.
f. The administering entities of the EIT products selected have a process in place to receive and investigate complaints of alleged barriers to access, and evaluate the timeliness in investigating such complaints and the effectiveness of the process.	Interviewed department staff and reviewed any written policies to determine the department’s processes for collecting accessibility complaints from users of its website and web product, investigating, and addressing those complaints. At Community Colleges and Covered California, we used data from the databases described in the “Methods Used to Assess Data Reliability” table to search complaint records at each department using keywords related to web accessibility. We searched email records available at Franchise Tax Board using the same keywords. We identified complaints, if any, related to web accessibility. At CalHR, there was no reliable record of complaints received by the department because the department did not consistently track these complaints. At Covered California and Franchise Tax Board, our search of complaint records yielded no results for web accessibility complaints. At CalHR we identified isolated complaints in the web manager’s email records. At Community Colleges, we reviewed the complaints we identified and determined whether the department addressed the complaints and how quickly it did so.
6. Review and assess any other issues that are significant to the audit.	Reviewed the September 2012 report issued by the U.S. Department of Justice regarding Section 508 compliance to learn about best practices for Section 508 compliance.

Sources: California State Auditor’s analysis of Joint Legislative Audit Committee audit request 2014-131, and information and documentation identified in the table column titled Method.

Assessment of Data Reliability

In performing this audit, we relied on various electronic data files that we obtained from the entities listed in Table 4. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of computer-processed information that we use to support our findings, conclusions, or recommendations. Table 4 describes the analyses we conducted using data from these systems, our methodology for testing them, and the limitations we identified in the data. Although we recognize that these limitations may impact the precision of the numbers we present, in total there is sufficient evidence to support our audit findings, conclusions, and recommendations.

Table 4
Methods Used to Assess Data Reliability
Information System	Purpose	Method and Result	Conclusion
Covered California		We did not perform accuracy and completeness testing of these data because the systems are paperless systems and hard-copy source documentation was not available for review. Alternatively, following U.S. Government Accountability Office guidelines, we could have reviewed the adequacy of selected system controls that include general and application controls. However, we did not conduct these reviews because this audit involved five such paperless systems across three departments and to do so for each would have been cost-prohibitive. At the time of our review, none of these systems specifically tracked accessibility defects or complaints. We therefore searched for keywords associated with accessibility to identify items for our analysis. As a result, it is possible that defects or complaints exist in those systems that we did not identify.	Undetermined reliability for the purposes of this audit. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations.
Application Lifecycle Management system Data related to defects in the California Healthcare Eligibility, Enrollment, and Retention System (CalHEERS) and the remediation of those defects	To identify a selection of web accessibility defects Covered California identified while testing CalHEERS during the development and maintenance of the product through December 16, 2014. To determine whether Covered California resolved those defects and, if so, the average number of days it took to do so. To identify a selection of accessibility complaints made by CalHEERS users from June 2013 through December 15, 2014.
Remedy IT Service Management Suite system Data related to tracking and resolving reported incidents, problems, and change requests for CalHEERS	To identify a selection of accessibility complaints made by CalHEERS users from June 2013 through December 15, 2014.
California Community Colleges (Community Colleges)
JIRA system Data related to tracking the development process for the OpenCCC applications	To identify a selection of web accessibility defects Community Colleges identified while testing its online application service during development and maintenance of this service up to March 6, 2015. To determine whether Community Colleges resolved those defects and, if so, the average number of days it took to do so.
ZenDesk system Data related to user requests for assistance in using the OpenCCC applications	To identify a selection of accessibility complaints made by online application users since the release of the OpenCCCApply website in November 2012 through February 5, 2015. To determine whether Community Colleges addressed those complaints.
State of California Franchise Tax Board (Franchise Tax Board)
ClearQuest system Data related to enhancing, changing, and maintaining the CalFile application	To identify a selection of web accessibility defects Franchise Tax Board identified from the time it began testing the CalFile application for accessibility in 2012 through December 2013. To determine whether Franchise Tax Board resolved those defects and, if so, the average number of days it took to do so.

Source: California State Auditor’s analysis of various documents, interviews, and data from the entities listed above.

Note: For each data system, the time frame of the data we analyzed was determined by the availability of data in that system related to accessibility, up to the date we completed our review.

Footnotes

³ A web portal is a website that is or proposes to be a major starting site for users. In this case, the web portal being developed was the website located at www.ca.gov. Go back to text

⁴ At the time the policy letter was written, the department was known as the Office of the Chief Information Officer. Later it was reestablished as the California Technology Agency and in 2013 it became CalTech. Go back to text

⁵ Most of the content of this paragraph was derived from a copyrighted draft report issued by the W3C’s Web Accessibility Initiative in 2012, found at www.w3.org/WAI/intro/people-use-web/Overview.html. Go back to text

⁶ Most of the content of this paragraph was derived from a copyrighted document issued by the W3C’s Web Accessibility Initiative in 2005, found at www.w3.org/WAI/intro/components.php. Go back to text