Report 99131 Summary - April 2000

STAR Program: Ongoing Conflicts Between the State Board of Education and the Superintendent of Public Instruction as Well as Continued Errors Impede the Program's Success

RESULTS IN BRIEF

The implementation of the Standardized Testing and Reporting (STAR) program in California's public schools has been problematic due to open conflict between the State Board of Education (board), which is appointed by the governor, and the superintendent of public instruction (superintendent), an elected official. The decades-old conflict over each party's respective roles and responsibilities for administering the California Department of Education (department), as well as continued errors on the part of the school districts and the test publisher has negatively affected the development, administration, and implementation of the STAR program.

Implementing the program has been especially difficult because, due to this conflict, the superintendent has not developed an annual implementation plan. State law requires the superintendent to submit to the Legislature a plan for producing valid, reliable, and comparable individual student scores. Ideally, this plan would clearly outline the board's and department's responsibilities for implementing the program, establish criteria for measuring the reliability of test results, and provide an estimate of program costs.

One example of disputes between these parties is a breakdown in communication that caused the test publisher to temporarily halt its production of testing materials and incur significant expense to resume printing the materials. Specifically, the test publisher was to field-test 16 writing questions for grades four and seven. Although the board's executive director approved all the questions, the department approved only 14. The test publisher began production of the field-testing materials with 14 questions for each grade until the board pointed out that the contract required 16 questions for the field test. To resolve this conflict, the publisher halted production and then submitted additional questions to the department. The department approved two additional questions for each grade, and the publisher resumed production in late January 2000, at significant added expense.

The Legislature's wish to adopt a single statewide achievement test as quickly as possible has created additional problems for the STAR program. To speed the process for selecting a standardized achievement test, the Legislature imposed time pressures on both the superintendent and the board to evaluate potential test publishers and select a final test. State law enacted in October 1997 gave the superintendent three weeks to evaluate candidates and send forward a recommendation to the board. The same state law gave the board a little over a month to have its independent experts evaluate candidates and designate an achievement test. This unrealistic timeline made it difficult for the board to thoroughly evaluate the quality and capabilities of the publishers under consideration; it also did not allow the board to thoroughly evaluate the final candidate's procedures for monitoring its subcontractors, contingency plans, and computer capacity.

The compressed deadlines for selection of the test and preparation of the STAR program have had a cost: The test publisher's performance during the first two test cycles of the program has been deficient in certain areas. The department did not closely monitor the publisher's performance during these years. Missed deadlines, unreliable data, and inaccurate reporting of achievement test results plagued the program. The department contends that it did not have the legal authority or resources to monitor the publisher's performance because, as required by law, the test publisher had contracted directly with the school districts, not with the department. However, state law specifically requires the superintendent to establish a method of working with the test publisher, so the superintendent did have the authority to monitor the publisher's performance. Nevertheless, the superintendent did not develop a clear description of the work to be done by the publisher, establish a schedule of completion for major activities and milestones, create a process for monitoring the publisher's performance, or define roles and responsibilities for the department, board, and test publisher.

Some action has been taken by the Legislature, board, department, and test publisher to improve the management of the STAR program since the first two test cycles. To improve the department's oversight of the publisher, the department now contracts directly with the publisher. The test publisher has also sought guidance from its consultant on improving its operations. But further improvements must be made to ensure the success of the Public Schools Accountability Act of 1999, which creates a greater responsibility for the department, school districts, and test publisher to ensure the accuracy, completeness, and timeliness of the achievement test results. This act requires the superintendent to include the test results in the Academic Performance Index (API), a mechanism that will determine how the State will distribute about $150 million to schools and teachers. The API will be implemented for the first time in fall 2000.

Currently, the achievement test results are the sole component of the API and, therefore, the only determinant of these monetary awards. To ensure the integrity of the testing process and maximum fairness for the students assessed, the school districts, the publisher, and the department must push for better test security. One way to do this is to require all school districts and testing personnel to attend the publisher's training sessions to be sure that they properly follow the step-by-step procedures for administering the achievement test and securing test materials.

RECOMMENDATIONS

To facilitate communication between the board, superintendent, and the department and create a more productive environment for the STAR program, the Legislature should establish a mechanism for appointing a mediator to resolve disputes that will most certainly continue concerning these entities' respective roles and responsibilities.

To ensure that the STAR program is successful, the board, superintendent, and department must fulfill their designated responsibilities and improve their policies and procedures in the following ways:

AGENCY COMMENTS

While the board agrees with our findings, it generally does not agree with our recommendations. Specifically, the board believes that the governance issue between it and the department is much broader than the STAR program and to the extent that a solution is found, it should be established in this broader domain. Furthermore, while the board agrees that it needs to ensure that the publisher implements those recommendations that will improve the STAR program, it does not necessarily agree that the contract must be amended to accomplish this result. However, the board does agree that a formal meeting schedule to facilitate communication between it and the department is necessary.

The department agrees with our recommendations. In addition, the department provides information concerning its view of the STAR program's implementation.