Report 2016-129 All Recommendation Responses

Report 2016-129: K-12 High Speed Network: Improved Budgeting, Greater Transparency, and Increased Oversight Are Needed to Ensure That the Network Is Providing Reliable Services at the Lowest Cost to the State (Release Date: May 2017)

Recommendation for Legislative Action

To help ensure continuous network operations while preserving state resources, the Legislature should appropriate to the K12HSN program an amount that does not exceed $10.4 million for fiscal year 2017-18. If the Legislature wishes to appropriate a lower amount for the program, it should direct ICOE to modify one or more of the planned network upgrades we highlight in our report, either by delaying the upgrade to a subsequent fiscal year or by pursuing a less expensive option.

Description of Legislative Action

AB 99 (Chapter 15, Statutes of 2017), the omnibus education budget trailer bill, in part, requires the lead education agency responsible for the administration of the K12HSN to develop an annual budget request for K12HSN to be considered for the annual Budget Act. The lead education agency is required, before expending any funds for planned network upgrade projects that exceed $25,000 in cost, to develop a methodology to determine and prioritize planned network upgrade projects, and, commencing with the 2017-18 fiscal year and in each fiscal year thereafter, to utilize that methodology for planned network upgrade projects that exceed $25,000 in cost. Authorizes the Superintendent of Public Education to request data and other programmatic information from the lead education agency as needed to oversee the program.

California State Auditor's Assessment of 6-Month Status: Legislation Enacted

Recommendation #2 To: Imperial County Office of Education

To better inform decision makers at the state level about the amount of funding necessary to operate and maintain the network, ICOE should formally amend its annual budget documents by November 2017 to specify multiple potential levels of network expenditures for the coming year, and it should detail the specific network upgrades and project costs included in each scenario. As part of this process, ICOE should also provide information about how these upgrades will affect the network's functionality.

6-Month Agency Response

ICOE presented its first draft of 2018-19 proposed budget to Education on Thursday, November 9, 2017. The documentation included a list of proposed upgrades as well as options for increased capacity to sites where multiple choices were considered. A revised budget document was presented to Education on November 22, 2018, based on feedback received on November 9th.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

We reviewed the budget submissions that ICOE references in its response and confirmed that the current format offers the State two different expenditure scenarios for network circuit and equipment costs that correspond to an attached list of capacity upgrade options for the circuits in question. These scenarios vary from one another by approximately $530,000. This information, combined with the information the Education Code now requires ICOE to provide for all network upgrades in excess of $25,000, will enable state decision makers to reach more informed conclusions about which network upgrades to fund and, as a result, how much state funding to appropriate to the K12HSN program each year.

60-Day Agency Response

Steps are underway to ensure that budget documents are developed from an earlier Request for Proposal process. The process moved forward on the calendar from approximately October or November to July. The program budget documents will be developed for the 2018-2019 fiscal year by November, 2017. These documents will provide multiple potential levels of network expenditures. The documents will include specifics on discreet projects and rationale for each proposed solution. The process of determining which node sites require new bids due to expiring contracts, and which node sites require new bids to evaluate increased levels of utilization began in May. Then advanced to joint conversations with CENIC in early June and resulted in the posting of the RFP on July 14, 2017. The process of securing proposals will close on September 8, 2017. Processing of submitted proposals will occur between September 8th and mid-October, with circuit award recommendations being finalized by October 23, 2017. As part of the discussions between ICOE and state stakeholders, ICOE will provide information to stakeholders on multiple options for proposed capacity changes, and on how proposed upgrades will affect the network's functionality.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #3 To: Imperial County Office of Education

To ensure that its projected program costs are as accurate as possible, ICOE should institute by November 2017 a formal practice for reviewing its budget planning document against its current network design plans and correct any inaccuracies before finalizing and submitting its budget.

Annual Follow-Up Agency Response From October 2018

ICOE began implementing new practices with regard to budget review, review of network design plans and budget submission in October 2017. These practices continued through 2018 with predecessor steps rolling forward to earlier dates on the calendar for activities such as data collection (Jan-May), assessment of node site needs (May-June), developing a list of node sites under consideration for upgrade (June), and posting of E-rate required documents for bidding purposes (August). Memorializing new protocols as a written policy has now also occurred with said policy included on the K12HSN website for public access at www.k12hsn.org. The policy is being provided via email to Mark Reinardy per instructions. A PDF file depicting the project management dashboard related to these steps is also being provided.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

As ICOE indicates in its response, it asserted during earlier status reports that it had implemented this recommendation. However, the documentation ICOE submitted to support its claim was limited to the calendar schedule by which it determines its program, and thus budgetary needs. The audit findings that drove this recommendation were not the consequence of shortcomings in that process (which we addressed through recommendation #6), but rather insufficient processes for ensuring that the amounts in its budget submission to the State accurately reflected expected costs.

Along with its current response, ICOE provided documentation outlining not only the timeline for developing the annual K12HSN budget, but also the roles and responsibilities of specific staff in performing multiple levels of review of budget-related documentation. Diligently adhering to these protocols will assist ICOE in its efforts to increase the accuracy of its budget proposal for the K12HSN program.

1-Year Agency Response

ICOE has developed and posted the K12 Node Site Upgrade Decision process that aligns with required budget documents for presentation to Education. Inputs to the development of the process include the Network Implementation Committee, CENIC, the California Department of Education and internal processing. This practice is currently in place and will be revisited annually during the October/November time frame in alignment with recommendation #2.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

ICOE's response speaks to network upgrade decision-making issues we address in Recommendation 6, as well as the ultimate format of its budget presentations to the State, which we address in Recommendation 2. However, the response does not include information about how ICOE ensures that the cost information derived from its network planning decisions is accurately reflected in its proposed budget. Our audit found significant inconsistencies between ICOE's planned activities with the network and the costs included in its budget documents.

When we followed up with ICOE, it provided information about an electronic application it is using to memorialize the review of relevant documents. However, it did not provide any documentation about how that review is to take place or who is responsible for it--key elements of our recommendation that we emphasized in our previous evaluation of ICOE's response to this recommendation.

6-Month Agency Response

ICOE has aligned planning documents to match the proposed budget documents presented to Education. This practice is currently in place and will be revisited annually during the October/November time frame in alignment with recommendation #2.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

ICOE provided up-to-date versions of the planning documents its used to create its anticipated budget request for the 2018-19 fiscal year and explained that, subsequent to our review, K12HSN's fiscal manager is now assisting network staff with the creation of those documents and is reviewing them to ensure the documents accurately reflect anticipated network costs. However, ICOE has yet to memorialize this process either as a formal program protocol or in the duty statements of the staff positions involved. We recommended a formal implementation of this review process in order to help ensure that it occurs consistently and to better establish accountability for the accuracy of ICOE's budget submissions.

60-Day Agency Response

A new process will be utilized when the request for proposals closes. The process will involve a comparison of the pricing sheets submitted by proposers against network design layouts. The process will permit K12HSN staff to confirm costs proposed by vendors, ring designs, and wherever possible, achieving carrier and hub site diversity. The process will occur in October of every year and involve comparing various scenarios against budgeting documents.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: Imperial County Office of Education

To help ensure that the K12HSN program maintains the necessary amount of state funds in reserve, ICOE should prepare a formal methodology for a proposed equipment reserve that is based on the actual likelihood of equipment failure and the costs associated with replacing that equipment.

6-Month Agency Response

ICOE convened the Network Implementation Committee (NIC) in September to evaluate an appropriate level of reserves based on current warranty coverage, expected use of life, and equipment cost trends. The NIC recommended that K12HSN designate an amount in the unused funds to cover at minimum the most expensive piece of hardware running on the network and add some buffers to cover other circuit related expenses. These designated funds will be used to protect against catastrophic events, where an entire node might fail and the equipment is not covered by warranties or regular support agreements. Based on input from the NIC, the 2018-19 budget documents designate a reserve of $500,000 for equipment replacement, which will be evaluated on an annual basis ongoing during budget preparation.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

ICOE confirmed for us that, after considering the equipment reserve issue with network stakeholders as referenced in its response, the K12HSN program established a methodology wherein it set the dollar amount of the reserve equal to 125 percent of the piece of network equipment that would be most expensive to replace. As such, the network would be covered in the unlikely event that such a piece of equipment that is not covered by warranty fails unexpectedly. ICOE intends to apply this methodology to each year's budget, meaning that the size of the reserve would fluctuate some from year to year based on the market price of the equipment in question.

60-Day Agency Response

The Network Implementation Committee (NIC), which was formed in 2005 by soliciting participation from technology directors or chief technology officers from each of the 11 regions of the state, is being called upon to re-engage in the K12HSN work in a more active way. While the group has met quarterly by videoconference since about 2008, the K12HSN staff will be convening the group, including new members, in face-to-face meetings to assist in these efforts. The first session has been scheduled for September 8, 2017 in San Diego, California and meeting invitations are being sent to the updated list of NIC members. A draft agenda for that meeting is attached.

California State Auditor's Assessment of 60-Day Status: Pending

ICOE provided an agenda for the September 2017 meeting that includes a discussion item addressing strategic planning for an appropriate equipment reserve. We will review the results of this effort as part of our evaluation of the recommendation's implementation status in November 2017.

Recommendation #5 To: Imperial County Office of Education

To ensure that it is able to continue critical network services with reduced operating reserves, ICOE should establish procedures to routinely monitor the K12HSN program's cash balance and to evaluate upcoming costs. If at any point ICOE determines that it will be unable to fund its costs due to delayed subsidy payments, it should notify Education regarding the size and timing of the anticipated shortfall and postpone significant discretionary expenditures, such as upgrading network site equipment, until ICOE collects the subsidies it is owed.

60-Day Agency Response

ICOE follows stringent budgeting practices and schedules for reviewing actual expenditures against budgeted expenditures and available cash to meet expenditures. A report reflecting the cash position of K12HSN is monitored weekly. Monthly reports are prepared within one week after the last day of the month and provided to the Imperial County School Board (ICSB) at their monthly meetings held on the 2nd Monday of the month.

The circuit orders, construction projects, and equipment procurement undertaken by the K-12 High Speed Network program through its network operator, CENIC have an unavoidable fluidity to them. A project may be planned in October of one year, come to fruition in August of the following year, and see invoices processed -particularly when it comes to the equipment purchasing -the following January. The challenge of aligning cash in treasury with timing of payments is complicated by both unpredictability in timing of the receipt of subsidy reimbursements, and by the unpredictability of service delivery and invoicing. Compound the unknowns of one project with the multiple projects always underway and it becomes clear that the exercises required are a delicate balancing act. Nevertheless, ICOE agrees to maintain open communications with Education and provide early notice of any potential shortfalls in available cash. Monthly phone meetings occur between K12HSN and Education. Updates for Education related to budget needs and project expenditures will occur then. Should there be an unanticipated and concerning scenario regarding cash balances and projects, K12HSN will contact Education immediately with all relevant information.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

ICOE provided documentation of its monitoring efforts in the form of monthly reports to the Imperial County School Board and a calendar denoting ICOE's scheduled phone calls with Education. Although ICOE indicated that the implementation date of this recommendation is ongoing, we consider the recommendation to be fully implemented because ICOE has demonstrated that it intends to continue monitoring its cash flow indefinitely, and the steps it has taken will allow it to identify and respond to a cash flow issue should one emerge.

Recommendation #6 To: Imperial County Office of Education

To better guarantee that network upgrades are necessary and are achieved at the lowest possible cost to the State, ICOE should develop a formal methodology for reviewing circuit capacity needs. This methodology should include consideration of multiyear trends in network traffic and the implications prospective upgrades may have for other parts of the network. Doing so would not only assist ICOE when determining the magnitude of circuit upgrades relative to the cost involved, but would also help ICOE determine whether it can delay upgrades until it can establish those upgrades' eligibility for E-Rate subsidies.

Annual Follow-Up Agency Response From October 2018

Over the course of the past year the program has implemented a formal methodology which includes extending data retention on utilization for a minimum of five years, soliciting additional inputs from node sites and district technologists and preparing trending information and over-subscription rates on all node sites under consideration for upgrade. During the bidding process, bids are solicited for multiple bandwidths to ensure as many options as possible are considered in selection of appropriate bandwidths for each node site. The upgrade decision process has been memorialized and is available for review on the K12HSN website at www.k12hsn.org under "Policies" and is being provided via email to Mark Reinardy.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Based on the range of factors ICOE must consider when making upgrade decisions, we stated in our assessment of its one-year response that we would be looking for evidence that ICOE has consistently and systematically considered all relevant measures when evaluating implementation of this recommendation. In addition to the written process for considering upgrade decisions that ICOE describes in its current response, ICOE provided evidence that it collected and considered factors such as cost, multi-year network usage trends, and "downstream" capacity and usage by offices of education and school districts. Using this information, ICOE made preliminary upgrade determinations that it provided to the Department of Education and Department of Finance in October 2018 for their review, along with its proposed fiscal year 2019-20 K12HSN budget. Based on the information ICOE has collected and the evidence it presented to us, which indicates it has performed equivalent analyses for a range of network upgrades, we conclude that ICOE has fully implemented our recommendation. However, we intend to follow up with ICOE after the fiscal year 2019-20 upgrade decisions have been finalized in order to thoroughly validate ICOE's application of its methodology and ensure its analyses are based on complete and accurate information.

1-Year Agency Response

ICOE has developed a new template to document factors considered for the upgrade recommendations to Education. This template was created in accordance with Ed. Code section 11800 parameters and aims to document and memorialize all factors, qualitative and quantitative, considered for upgrade decisions. These factors will continue to drive the decision-making process for K12HSN administration, in addition to ever-evolving market conditions, overall network operation and long-term strategic planning. The methodology includes:

New timeline to align with state budget planning cycles as presented in Recommendation #3

Oversubscription rates downstream (DataLINK) for each node and a comparison to statewide averages

Other downstream inputs from DataLINK including district-level information on number of devices supported, cloud-based activities and other triggers for utilization growth

Trending data from previous years (Data collection on node site traffic usage in the months of May and October which represent peak school year traffic).

Network reliability data based on the CENIC Network Operation Center announcements

Multiple bandwidth increase options and financial impact analysis based on RFP responses received

Review and receive input from the Network Implementation Committee, Regional leads, CENIC Engineers and K12HSN staff

Based on 2018-19 decisions being due in March 2018 to meet E-rate deadlines, the full implementation of the new processes will begin in the 2018 calendar year in preparation for the 2019-2020 budget cycle.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Based on its 1-year response and the supporting documentation ICOE provided with respect to this recommendation's status, we determined that the recommendation is partially implemented insofar as ICOE has created and provided general written processes and timelines by which it will utilize the recommended methodology. In addition to these documents, ICOE provided an evaluation template that reflects some, but not all, of the measures we identified during our audit as important considerations for upgrade decisions.

When evaluating ICOE's implementation of this methodology as part of the 2019-2020 budget cycle referenced in its response, we will be looking for evidence that ICOE has consistently and systematically considered all relevant measures as part of selecting network upgrades.

6-Month Agency Response

ICOE convened the Network Implementation Committee (NIC) on September 7, 2017 to discuss the proposed new process for network upgrades in line with other recommendations of the audit. The methodology includes:

New timeline to align with state budget planning cycles

Improved tool for collecting downstream capacity information from school districts (DataLINK).

Develop preliminary list of node upgrades based on:

Oversubscription rates downstream (DataLINK)

Trending data (Data collection on node site traffic usage in the months of May and October (peak school year traffic).

Bandwidth usage data from Orion and Cricket monitoring tools

Technical review process by CENIC engineers

Review process by the NIC on proposed upgrades

The new process is scheduled to be implemented starting the 2018 calendar year in preparation for the 2019-2020 budget cycle.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

As indicated by its response, ICOE has identified a range of factors it plans to consider as components of its methodology for identifying the need and magnitude of network capacity upgrades. However, ICOE has not yet formally applied those components in the context of the network planning process in order to reach upgrade decisions. ICOE intends to do so as part of its efforts leading up to the submission of its fiscal year 2019-2020 K12HSN budget. Our expectation regarding that process, which aligns with changes made to the state Education Code regarding the administration of K12HSN, is that through the process, ICOE will produce a record of which factors ultimately drove ICOE's determinations for the specific network upgrades in question. Using information from that process, we will be able to evaluate the extent to which ICOE has implemented its methodology.

60-Day Agency Response

ICOE adjusted the settings on the Orion/Solarwinds monitoring system by June 15, 2017 so that average utilization per day at each node site is being stored for five years (1,825 days). The multi-year data reflecting network traffic will be used in processing decisions to maintain current bandwidth or increase bandwidth to meet demonstrated need. This process will be informed by a re-energized effort to collect full information from node site liaisons on the anticipated changes in downstream demand, school district initiatives and all other relevant factors. The implications that each prospective upgrade will have on other parts of the network will also be considered.

As mentioned before the Network Implementation Committee (NIC) will be called upon as an advisory panel to guide the effort to formally collect information on current utilization and anticipated growth over the next 6, 12 and 18 months. The support anticipated from node site liaisons will be communicated via the NIC, the annual node site meeting staged during an annual conference for technologists, and via email announcements to the technical and administrative contacts lists maintained by ICOE. The K12HSN will work with node site liaisons to conduct data collection reflecting the plans of the districts in their service areas and, where available, will collect links to any URLs available for monitoring usage. The whole effort will be documented into a written methodology that will be publicly available.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #7 To: Imperial County Office of Education

To provide as many options for network upgrades as possible and to help ICOE provide the most cost-effective upgrade options without risking its eligibility for subsidies, ICOE should adopt the practice of requesting bids at all feasible levels of capacity upgrades as opposed to only those levels that represent a tenfold increase in circuit capacity.

60-Day Agency Response

As evidenced by the attached Request for Proposals, Exhibit D, for circuits being considered for potential installation, bids are being requested at all feasible levels. This is true both for circuits which are approaching their expiration date and are therefore in need of replacement, as well as for circuits experiencing congestion or nearing saturation for which increased capacity is being evaluated.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

ICOE provided us with its current K12HSN bidding documents that include requests for pricing information from service providers at all levels of network capacity.

Recommendation #8 To: Imperial County Office of Education

To help facilitate the review of circuit capacity needs, ICOE should maintain historical data for network traffic as long as technically feasible. It should also ensure that its monitoring software includes all network sites.

1-Year Agency Response

In the summer of 2017 ICOE changed settings on its Orion monitoring system to retain a minimum of five years' data. ICOE agrees that all circuits procured from vendors by CENIC on behalf of K12HSN and the state as well as the dark fiber connections serving node sites should be monitored. ICOE has worked with CENIC to secure access to the dark fiber connections which had previously been shielded from K12HSN monitoring. Currently all of the "lit" circuits and all of the dark fiber connections are included in the monitoring system in use by ICOE. The CENIC network operations center (NOC) notifies K12HSN staff via e-mail distribution lists when new circuits are put into service or decommissioned to ensure that K12HSN adds them to or removes them from the Orion monitoring system.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

As noted in our evaluation of ICOE's 60-day and 6-month responses to this recommendation, ICOE has already taken steps to implement the portion of the recommendation related to data retention. In its 1-year response, ICOE confirmed that it has now successfully worked with CENIC to ensure monitoring access for all active circuits. ICOE also provided documentation of notifications CENIC provides to ICOE staff when a new circuit is activated or an existing circuit is decommissioned. In the former scenario, the notification also informs ICOE staff how to begin monitoring the newly active circuit.

6-Month Agency Response

ICOE agrees that all circuits procured from vendors by CENIC on behalf of K12HSN and the state should be monitored, there are technical barriers to gaining access to the connections over dark fiber. ICOE is working with CENIC to ensure that not just all of the "lit" circuits but also all dark fiber connections are included in the monitoring system in use by ICOE.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

As noted in our evaluation of ICOE's 60-day response to this recommendation, ICOE has initiated a process to automatically retain network usage data for five years with its Orion monitoring software and will be archiving the data from the system in order to retain it past five years. We will evaluate the outcome of ICOE's work with CENIC that is reported in ICOE's one-year response to determine whether that work results in processes to ensure network and outage monitoring for all active K12HSN circuits.

60-Day Agency Response

ICOE changed the settings in its monitoring system by June 15, 2017, so that Orion/Solarwinds is now retaining daily average utilization for five years (1,825 days). ICOE may, in the future, substitute a different monitoring system for its existing system, nevertheless, steps will be taken to retain historical data for purposes of future use in the activity of evaluating growth trends. Further, the requirements created in recent K12HSN-related budget bill language will be followed.

ICOE has implemented new practices and notice requests from CENIC in order to ensure that all K-12 aggregation node sites are subject to monitoring in the K12HSN system. In addition, for any non-standard connections such as those resulting from a "local build" instead of a circuit purchased from a telecommunications provider, ICOE will work cooperatively with CENIC and the node site staff members to ensure that all program-funded connections are monitored.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

ICOE has initiated a process to automatically retain network usage data for five years with its Orion monitoring software and will be archiving the data from the system in order to retain it past five years. ICOE has taken steps to help ensure CENIC notifies ICOE about newly active circuits so that those circuits can be included in Orion, but ICOE still needs to take further steps to ensure that all currently active circuits are monitored.

Recommendation #9 To: Imperial County Office of Education

To reduce the risk of having to react to large increases in network traffic, ICOE should formalize a process to include input from network site administrators during network upgrade planning.

1-Year Agency Response

Consistent with the new process to meet with K12 Node site leads during the first half of the year, in 2018 K12HSN staff met with node site staff members in April and May to ensure that input from the field was collected for consideration. These meetings permitted K12HSN staff to seek support and prioritization on their part for the new data inputs into DataLINK, which will inform the Circuit Evaluation template used in recommendation #6. New data components have been included in DataLINK seeking district level input from all California school districts including county offices of education. New data fields include peak utilization, anticipated growth in utilization in the coming year, device counts, and information on which cloud-based services are utilized by the districts. These additional fields will provide important new information on the rate of growth that can be expected at districts and at the node sites that serve them. Additionally, over-subscription rates are being reviewed and compared to historic information to inform the upgrade planning. The new review process by the Network Implementation Committee (NIC) of potential upgrades has been developed so they can weigh-in on the proposed list of upgrades.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

In addition to the expanded data collection tool referenced in our assessment of ICOE's six-month response to this recommendation, ICOE has now provided evidence of incorporating the collection of information from site administrators as part of its K12HSN regular node site meetings. ICOE also provided evidence of attempts to solicit equivalent information from the school district level, as well as initial data regarding the prevalence of changes in individual sites' bandwidth capacity. We expect to see ICOE use the information gleaned from these efforts as part of implementing the formal network upgrade methodology we referenced in recommendation #6.

6-Month Agency Response

ICOE agrees that factors related to "downstream traffic" play a role in predicting growth rates and planning for capacity changes. In order to add context to the review of current use, historic use and growth trends, ICOE is in the process of increasing data fields collected from California county offices of education, schools and school districts. These additional fields will provide important new information on the rate of growth that can be expected at districts and at the node sites that serve them. ICOE will continue to meet with Node site administrators after the April data collection period concludes. The new review process by the NIC of potential upgrades has been developed so they can weigh-in on the proposed list of upgrades.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

ICOE has expanded the tool it uses to collect data from school districts to include questions about districts' planned network-related activities and other online initiatives for the coming year, which can be used to help ICOE project utilization of K12HSN network circuits. ICOE is still preparing for its first round of collecting this data, which it will conduct between January and April 2018 to conform to the broader network planning timeline. We will reassess the status of ICOE's response to the recommendation after ICOE distributes these requests for information and begins to collect the data.

60-Day Agency Response

At all times prior to 2016, the annual effort to collect updated information on district and school level data for its DataLINK database included the request that anticipated "downstream" upgrades be reflected in the newly installed data. The request for this data was commonly given less priority than other components of the requested information.

As mentioned before the Network Implementation Committee (NIC) is being called upon as an advisory panel to guide the effort to formally collect information on current utilization and anticipated growth over the next 6, 12 and 18 months. In addition to the support of the NIC, at the K12HSN annual node site meeting staged at the annual conference for technologists, ICOE will communicate to all node sites how and why the program will be expecting fuller and deeper reporting from them related to downstream traffic and planned initiatives in the districts they serve. Where available, ICOE will collect links to any URLs available for monitoring usage. The entire effort will be documented in a written methodology which will be publicly available.

In addition, in a future iteration of the DataLINK system, ICOE will be refining the data components to be solicited in order to support the ease and simplicity of this element of the data collection.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

ICOE demonstrated the use of its DataLINK database to collect information on network upgrades. Full implementation is pending ICOE's inclusion of the integration of this information into the formal network upgrade methodology outlined in Recommendation #6.

Recommendation #10 To: Education, Department of

To increase transparency in the K12HSN program and help ensure that the State has sufficient information to measure the program's effectiveness, Education should direct ICOE to report annually on specific performance measures. These performance measures should include the following metrics:
- Cost per unit of capacity used.
- Network bandwidth.
- Frequency, duration, cause, and location of network outages or interruptions.
- Latency and packet loss on network circuits.
Education should stipulate that the receipt of grant funds is conditional based on the recipient's agreement to provide these measures and other information deemed necessary by Education, either on request or at regular intervals determined by Education. If Education believes that it does not currently have legal authority to direct ICOE to report on this information, it should seek legislative change to obtain that authority.

1-Year Agency Response

Fully Implemented. In December 2017, K12HSN presented a plan, which Education approved, to define and collect the following: (1) cost per unit of capacity used;(2) network bandwidth utilization;(3) frequency, duration, cause, and location of network outages or interruptions; and (4) circuit performance relating to loss of packets and latency. In addition, K12HSN has already provided information in the 4 recommended areas to Education for the first 6 months of fiscal year 2017-18. On May 10, 2018, Education confirmed with K12HSN that the information will be included in their future annual reports.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

As we noted in our assessment of Education's six-month response to this recommendation in November 2017, implementation of the recommendation was pending a commitment from ICOE regarding the inclusion of these metrics in the K12HSN program's annual plan.

Along with its one-year response, Education provided documentation of this commitment from ICOE, as well as a plan outlining the specific information to be reported and six months' worth of preliminary monitoring results.

6-Month Agency Response

Education is continuing to work towards fully implementing this recommendation. On September 7, 2017, Education attended the Network Implementation Committee meeting and planning session at which the audit recommendations were discussed. In addition, CDE and K12HSN discussed metrics and reporting options during monthly operations calls. On October 18, 2017, Education provided K12HSN a formal written request to submit the plan for reporting metrics.

California State Auditor's Assessment of 6-Month Status: Pending

As we noted in our 60-day evaluation of Education's implementation of this recommendation, changes to state law and the fiscal year 2017-18 funding structure for the K12HSN program have precluded the need for Education to amend its grant agreement with ICOE. Instead, Education indicates that it is continuing to work with ICOE to develop an annual reporting plan containing the performance metrics in our recommendation.

The status of this recommendation is therefore pending our review of the specific reporting metrics included in the required annual plan once the plan's structure is ultimately adopted by Education and ICOE.

60-Day Agency Response

NOTE: THE IMPLEMENTATION DATE SHOULD BE JUNE 2018, BUT THE SYSTEM ONLY GOES TO 2017.

Education was prepared to incorporate language in the grant award notification for Fiscal Year (FY) 2017-18, requiring the K12HSN grant recipient to submit an annual report, which would have included the following: (1) cost per unit of capacity used; (2) network bandwidth utilization; (3) frequency, duration, cause, and location of network outages or interruptions; and (4) circuit performance relating to loss of packets and latency. However, the FY 2017-18 California State Budget did not include K12HSN grant funding; therefore, Education cannot issue any grant awards to them. The K12HSN will utilize existing reserves to fund the program for FY 2017-18.

On June 27, 2017, Education Code 11800 was updated and now authorizes the Superintendent to request data and other programmatic information as needed to oversee the K12HSN program. Therefore, Education will: (1) request a plan for reporting performance metrics from K12HSN by October 1, 2017; (2) review and approve the plan to report performance metrics by December 31, 2017; and (3) review and approve performance metrics for the FY 2017-18 school year by June 30, 2018.

California State Auditor's Assessment of 60-Day Status: Pending

Given the changes to state law and Education's planned approach, we will review the content of the performance reporting plan referenced in this response when Education approves the plan in order to ensure it contains the metrics we set forth in our recommendation.

Recommendation #11 To: Imperial County Office of Education

To better support future reporting efforts for the K12HSN program, ICOE should amend its contract with CENIC to require CENIC to report on specific network performance measures, including the frequency, cause, location, and duration of network outages or interruptions.

1-Year Agency Response

ICOE agrees with the recommendation that it monitor network performance. ICOE is translating existing outage emails from the CENIC Network Operations Center into an Incident Report in a manual process. The Incident Report details each circuit outage and whether the outage resulted in a "hard down" or not. In the vast majority of cases, the load balancing circuit is able to keep the site from being out of production. This is especially important when the circuit failure turns out to be hard to identify or difficult to repair. ICOE provides the Incident Report to California Department of Education monthly, typically by the 10th of the month following.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

As we discuss in our assessment of ICOE's six-month response to this recommendation, ICOE was at that time still determining whether it would provide the greatest benefit to the K12HSN program to add the recommended monitoring activities to its contract with CENIC, or to take primary responsibility itself for that monitoring. In addition to providing updated data regarding network interruptions, ICOE also confirmed in its one-year response that it has taken primary responsibility for monitoring connectivity using supporting data provided by CENIC.

6-Month Agency Response

ICOE agrees with the recommendation that it monitor outages on the network and track service credits when they are earned to ensure they are recovered from the provider. Currently ICOE is translating existing outage reports from the CENIC Network Operations Center into an Incident Report in a manual process. The Incident Report details each circuit outage and whether the outage resulted in a "hard down" or not. In the vast majority of cases, the load balancing circuit is able to keep the site from being out of production. This is especially important when the circuit failure turns out to be hard to identify or difficult to repair. CENIC and ICOE are working collaboratively to create an automated system that will translate the email notices into the database currently being maintained manually. Until such time as an (affordable) automated system is identified, ICOE will continue with the manual monthly reports.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

ICOE provided us with documentation regarding the monitoring efforts it is currently undertaking using data provided by CENIC. ICOE plans to continue these efforts until it reaches agreement with CENIC about the most efficient and effective means of automating the process on a permanent basis, including determining the most effective use of that information with respect to holding Internet service providers accountable. At this time, it is not clear whether this will be accomplished through an amendment to ICOE's contract with CENIC. We will evaluate the recommendation's status again when those decisions have been made.

60-Day Agency Response

On June 8, 2017 ICOE and CENIC met to discuss new processes that will assist in ICOE's ability to report on network performance. Draft language for a contract amendment with CENIC is being developed and will be shared with CENIC for discussion by July 27, 2017.

In addition, ICOE's independent monitoring system is being evaluated for an approach to automate delivery of information from the system into a report format that provides most of the information being requested from Education. The "reason for outage" is a data element that is only available from CENIC. Currently using manual steps, the location, duration and frequency are available to ICOE from its own system and the program is working to determine the best approach to extracting this data from the monitoring system into outage reports.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #12 To: Imperial County Office of Education

To ensure that the K12HSN program receives all of the service credits to which it is entitled, ICOE should amend its contract with CENIC to clarify CENIC's responsibilities in this area, including reporting to ICOE about network outages or interruptions and requests for credits to service providers, along with the outcomes of those requests.

1-Year Agency Response

ICOE and CENIC have negotiated a contract amendment under which CENIC agrees to track network incidents and outages and pursue service credits when they are earned. ICOE staff will compare CENIC-generated information related to outages and service credits to its own incident reports and ensure that all earned service credits are reflected in subsequent circuit cost summaries and provided by the respective telecommunications providers.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

ICOE provided a signed contract addendum in which CENIC agreed to track and report on an annual basis all network outages and the status of service provider credits to ICOE. ICOE has indicated it will compare this reporting to the data it is independently tracking (see recommendation #11). Through that process, ICOE will ensure that CENIC has taken appropriate action on all outages that may qualify for credits to the program.

6-Month Agency Response

ICOE has developed a tool to track network incidents and outages. ICOE staff is manually harvesting the data from the CENIC Operational Announcements listserv, since there is no easy way to pull this information in a database format. ICOE has started conversations with CENIC about the possibility of automating this process for K12. The data is currently being updated on a monthly basis. The tool will continue to be refined to provide more details related to each circuit on the network and determine whether the incidents are subject to service credits from the providers. Contract agreements are currently in draft and being reviewed by CENIC.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

As outlined in our response to Recommendation 11, ICOE is collecting the data needed to hold service providers accountable for network interruptions but has not yet implemented a process by which it--or CENIC--will do so. At this time, it is not clear whether this will be accomplished through an amendment to ICOE's contract with CENIC. We will evaluate the recommendation's status again when those decisions have been made.

60-Day Agency Response

On June 8, 2017 ICOE and CENIC met to discuss new processes that will assist in ICOE's ability to report on network performance. Draft language for a contract amendment with CENIC is being developed and will be shared with CENIC by July 27, 2017.

In addition, ICOE's independent monitoring system is being evaluated for an approach to automate delivery of information from the system into a report format that provides most of the information being requested from Education. The "reason for outage" is a data element that is only available from CENIC. Currently using manual steps, the location, duration and frequency are available to ICOE from its own system and we are working to determine the best approach to extracting this data from the monitoring system into outage reports.

Tracking of service credits is a component of the outage report format developed for the new reporting. CENIC understands ICOE's need to have information called out related to service credits, both when earned, and when received by ICOE.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #13 To: Imperial County Office of Education

To ensure efficient use of state funds, ICOE should conduct a cost and benefit analysis of its memorandum of understanding with the Butte County Office of Education to determine whether it represents the most cost-effective approach to providing the program activities it covers.

1-Year Agency Response

ICOE has taken steps to implement this recommendation. ICOE and Education have agreed to which roles ICOE will perform related to statewide level E-rate support. ICOE has developed an RFP to procure quotes from contractors for E-rate subject matter expertise and support. ICOE will not be contracting with Butte County Office of Education for E-rate support services beyond June, 2018. ICOE and CDE will jointly select a contractor to provide training, advisement, consulting and reporting functions during 2018-19. Decisions related to E-rate support services in future years will be informed by the lessons ICOE and CDE learn in 2018-19.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

As we note in our audit report, our primary concern with ICOE's contract with the Butte County Office of Education (Butte) for E-Rate support services was that the contract included costs for Butte's overhead and administration, despite the fact that the contracted employee did not appear to rely on Butte's office resources in performing the agreed upon work. Based on the actions ICOE has taken in ending its contractual relationship with Butte, we conclude ICOE concurs with our finding and has determined the contract did not represent the most efficient use of state funds. In pursuing its new agreement for services, however, ICOE should ensure that agreement does not include any similar unnecessary costs.

6-Month Agency Response

ICOE has taken steps to implement this recommendation. ICOE plans to develop an RFQ that will assist in assessing other means of accomplishing the work currently undertaken by Butte County Office of Education. The results will be utilized in ICOE's cost-benefit analysis to determine whether the current approach to providing E-rate support to school districts across the State is sound or whether there are other more effective means to accomplishing this support.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

ICOE has held conversations with Butte COE relative to the partnership and support role Butte COE has played, as well as the services provided to the K12HSN program by Butte COE. The services required have changed over time and a review of the current deliverables is appropriate. ICOE, in conjunction with Education, will determine the appropriate scope of work and explore alternative scenarios for providing the needed activities and services. Either the Network Implementation Committee, a third party consultant, or an internal team will conduct an analysis of the potential alternatives for meeting the program goals and the costs applicable to each option. The results of the cost-benefit analysis will be provided in a future response

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2016-129

Agency responses received are posted verbatim.