Skip Repetitive Navigation Links
California State Auditor Logo  

Report Number: 2016-129

Abbreviations

K-12 High Speed Network
Improved Budgeting, Greater Transparency, and Increased Oversight Are Needed to Ensure That the Network Is Providing Reliable Services at the Lowest Cost to the State

Use the links below to skip to the section you wish to view:



Excessive K12HSN Operating Reserves Have Recently Been Reduced, but Questions Remain About What a Prudent Reserve Should Be

Key Points:


Inadequate Budgeting and a Lack of Oversight at the State Level Contributed to ICOE’s Excessive Operating Reserves

The annual expenditures ICOE incurred for the K12HSN program have historically been considerably lower than the amount of the state appropriation for the program, which led to the program’s accumulating large operating reserves. According to ICOE’s fiscal year 2015–16 audited financial statements, the amount of its state appropriation that ICOE spent annually averaged about $7 million from fiscal years 2010–11 through 2014–15, while the state appropriation remained steady at about $8.3 million per fiscal year—the amount that ICOE continued to request despite accumulating a substantial reserve. As Figure 3 illustrates, the program’s operating reserve grew from $8.7 million at the end of fiscal year 2010–11 to $14.7 million by the end of fiscal year 2014–15. For fiscal year 2015–16, the State withheld the appropriation to the program because of this large reserve. As a result, ICOE used most of its reserve to operate the program. At the end of fiscal year 2015–16, ICOE’s operating reserve for the program totaled approximately $5.7 million.

Figure 3
K12HSN Year-End Operating Reserves
Fiscal Years 2010–11 Through 2015–16

Figure 3, a graphic showing the K12HSN year-end operating reserves for each fiscal year from 2010–11 through 2015–16.

Source: ICOE audited financial statements.

* Because K12HSN received no state funding in fiscal year 2015–16, ICOE used the K12HSN operating reserve to pay for the portion of program expenditures not covered by federal Internet subsidies or other funding sources, thereby reducing the reserve amount by approximately $9 million.


Inadequate budgeting and a lack of oversight at the state level contributed to ICOE’s excessive operating reserve balance. ICOE’s approach in submitting its budget during this time was to present information to the State at a summary level without any detail to support its projected costs. Specifically, the budget documents ICOE provided to the State included only a single line item combining all projected expenditures pertaining to CENIC, such as equipment, maintenance, network services, and management fees. In contrast, for its internal expenditures, ICOE identified amounts individually by type, even though the total amount of those expenditures was less than a quarter of the CENIC expenditure line item. In addition, ICOE continued to request the same appropriation amount of $8.3 million from the State each year over this period, even though K12HSN’s operating reserves were more than $8 million and continuing to increase.

The K12HSN program’s chief executive officer explained that it has been difficult historically for ICOE to anticipate network service costs and that during this period ICOE took a conservative approach by projecting higher cost estimates. The program’s chief operating officer indicated that ICOE did not intend to generate the large reserve. Nonetheless, we expected that ICOE would reduce its annual budget requests in response to its consistently spending less than the State provided. According to the chief operating officer, the uncertainty about the timing of E‑Rate and Teleconnect Fund payments made requesting a lower state appropriation risky. Nevertheless, the reserve had grown well beyond the amount needed to mitigate those concerns.

The consistent amount of the State’s appropriation leading up to fiscal year 2015–16 compared to the actual expenditures incurred leads us to conclude that there was an apparent lack of fiscal oversight of the program at the state level. The former director of the Educational Data Management Division at Education, who reviewed the budget requests for fiscal years 2013–14 and 2014–15, informed us that the division performed a detailed review of network sites that were lacking capacity and the cost for upgrading that capacity as part of ensuring the network could support statewide student testing. However, the formal budget documents ICOE provided to Education in those years were presented at a summary level and do not identify the costs of individual network site upgrades. Further, Education’s budgetary review did not result in adjustments to the state appropriation amount from previous fiscal years, even as the K12HSN program’s operating reserves grew. The division’s current director explained that Education does not play a direct role in decision making for individual K12HSN projects and that his unit is not currently staffed to conduct a more detailed review.

ICOE has made efforts in recent years to improve its budgeting practices. For example, ICOE has begun preparing detailed internal projections for specific costs contained in the CENIC portion of its budget. These projections consist of estimated Internet service and equipment costs for each of the network sites. According to the program’s chief executive officer, ICOE first began using this method during its fiscal year 2015–16 budgeting process. Additionally, starting with the fiscal year 2016–17 budget, ICOE modified the format of the budget proposals it provides to the State to include additional detail regarding the types of costs that make up the total projected CENIC expenditures, such as administrative services and total equipment costs.


Despite Its Recent Efforts, ICOE Needs to Improve Its Budgeting Process

Although ICOE has recently taken steps to include more information during the K12HSN budgeting process, we identified issues with the accuracy and detail of ICOE’s current budget documents. Since the beginning of our audit in November 2016, and partly because of questions we asked, ICOE has reduced projected K12HSN net expenditures for fiscal year 2017–18 by more than $3 million, from $20.3 million to $16.9 million. The reduced projection for program expenditures has resulted in ICOE’s requesting a lower state appropriation, which was initially established at $14 million and has subsequently been revised to $10.4 million. Table 2 summarizes the size, timing, and cause of the reductions that ICOE has made during the period of our audit to its fiscal year 2017–18 budgeted program expenditures.

Table 2
Recap of K12HSN Budget Projections for Fiscal Year 2017–18

DATE PROVIDED TO CALIFORNIA STATE AUDITOR REQUESTED STATE APPROPRIATION PROJECTED NET EXPENDITURES NOTES
December 2, 2016 $14,000,000 $20,265,589 This projection represents the initial budget that ICOE provided to us, for which it had intended to seek approval.
February 23, 2017 13,434,120 17,605,142 ICOE generated this version of its budget to illustrate its request if the program were to maintain a $2 million operating reserve for equipment. ICOE reduced projected expenditures as a result of deciding not to pursue some network projects in fiscal year 2017–18 that it had planned when we began our audit.
February 23, 2017 11,434,120 17,605,142 ICOE generated this projection to illustrate its request if the program were not to maintain the $2 million operating reserve for equipment.
February 27, 2017 10,734,120 17,255,142 ICOE reduced projected expenditures based on our inquiry about double‑counting equipment expenditures when we reviewed the previous versions of its budget.
March 27, 2017 10,390,581 16,911,603 ICOE made revisions based on discrepancies we identified when reviewing the circuit planning tool used to determine the projected CENIC expense.

Source: California State Auditor’s analysis of K12HSN’s fiscal year 2017–18 budget worksheets and documentation pertaining to the budget assumptions.

Note: Reductions in ICOE’s requested appropriation do not always equal reductions in projected net expenditures because some expenditure reductions also affected projected Internet subsidies or resulted in other adjustments to revenue attributable to cost savings carried over from fiscal year 2016–17.

* For the purpose of identifying net expenditures, we excluded the federal and state subsidies advanced to K12HSN.


Some of these reductions pertain to expensive equipment upgrades that ICOE has decided not to fund but that were still included in its planning document supporting the budget. When we reviewed the detailed planning document that supports the CENIC line item in ICOE’s fiscal year 2017–18 budget for K12HSN, we found numerous instances in which the document’s cost projections were inconsistent with information contained in other K12HSN planning documents. Because of these inconsistencies, we concluded that ICOE was requesting funding for network projects it was no longer pursuing, was overestimating the anticipated costs of other planned projects, and was omitting in some cases the costs that it plans to incur in fiscal year 2017–18. For example, revisions ICOE made to its network maps after we began our audit indicate that ICOE is no longer planning a network upgrade between two network sites in Fresno County. However, the detailed planning document still included the project as a planned cost for fiscal year 2017–18, and this cost was included in ICOE’s budget request documents. As a result, the budget request was overstating K12HSN’s projected costs by about $100,000 for this project alone. When we raised this issue with the program’s chief executive officer, he confirmed the inconsistencies, and ICOE corrected its planning document and the resulting budget request. In total, these corrections resulted in a $344,000 reduction in ICOE’s budget request. We discuss these changes to ICOE’s plans for the network in greater detail in the following section of this report.

ICOE’s projected expenditures also double‑counted a portion of its budgeted equipment costs as both capital expenditures for ICOE and as part of the CENIC expenditures in the K12HSN budget. When we called this discrepancy to the attention of the chief executive officer, he acknowledged the error and reduced K12HSN’s projected equipment costs for fiscal year 2017–18 by $350,000 to correct for it.

Given these accuracy issues, there is a clear need for ICOE to provide additional transparency at the state level about the specific network decisions that affect its costs. Our review of the budget documents ICOE submitted to Education for fiscal years 2016–17 and 2017–18 indicates that ICOE is still presenting CENIC‑related network expenditures without identifying the specific network circuit upgrades associated with those costs. Considering how costly a single network upgrade can be, it is essential that ICOE provide a greater level of detail during the budgeting process to explain its rationale for those projects and whether more cost‑effective options are possible.

ICOE’s fiscal year 2017–18 budget documents include multiple scenarios for how various levels of state funding would affect the K12HSN program’s finances given ICOE’s expenditure projections. However, these documents do not include or propose any adjustments to anticipated expenditures to account for higher or lower funding levels. A more meaningful approach would be for ICOE to offer a range of possible expenditures to provide the State with options for adjusting its appropriation amount without adversely affecting the program. For example, ICOE could submit budget documents that include or exclude certain expensive planned network upgrade projects for the next fiscal year, along with information about specific upgrades that could be postponed and the postponements’ impact on the network.

Alternatively, ICOE could provide the State with the range of possible network capacities that would result from proposed upgrades, which could have very different cost implications. By including a detailed list of potential projects and assigning them priority levels for the upcoming year, along with those projects’ impact on K12HSN expenditures, ICOE would better inform the State about the need for the funds it is requesting, as well as the potential benefits or risks associated with various funding levels for the program.


Uncertainty Remains Regarding the Cash Flow and Sustainability of the K12HSN Program

The K12HSN program’s cash flow and the sustainability of its program expenditures have implications for the size of a prudent operating reserve. ICOE has expressed concerns in this area because of the timing of reimbursements for Internet subsidies that support the operation of the network. Typically, these reimbursements make up about 30 percent of K12HSN’s revenue, as was shown earlier in Table 1, and thus they represent a significant portion of its revenue. As we explain in the Introduction, there are two general processes by which K12HSN can recognize the financial benefit of Internet subsidies: the service provider can either credit the program up front through an advance, or the provider can bill K12HSN and refund the subsidy later. Service providers generally advance the value of the Teleconnect Fund subsidy for qualifying network services. In particular, CENIC advances the entire value of the subsidy to K12HSN for the portion of the program’s backbone costs that the subsidy covers. In fiscal year 2015–16, 98 percent of Teleconnect Fund subsidies were provided to the program as upfront discounts to ICOE. In contrast, service providers advanced only 28 percent of about $5.7 million in federal E‑Rate subsidies for that same year. ICOE had to therefore use other available funds to cover the remaining 72 percent of those costs until the federal government pledged to reimburse the service providers and those providers refunded the subsidies to K12HSN. These reimbursements have arrived historically in the fiscal year following the year in which ICOE initially incurred the expenditures.

According to the K12HSN program’s chief operating officer, ICOE felt the need to maintain a substantial operating reserve for K12HSN in case reimbursements were received later than expected. Figure 4 shows the timing of E‑Rate and Teleconnect Fund reimbursements for fiscal years 2013–14 through 2015–16. The majority of the reimbursements were associated with the E‑Rate program and arrived late in the fiscal year. This delay was especially evident in fiscal year 2015–16. In that year, ICOE had received only 22 percent of the total subsidy for the year as of April 30, 2016. By that same month in fiscal years 2013–14 and 2014–15, ICOE had already collected all or nearly all of its E-Rate funds. Consequently, ICOE had to support the cost of operations for the first three quarters of fiscal year 2015–16 using the program’s operating reserves. Further, two reimbursements representing 8 percent of the E‑Rate reimbursements ICOE was owed did not arrive until October 2016, after the close of the 2015–16 fiscal year. ICOE has expressed concern that these delays could become even longer in future years, placing additional pressure on K12HSN’s cash position. This concern has arisen because of changes the Federal Communications Commission made to the E‑Rate program in 2014, which include adding new technologies to those qualifying for the subsidy and updating the electronic application submission process. According to ICOE’s E‑Rate consultant, these changes lengthen the process for receiving reimbursements because of extensions to deadlines for filing funding applications. Although ICOE has historically had sufficient funds to support operations while waiting for E‑Rate reimbursements, the trend shown in Figure 4 indicates that the delays associated with those reimbursements pose cash‑flow challenges for the K12HSN program.

Figure 4
Cumulative Percentages of Internet Subsidy Revenue Received by ICOE
Fiscal Years 2013–14 Through 2015–16

Figure 4, a graphic showing the timing of Internet subsidy revenue received by ICOE during each fiscal year for the period from fiscal years 2013–14 through 2015–16, presented by the monthly cumulative total received.

Sources: California State Auditor’s analysis of statements for the bank account held on behalf of the K12HSN program by CENIC and ICOE’s financial records.

Note: For fiscal years 2013–14 through 2015–16, 96 percent of ICOE’s accrued subsidy revenue was for the federal E‑Rate subsidy, and only 4 percent was for the Teleconnect Fund subsidy. This figure addresses only subsidy revenue owed to ICOE in the form of reimbursements because the service providers otherwise advance the subsidies and ICOE does not pay for those portions of its Internet services.

* As of the end of fiscal year 2015–16, ICOE had received only 92 percent of the total subsidy reimbursement amount it was owed for fiscal year 2014–15. ICOE received the remaining 8 percent in October 2016.


The State has taken steps to address ICOE’s concerns about the timing of E‑Rate reimbursements. Before 2016 Education provided the annual state appropriation to ICOE in two installments. For example, Education paid 90 percent in March and January of fiscal years 2013–14 and 2014–15, respectively, and it paid the remaining 10 percent in December of the following fiscal year in both instances. In 2016 the Legislature amended EDC 11800 to direct Education to pay 75 percent of the total state K12HSN annual funding amount by August 31 of each fiscal year and the remaining 25 percent by January 31. This new funding schedule, which took effect in fiscal year 2016–17, addresses some of K12HSN’s cash‑flow challenges by providing a substantial portion of the state appropriation early in the fiscal year. However, as we discuss in greater detail later, the adjustment to the funding schedule alone may not be enough to sustain the program throughout the year, given ICOE’s planned expenditures.

In addition to maintaining an operating reserve to ensure that it can meet planned expenditures, ICOE’s proposed budget had designated portions of the K12HSN program’s excess cash for emergency replacement of equipment. ICOE initially informed us that its target operating reserve for this purpose was $2.5 million and that this amount was based on the statewide cost to replace failed equipment across network sites in 2013. However, since 2013 ICOE has already replaced the equipment for many of the network sites, staggering the useful lives of current network equipment and reducing the likelihood of a comprehensive failure.

ICOE agreed with our concerns regarding its methodology, but it has not yet developed a new way to determine a prudent equipment reserve. Developing this methodology will help the State determine whether a reserve level above the minimum necessary to ensure regular network operations is justified.

Using the K12HSN budget for fiscal year 2016–17 and the state appropriation amount of $8 million, we estimate that the program’s operating reserves at the end of the fiscal year will total almost $600,000 in addition to another $6.4 million expected from E‑Rate reimbursements in the following year. The fiscal year 2017–18 budget that ICOE submitted to the State in March 2017 requested $10.4 million. The budget identified $16.9 million in net expenditures for the year, some of which will be reimbursed as subsidy payments after the close of fiscal year 2017–18. Using these budgeted expenditures and the timing of past E‑Rate reimbursements, we projected the K12HSN program’s cash levels for fiscal year 2017–18 under two possible funding scenarios—the $8 million in the Governor’s proposed budget and the $10.4 million that ICOE submitted as its request.

Our analysis indicates that to avoid program deficits, ICOE will require an increase in the State’s historical appropriation in fiscal year 2017–18. We estimated that under the $8 million scenario, the program would begin the 2017–18 fiscal year with an operating reserve of nearly $600,000 and end the year with a deficit of $2.2 million. In contrast, with a $10.4 million appropriation, the program would end the year with a reserve of slightly more than $200,000. However, depending on the timing of E‑Rate reimbursements and how quickly ICOE implements some of its planned network upgrades, the K12HSN program could deplete its reserve at some point during fiscal year 2017–18, even with this larger appropriation. In past years, ICOE has addressed the issue of potentially problematic delays in subsidy reimbursements by carrying a large cash reserve for the program. Although it is aware of this issue, because the K12HSN program no longer holds a large reserve and because program revenue and expenditures have not aligned historically, it is important that ICOE closely monitor the program’s cash levels and adjust the timing of its expenditures as needed. In the next section, we discuss our concerns with aspects of the network planning process that have driven projections of the program’s future costs, including planned costs for some expensive network upgrades that have contributed to ICOE’s request for a larger state appropriation in fiscal year 2017–18.


Recommendations

Legislature

To help ensure continuous network operations while preserving state resources, the Legislature should appropriate to the K12HSN program an amount that does not exceed $10.4 million for fiscal year 2017–18. If the Legislature wishes to appropriate a lower amount for the program, it should direct ICOE to modify one or more of the planned network upgrades we highlight in the following section, either by delaying the upgrade to a subsequent fiscal year or by pursuing a less expensive option.

ICOE

To better inform decision makers at the state level about the amount of funding necessary to operate and maintain the network, ICOE should formally amend its annual budget documents by November 2017 to specify multiple potential levels of network expenditures for the coming year, and it should detail the specific network upgrades and project costs included in each scenario. As part of this process, ICOE should also provide information about how these upgrades will affect the network’s functionality.

To ensure that its projected program costs are as accurate as possible, ICOE should institute by November 2017 a formal practice for reviewing its budget planning document against its current network design plans and correct any inaccuracies before finalizing and submitting its budget.

To help ensure that the K12HSN program maintains the necessary amount of state funds in reserve, ICOE should prepare a formal methodology for a proposed equipment reserve that is based on the actual likelihood of equipment failure and the costs associated with replacing that equipment.

To ensure that it is able to continue critical network services with reduced operating reserves, ICOE should establish procedures to routinely monitor the K12HSN program’s cash balance and to evaluate upcoming costs. If at any point ICOE determines that it will be unable to fund its costs due to delayed subsidy payments, it should notify Education regarding the size and timing of the anticipated shortfall and postpone significant discretionary expenditures, such as upgrading network site equipment, until ICOE collects the subsidies it is owed.


Back to top




ICOE Needs to Improve Its Planning Processes in Order to Manage Network Development at the Lowest Possible Cost to the State

Key Points:


ICOE Chose Network Upgrades That Were Not the Most Cost-Effective Options Available

ICOE’s current process for determining the capacity needs for the network it manages does not sufficiently account for cost differences among available options. The cost of increases to network capacity depends on the service providers’ pricing, the location of the circuits being upgraded, and the magnitude of the capacity increase—the quantity of additional capacity that will be provided. When ICOE decides to increase the capacity of a circuit, it generally has the option to double, triple, quadruple, or provide a tenfold increase in the circuit’s capacity as measured in gigabits per second (gigs). In some circumstances, ICOE may choose to upgrade capacity using dark fiber, which is a type of circuit that can provide capacities up to and exceeding 100 gigs. Generally speaking, increasing a circuit’s capacity tenfold is more expensive than other options and also requires the acquisition and installation of new equipment.1 In some cases, when upfront installation costs are low—as they were for one of the circuits that ICOE procured for the network site at Santa Clara—procuring dark fiber may be less expensive than other available options. The program’s chief executive officer explained that ICOE has opted for tenfold capacity increases generally because it has had difficulty predicting the future use of a circuit and would rather err on the side of providing too much network capacity than too little.

By consistently pursuing large increases in capacity, ICOE has chosen costly upgrade options even though less expensive options were available for more intermediate capacity increases. We reviewed ICOE’s decisions to upgrade the capacities of 20 K12HSN circuits, as well as its decisions to upgrade another 10 circuits connected to those circuits for a total of 30 circuits. We included the additional 10 circuits in our review because of ICOE’s practice of upgrading some circuits to provide backup routes to the backbone for network sites to use when their primary circuits fail. We found that ICOE consistently selected the least expensive option among various service providers at a given capacity level. For example, when choosing a service provider for the 100‑gig circuit connecting its Chaffey and San Bernardino network sites, ICOE chose the least costly provider, which ICOE’s bid documents indicate was $164,000 less in annual service costs than the competing bids. However, on multiple occasions, ICOE decided to pursue capacity levels that were not cost‑effective. Specifically, in nine of 30 instances, ICOE chose a more expensive tenfold capacity upgrade option even though it received bids for other less costly incremental upgrade options.

For example, ICOE chose to upgrade one of its Contra Costa circuits from 10 gigs of capacity to 100 gigs. ICOE also requested and received bids from service providers for doubling the circuit’s capacity to 20 gigs. According to its planning documents, the decision to select the 100‑gig upgrade will cost the K12HSN program an additional $129,000 in one‑time costs and $30,000 more in annual costs. In another instance, ICOE chose to increase the capacity of one of its Alameda site circuits from 10 gigs to 100 gigs. ICOE also had the option of tripling the circuit’s capacity to 30 gigs. According to ICOE’s planning documents, selecting the 100‑gig option will cost K12HSN an additional $129,000 in one‑time costs and another $87,600 in annual costs compared to the cost of the upgrade to 30 gigs. Table 3 provides equivalent information and cost implications for the nine upgrade decisions for which we identified less expensive options. Among the nine upgrades we could compare to less expensive options, ICOE’s planned upgrades will cost a total of $206,056 in additional annually recurring costs and $2 million in additional one‑time costs. These additional costs are would result from ICOE’s practice of pursuing tenfold upgrades when increasing circuits’ capacities even when service providers have offered less expensive incremental increases.

Moreover, we identified some instances in which ICOE did not even consider less expensive upgrade options for the upcoming 2017–18 fiscal year. In four of the decisions we reviewed, which are also listed in Table 3, ICOE chose to upgrade circuits from 10 gigs to 100 gigs or dark fiber without requesting that service providers submit pricing details for 20‑gig, 30‑gig, or 40‑gig alternative options. ICOE thus cannot demonstrate that the results of its decisions will be cost‑effective, and we could not determine how much ICOE could have saved with an alternative option. Furthermore, federal regulations require competitive bids for all services supported by E‑Rate subsidies; agencies applying for these subsidies are directed to post their requests for those services on USAC’s website. Consequently, ICOE will not qualify for federal E‑Rate subsidies if it procures these alternative capacity options in the future without first competitively bidding them. Instead, its options are limited to moving ahead with the planned expensive upgrade or waiting to rebid the circuit in the following year. According to the program’s chief executive officer, ICOE did not request bids for the 20‑gig or 40‑gig capacity options for fiscal year 2017–18 because the circuits were already operating at 10 gigs, and ICOE’s general rule is to upgrade circuit capacity tenfold, so it decided to upgrade the circuits to 100 gigs or dark fiber. However, our review of these circuits’ actual usage determined that ICOE cannot justify this universal and costly approach when less‑expensive options exist.

Table 3
Comparison of Costs Between ICOE’s Circuit Upgrades and Alternative Options

INITIAL FISCAL YEAR OF OPERATION OR PLANNED OPERATION REASON FOR UPGRADE CAPACITY UPGRADE LEVEL CIRCUIT REVIEWED ICOE'S PLANNED UPGRADE COSTS ADDITIONAL COSTS:
DIFFERENCE BETWEEN PLANNED UPGRADE AND LEAST COSTLY ALTERNATIVE
ANNUAL COST ONE‑TIME UPFRONT COST ADDITIONAL ANNUAL COST (SAVINGS) ADDITIONAL ONE‑TIME UPFRONT COST
2015–16 Capacity 1 gig to 10 gigs Mendocino County Office of Education (COE) to CalREN Sunnyvale $119,040 $47,640 $63,840 $47,640
2015–16 Backup route 10 gigs to dark fiber Los Angeles COE to CalREN Los Angeles 88,800 364,000 (24,479) 363,000
2016–17 Capacity 10 gigs to dark fiber Chaffey Joint Unified High School District (JUHSD) to CalREN Los Angeles 42,000 633,000 (72,072) 484,979
2016–17 Capacity 10 gigs to 100 gigs Alameda COE to CalREN Oakland 193,200 129,000 87,600 129,000
2016–17 Capacity 10 gigs to 100 gigs Contra Costa COE to CalREN Oakland 84,000 129,000 30,000 129,000
2016–17 Backup route 10 gigs to dark fiber Chaffey JUHSD to San Bernardino County Superintendent of Schools (CSS) 100,800 438,000 22,728 204,767
2016–17 Backup route 10 gigs to dark fiber Santa Clara COE to CalREN Oakland 16,697 542,954 (12,103) 469,034
2016–17 Backup route 10 gigs to 100 gigs Fresno COE 1 to CalREN Fresno 149,940 129,983 80,052 128,983
2017–18 Lower network costs* 1 gig to 10 gigs Lassen COE to CalREN Sacramento 107,940 50,000 30,490 50,000
2017–18 Capacity 10 gigs to 100 gigs Fresno COE 2 to CalREN Fergus 96,000 202,500 In December 2016, when ICOE put these circuits out to bid for upgrade, it did not request bids for upgrades at capacity levels lower than 100 gigs/dark fiber. As a result, we could not compare the costs of ICOE’s chosen upgrades to those of lower capacity upgrades.
2017–18 Capacity 10 gigs to 100 gigs Riverside COE to Calhoun to CalREN Palm Desert 147,516 200,000
2017–18 Capacity 10 gigs to dark fiber San Bernardino CSS to CalREN University of California Riverside 78,000 400,000
2017–18 Backup route 10 gigs to 100 gigs Sacramento COE to CalREN West Sacramento 60,000 0
  Totals $1,283,933 $3,266,077 $206,056 $2,006,403

Source: California State Auditor’s analysis of ICOE planning documents.

Note: The information in this table is based on cost estimates used in the budget projections ICOE submitted to the State on March 29, 2017.

* ICOE upgraded this circuit from 1 gig to 10 gigs even though neither this circuit’s usage nor that of any other circuit partnered with it met ICOE’s criteria for an upgrade. ICOE was able to acquire the greater capacity at a lower cost than the cost for the original 1‑gig capacity. However, ICOE could have selected a lower‑capacity option of 2 gigs and reduced network costs even further.


Because It Had Incomplete Information, ICOE Could Not Justify the Costs of Its Most Expensive Upgrade Decisions

ICOE’s current methodology for determining the timing and magnitude of increases to network capacity does not ensure that those increases are justifiable. The K12HSN program’s chief executive officer explained that ICOE generally considers increasing a circuit’s capacity when it experiences usage peaks that approach or exceed 50 percent of total capacity. Although ICOE has not performed a formal analysis to support the rationale for its practice, the program’s chief executive officer explained that the 50 percent threshold is based on historical patterns of use at certain network sites, and it is intended to allow time for ICOE to upgrade the circuit before it reaches full capacity and the speed of data transmission is negatively affected.

Although ICOE chose to upgrade specific circuits identified in Table 3 because they were operating near or above 50 percent of their capacity, we found that ICOE’s available usage data did not demonstrate a need for the level of upgrade it had chosen. For example, Figure 5 shows usage data for a circuit that ICOE decided to upgrade because its usage reached 50 percent of its capacity. However, the usage data ICOE retained did not demonstrate an inherent need for a tenfold increase in capacity, as is ICOE’s general practice. By upgrading to a 20‑gig circuit instead, ICOE could have saved $129,000 in one‑time upfront costs and $30,000 annually at this site while likely meeting the site’s needs for at least the short term. We believe ICOE could do more to improve its ability to predict circuit use, thereby avoiding costs that are higher than necessary for providing capacity.

Figure 5
Capacity Levels and Costs for a Specific Upgrade to the K12HSN Program’s Network

Figure 5, a graphic showing capacity levels and costs for a specific K12HSN program network upgrade.

Sources: California State Auditor’s analysis of daily maximum usage data from Orion for the K12HSN Contra Costa County Office of Education network site and ICOE’s planning documents for K12HSN.


ICOE has not possessed sufficient data on historical network use to justify its decisions to pursue expensive capacity upgrades instead of more cost‑effective alternatives. The primary factor that ICOE considers to support its upgrade decisions is the use of its circuits. ICOE reviews usage data collected by CENIC’s monitoring software, Cricket, and by its own network monitoring software, SolarWinds Orion (Orion). These programs provide measurements of a circuit’s usage over time, expressed in terms of gigs. Cricket retains these data for roughly one year, and Orion retains the data for about two years. Both systems purge any data beyond their retention periods. ICOE could archive Orion data before the retention period expires but has not done so in the past. Although this two‑year period may be sufficient to enable ICOE to determine whether a circuit requires an upgrade—for example, if a circuit is being used at maximum capacity for multiple consecutive months—the period is not long enough to demonstrate the level of new capacity required. ICOE intends to use Orion to monitor the use of all K12HSN circuits against their capacity.

However, as part of our overall review of K12HSN circuit usage that included the 30 circuits discussed previously, we found eight instances in which ICOE either did not monitor a circuit or began monitoring it only after a significant amount of time had passed since its installation, thereby limiting any review of the circuit’s usage at the time of or for the period following its last upgrade. According to the program’s chief executive officer, any unmonitored circuits are the result of administrative error. When ICOE does not monitor a circuit with Orion, its only option for systematically evaluating historical usage is to rely on Cricket’s limited one‑year view.

Access to additional historical data may be useful in assessing a circuit’s needs as its usage increases. For example, if ICOE decides to upgrade a circuit’s capacity in a given year based on recent usage, by the time it revisits that circuit for another potential upgrade in two years, the original data that it used to determine the initial upgrade will have been purged. This information would have allowed ICOE to compare rates of growth in usage before and after it last upgraded the circuit in order to evaluate whether growth rates have been constant or are accelerating, as well as to account for any historical jumps in usage that might be followed by long, stable periods. Additionally, retaining historical data pertaining to the circuits serving high‑use network sites could help ICOE predict how circuits serving sites that currently have lower capacity needs may ultimately behave.

If it retains these data by periodically downloading them from Orion, ICOE could maintain indefinite access to circuits’ historical usage at minimal cost. The K12HSN program’s chief executive officer questioned whether retaining data for more than two years would add additional value but agreed that it would be easy to do. We believe that doing so can only improve ICOE’s ability to project the long‑term capacity needs of individual circuits, which in turn would help ensure that the costs of network upgrades are necessary and prudent.

In addition, ICOE has not systematically collected information from network sites about possible future use trends when deciding whether to increase those circuits’ capacities. ICOE maintains a database that stores information about the network capacities of school district circuits that connect to the network sites. According to the program’s chief executive officer, ICOE uses the information in the database to gain insight into the maximum possible usage it can anticipate from any district and the maximum possible usage at a network site. However, the program’s chief operating officer informed us that this information is incomplete because it is updated voluntarily by districts and not verified by ICOE. Furthermore, we question whether this information would be of any value in determining network sites’ actual capacity needs because of the minimal likelihood that all districts would be using their maximum capacities at the same time.

Rather than focusing on maximum capacity, ICOE could improve its current practices by regularly collecting information about districts’ actual expected usage. Ideally, this practice would include formal communications with staff members at network sites—where districts connect to the network—and focus on those sites’ knowledge of the network usage plans of their connecting districts. With a greater understanding of network sites’ future usage plans, ICOE would be in a better position to identify those upgrade capacities that are truly needed and how soon those needs would materialize. The program’s chief executive officer explained that ICOE currently discusses anticipated capacity needs with representatives of network sites, but it does so informally on a case‑by‑case basis and generally when the network site initiates the conversation. He also stated that ICOE is interested in further monitoring the use and anticipated use of the network by districts and believes this effort would add value to its decision process, although it has not yet implemented this monitoring.


ICOE’s Practice of Providing Equivalent Capacity Backup Routes Results in Additional Costs When ICOE Increases the Capacity of a Primary Circuit

To ensure continuous access to the backbone for network users, particularly when dealing with power outages or other network interruptions, ICOE has structured the network to allow at least two routes between most locations and the backbone: a primary circuit and at least one alternative route. Although this strategy minimizes the likelihood that users will lose access to the backbone, it also means that in most cases an increase to a primary circuit’s capacity necessitates a commensurate increase to one or more other circuits to accommodate any growth in traffic expected for the primary circuit if that circuit fails. These increases would be necessary regardless of the level of traffic those other backup circuits incur for their regular dedicated purposes. In most cases, a site’s backup route involves a circuit to a neighboring network site and then using that site’s connection to the backbone.

This structure has significant cost implications in situations where neighboring sites have different levels of usage. For example, according to the program’s chief executive officer, usage levels for one of the circuits at the Los Angeles Unified School District (LAUSD) network site prompted ICOE in fiscal year 2015–16 to plan to install a new dark fiber circuit connecting the site to its neighboring network site at the Los Angeles County Office of Education, as well as to upgrade one of the county office site’s circuits from 10 gigs of capacity to dark fiber to ensure an effective backup route. As of March 2017, the LAUSD site’s circuit was operating at roughly 17 gigs, while the county office circuit was operating only at around 7 gigs. Upgrading the lower‑usage county office circuit to dark fiber—which represents the equivalent of 100 gigs in terms of potential capacity—in order to provide a backup route is expensive. According to ICOE’s planning documents, the combined cost for both circuit upgrades in their first year of operation will be greater than $1 million.

In other instances, as an alternative means to provide backup routes, ICOE chose to add new circuits to the network rather than to upgrade existing circuits. This approach is an alternative to ICOE’s frequent practice of pairing network sites and their circuits to provide backup routes to the backbone. Under this approach, once ICOE has decided to upgrade a network site’s primary circuit, it also installs a circuit between the site and another part of the backbone to provide a new backup route.

ICOE plans to add new circuits to provide backup routes for some network sites whose existing circuits ICOE has already upgraded or plans to upgrade. This practice can be more efficient than pairing network sites with different usage levels because doing so means the decision to upgrade an existing circuit to a higher capacity requires only a single additional circuit upgrade for redundancy rather than two upgrades. However, given the costs associated with high‑capacity upgrades, this practice still has significant cost implications for the K12HSN program. For example, ICOE planned to upgrade the existing circuit connecting the Santa Clara network site to the network backbone because of this circuit’s current usage. To provide a backup route for the upgraded circuit, ICOE planned to disconnect Santa Clara from its current network site partner and install a second circuit directly to the backbone. ICOE’s planning documents indicate that this additional circuit will cost roughly $550,000 in its first year.

Although its practice of providing backup routes adds significant costs for operating the network, ICOE cannot demonstrate the actual benefit this approach provides. We found that nearly all 86 network sites have backup routes for their circuits. Further, in five of the instances we reviewed wherein ICOE selected upgrades to 100 gigs or dark fiber at significant additional cost, it did so to provide backup routes consisting of completely new circuits or as upgrades to existing circuits whose usage was significantly less than the circuits with which they were paired. Table 3 lists these circuits and their costs. As we discuss in the following major section of this report, ICOE has not systematically tracked or reported on the frequency or duration of network outages; therefore, it is unclear how often the network actually relies on backup routes to keep schools and districts connected. As a result, the benefit achieved by the significant costs of the current backup route structure is also unclear.


Missed Internet Subsidies Resulted in Some Extra Costs for the State

ICOE has not ensured that it applies for the E‑Rate subsidy for all of its network upgrades. As the Introduction describes, CENIC manages all service provider contracts on behalf of ICOE and the K12HSN program. Although CENIC is the party of record in the E‑Rate application process, which it administers with the assistance of a third‑party contractor, ICOE is ultimately responsible as the K12HSN lead agency for ensuring that CENIC secures the E‑Rate subsidy whenever possible.

We reviewed documentation for 20 circuit upgrades to determine whether ICOE ensured that CENIC applied for the E‑Rate subsidy and, if so, whether CENIC submitted the subsidy applications within program deadlines. During this process, we identified one instance in which ICOE decided to proceed with a circuit upgrade that did not qualify for an E‑Rate subsidy because the circuit was procured without competitive bidding. Specifically, ICOE upgraded one of its circuits at the network site at Santa Maria Joint Unified High School District from 1 gig to 10 gigs of capacity. The K12HSN program’s chief executive officer told us that to meet the E‑Rate requirement, ICOE had initially sought competitive bids for the upgrade. However, the only bid ICOE received was from a service provider that would have difficulties installing the upgrade because of its limited infrastructure in the area.

According to the chief executive officer, ICOE asked its existing provider to install the upgrade after determining that the circuit’s peak use was at its capacity and ICOE anticipated that the provider could install the upgrade quickly. Because this procurement with the existing service provider occurred without competitive bidding, the circuit is not eligible for E‑Rate for the duration of that contract. We estimate that nearly $100,000 of the circuit’s annual costs would have been reimbursed by E‑Rate subsidies. After the circuit was installed in July 2016, ICOE again sought competitive bids for the same circuit in December 2016 as part of the most recent E‑Rate application period, and it received a bid that it plans to implement in fiscal year 2017–18. At this time, ICOE does not have a planned date to disconnect the circuit, which continues to incur unsubsidized costs.

In addition, CENIC informed us of one other instance beyond the 20 circuits we reviewed in which ICOE did not secure an E‑Rate subsidy for a circuit it procured. The K12HSN program’s chief executive officer clarified that the circuit was procured outside of E‑Rate as a temporary solution because the existing 1‑gig circuit was operating at capacity and ICOE’s planned upgrade was not occurring promptly. This circuit was in use between May 2015 and December 2016, and we estimate that E‑Rate could have reimbursed $32,000 of the circuit’s costs to the State in its first full year of service.

For the other 19 circuits we reviewed, we used application identification numbers provided to us by CENIC to track those applications through the various phases of the E‑Rate process for which CENIC is responsible. Two of the circuits—although physically part of the network—are owned and paid for by county offices of education, meaning that ICOE in its role as the K12HSN program administrator is not responsible for securing subsidies for those circuits. For the remaining 17 circuits, CENIC appears to have complied with deadlines for posting circuit projects for competitive bidding, submitting funding applications, and verifying to USAC that Internet services had begun. Because USAC does not require applicants to identify the individual broadband products for which they are seeking subsidies, we could not determine definitively whether specific circuits were included in the E‑Rate application materials we reviewed. Nevertheless, we were able to link a majority of the circuits we reviewed to the application forms using contract numbers.


Recommendations

ICOE

To better guarantee that network upgrades are necessary and are achieved at the lowest possible cost to the State, ICOE should develop a formal methodology for reviewing circuit capacity needs. This methodology should include consideration of multiyear trends in network traffic and the implications prospective upgrades may have for other parts of the network. Doing so would not only assist ICOE when determining the magnitude of circuit upgrades relative to the cost involved, but would also help ICOE determine whether it can delay upgrades until it can establish those upgrades’ eligibility for E‑Rate subsidies.

To provide as many options for network upgrades as possible and to help ICOE provide the most cost‑effective upgrade options without risking its eligibility for subsidies, ICOE should adopt the practice of requesting bids at all feasible levels of capacity upgrades as opposed to only those levels that represent a tenfold increase in circuit capacity.

To help facilitate the review of circuit capacity needs, ICOE should maintain historical data for network traffic as long as technically feasible. It should also ensure that its monitoring software includes all network sites.

To reduce the risk of having to react to large increases in network traffic, ICOE should formalize a process to include input from network site administrators during network upgrade planning.


Back to top




Measurement of the Program’s Effectiveness Has Omitted Key Information, and Oversight Has Been Inconsistent

Key Points:


Tracking and Reporting of the Network’s Performance Is Not Comprehensive

K12HSN’s Primary Responsibilities Include the Following:

Source: Education Code section 11800.

ICOE’s reporting on the network’s performance has not included performance measures for some of the goals and services established for the K12HSN program in state law. EDC 11800 establishes primary responsibilities for ICOE as the lead educational agency responsible for administering the K12HSN program. In addition, EDC 11800 assigns the superintendent, who oversees Education, responsibility for measuring the success of the K12HSN program, and makes the K12HSN advisory board responsible for recommending policy direction and providing broad operational guidance to the superintendent and ICOE. As shown in the text box, EDC 11800 requires K12HSN to provide critical services, including reliable and cost‑effective Internet service and interconnectivity among public school entities.

To report on the network’s performance, ICOE and CENIC—at the direction of ICOE—provide network capacity statistics to the K12HSN advisory board and Education through annual reports and presentations during the advisory board’s quarterly meetings. For example, ICOE provided information in its annual reports describing the number of network sites and their circuits’ capacity, as well as the percentages of schools and districts connected to the network. More recently, ICOE has also reported on the connectivity and capacity of individual districts and schools. However, neither the annual reports nor the information ICOE has provided at the advisory board meetings addresses the reliability of the network.

In its annual reports, ICOE stated that network operators at CENIC are able to control latency, a measurement of data delay, and packet loss, a measurement of data sent compared to data received. Videoconference users, for example, experience latency as a loss of video and audio synchronization, which may result in participants’ talking over each other because of the delay. With even a small amount of packet loss, a videoconference may periodically lose audio, and images may freeze. Industry standards support both measures as appropriate methods to gauge a network’s quality of service. However, ICOE has not provided statistics in its annual reports regarding the latency or packet loss experienced over the network. Additionally, none of the materials provided to the advisory board during the period from February 2012 through November 2016 contains any information on packet loss or latency over the portions of the network funded by the K12HSN program. Although we identified a reference to latency in a document from a meeting in May 2014, that document did not provide information on latency between CalREN and the network itself. Instead, it provided some measurements of latency between the backbone and the testing servers in Chicago as part of efforts to prepare for the California Assessment of Student Performance and Progress computer‑based testing described in the Introduction.

Moreover, ICOE has not reported to Education or the advisory board on the network’s overall reliability—including the frequency, duration, location, or cause of network service outages or interruptions. Doing so would help those responsible for overseeing ICOE’s administration of the K12HSN program determine whether network reliability is increasing or decreasing, and it would also ensure that ICOE effectively addresses any potential reliability issues associated with specific service providers or geographical regions.

According to the program’s chief operating officer, CENIC formally notifies ICOE when a network connection is not functioning, and ICOE monitors the outages in real time. However, at this time, ICOE tracks these outages informally, and the program’s chief executive officer stated that ICOE does not formally report them to Education. The chief operating officer also stated that ICOE attempts to ensure the network’s reliability by providing multiple connections to diverse sites using different service providers, so that if one provider’s service is not operating at full capacity, others may continue to function without interruption. However, because ICOE does not report on the frequency, duration, location, or cause of network outages, nor on the latency and packet loss experienced by the network, it cannot demonstrate the effectiveness of its efforts to monitor reliability.

ICOE has also not reported on another critical aspect of managing the network: the cost‑effectiveness of the Internet service it provides through CENIC on behalf of the State. Specifically, ICOE has not reported the costs of providing Internet service to its users in a manner that allows stakeholders to compare annually whether ICOE is doing better or worse in providing network connectivity at the lowest possible cost. Additionally, as we explain earlier in this report, ICOE has not provided sufficiently detailed information in its budgeting documents about the need for and cost implications of specific circuit upgrades.

ICOE has noted in its annual reports that contracting with CENIC results in deep discounts for the K12HSN program because of the volume of CENIC’s purchases of Internet services from service providers. However, ICOE has not quantified these savings or measured them against costs associated with CENIC’s administration of the network. As a result, ICOE has not established whether and why the decisions it has made for K12HSN are the most cost‑effective. When we asked about the lack of reporting, the program’s chief operating officer explained that ICOE could provide data on the historical cost per unit of network connectivity, but Education has not asked for this type of information in ICOE’s annual reports about K12HSN. Nevertheless, she acknowledged that ICOE could do more to demonstrate the cost‑effectiveness of its Internet service. We believe that ICOE could accomplish this objective by monitoring and reporting on network costs not only in terms of available capacity, but also per unit of network capacity actually used.

As mentioned earlier, the superintendent who oversees Education is primarily responsible for measuring the success of the K12HSN program. However, Education does not conduct comprehensive monitoring of network reliability, including frequency and duration of network interruptions or other performance measures. We spoke with the current and former directors of Education’s Educational Data Management Division, who informed us that there is no State General Fund allocation provided to Education for the work associated with overseeing K12HSN. The current director explained that Education monitors the network’s performance by reviewing outage reports from ICOE and testing summary reports from the private contractor responsible for the State’s online testing. We found that the testing summary reports are focused primarily on the number of tests started and completed during a given day and do not identify or categorize network outages or delays. The outage report we reviewed demonstrates the percentage of time per month that individual network sites are operating, but this report does not identify the date, duration, frequency, or cause of any outages affecting network availability. Although the current director provided us with the outage report for March 2017, he was unable to provide these reports from previous periods to demonstrate Education’s review. The former director stated that ICOE is not required to report—and EDC 11800 does not allow Education to compel ICOE to report—any specific performance measures or to use any specific format or frequency, and the current director agrees with this position.

We disagree with Education’s position that it is not allowed to require ICOE to report on specific performance measures. State law directs Education to measure the success of the program, which is not possible without collecting performance data. In addition, Education’s original grant award notice, which was signed and accepted by both Education and ICOE, identifies ICOE’s responsibilities related to performance measurement as the lead agency for K12HSN. This grant agreement was fairly robust and required ICOE, as a condition of receipt of the grant funds, to cooperate with Education’s evaluation of the program and comply with Education’s reporting requirements, including providing all requested documentation to Education in a timely manner. However, subsequent grant awards do not include this language; instead, they state in general terms that ICOE’s responsibilities include technical oversight of the project, and that funds from the grant award are to be used to facilitate collaboration among existing projects and services.

ICOE can use its relationship with CENIC to help it provide the types of information that will assist Education to measure the success of K12HSN as EDC 11800 requires. Currently, ICOE’s contract with CENIC does not require CENIC to report on any specific performance measures. Because CENIC’s own service expectations document states that it provides periodic reports to its board of directors on backbone outages, latency, and packet loss, we expect that it could report this same information to ICOE. If CENIC routinely shared performance reliability measures, ICOE could then report this information to users, Education, and other stakeholders to help demonstrate the program’s quality and effectiveness.


ICOE Has Reported Other Measures Inconsistently

In contrast to the necessary measures of network performance and reliability that ICOE has omitted from its reporting, ICOE has demonstrated the ability to report on other measures and services required by EDC 11800. However, it has not shared this information on a consistent basis in its annual reports or in its quarterly presentations to the advisory board. As required by EDC 11800, K12HSN’s advisory board issued a report in 2007 that identified recommendations for measuring the success of the network. Despite those recommendations, ICOE has not consistently collected and reported on some of the metrics this report recommended. For example, the report recommended quantifying the number of videoconferencing services that K12HSN provided since the inception of the program as well as establishing a baseline against which ICOE could then monitor the goal of increasing videoconferencing use each year. Nonetheless, ICOE has not reported these statistics consistently. ICOE informed us that Education has not asked for this information.

In May 2014, ICOE presented a report to the advisory board that contained statistics on the number and types of videoconferences K12HSN supported with its video application over the preceding eight years. ICOE did not include or update these statistics in its 2015 annual report, and we identified that it presented only one subsequent update to its advisory board in August 2016. Further, we noted inconsistencies when comparing some of the statistics reported in this update with statistics previously provided in May 2014. In response to our questions about these discrepancies, the K12HSN program’s chief operating officer informed us that ICOE changed its process for compiling this information, resulting in a different presentation of the statistics in 2016. When we asked ICOE about its inconsistent reporting on videoconferences, the program’s chief operating officer confirmed that this was a failure on ICOE’s part, but she said that ICOE would have reported these data had Education indicated that such information would be helpful. She indicated that ICOE is able to report on several aspects of videoconferencing use and provided us with some recently compiled statistics, which ICOE plans to include in its 2016 annual report.


Recommendations

Education

To increase transparency in the K12HSN program and help ensure that the State has sufficient information to measure the program’s effectiveness, Education should direct ICOE to report annually on specific performance measures. These performance measures should include the following metrics:

Education should stipulate that the receipt of grant funds is conditional based on the recipient’s agreement to provide these measures and other information deemed necessary by Education, either on request or at regular intervals determined by Education. If Education believes that it does not currently have legal authority to direct ICOE to report on this information, it should seek legislative change to obtain that authority.

ICOE

To better support future reporting efforts for the K12HSN program, ICOE should amend its contract with CENIC to require CENIC to report on specific network performance measures, including the frequency, cause, location, and duration of network outages or interruptions.


Back to top




Other Areas We Reviewed

To address the audit objectives approved by the Joint Legislative Audit Committee (Audit Committee), we reviewed the subject areas shown in Table 4. The table indicates the results of our review and any associated recommendations we made that are not already discussed in the other sections of this report.

Table 4
Other Areas Reviewed as Part of This Audit

CENIC’s Agreements With Service Providers

We reviewed 16 service-level agreements, one from each service provider currently serving one or more network sites. All of the agreements contain specific protections for CENIC in the event of a service outage or interruption. The agreements generally require CENIC to raise issues with the service provider before the provider will consider crediting CENIC’s account. CENIC’s chief financial officer informed us that service providers have issued credits to K12HSN and provided two examples from fiscal year 2016–17 of credits approximating $1,600 and $1,800. However, ICOE was unable to provide us with the frequency with which it received credits because of service outages or interruptions and had to seek this information from CENIC.

Recommendation

To ensure that the K12HSN program receives all of the service credits to which it is entitled, ICOE should amend its contract with CENIC to clarify CENIC’s responsibilities in this area, including reporting to ICOE about network outages or interruptions and requests for credits to service providers, along with the outcomes of those requests.
E-Rate Deposits

We reviewed 15 cash receipts pertaining to nearly $9 million of E-Rate subsidies received for the period from fiscal years 2013–14 through 2015–16. CENIC deposited these amounts—received from various service providers—in the bank account it holds for K12HSN. For each receipt, we verified that the amount of the check matched the amount CENIC deposited to the bank account. In addition, we determined that CENIC deposited each check promptly after indicating that it had received payment.
K12HSN Staffing and Administrative Costs
  • Because ICOE’s role as the lead educational agency for the K12HSN program is to provide administrative services and project oversight and not to provide or maintain Internet access—responsibilities that are assigned to the service providers and CENIC—we considered the entire amount ICOE spends on K12HSN’s operations and personnel as administrative costs. We summarize the major categories of these expenditures in Table 1.

  • Administrative costs, primarily staff compensation, have been relatively stable. K12HSN has maintained consistent staffing levels, with minimal fluctuation in overall salary costs during fiscal years 2013–14 through 2015–16. We calculated that the K12HSN program’s total staffing increased only minimally during our review period, from 13.7 full time equivalent (FTE) employees at ICOE during fiscal year 2013–14 to 14.4 FTEs during fiscal year 2015–16. Total salary and other monetary compensation for those employees equaled $1.05 million and $1.07 million in fiscal years 2013–14 and 2015–16, respectively.

  • Because K12HSN staff members are ICOE employees, we compared their compensation against ICOE’s salary schedules and found that program salaries were all within the appropriate salary ranges. We noted that most management employees are compensated at the higher steps of their pay ranges, whereas classified employees are predominantly compensated at the lower steps of their pay ranges. Therefore, we also reviewed K12HSN employees’ salary histories along with salary schedules and pay policies to ensure that ICOE complied with its policies in determining salary levels. We found that ICOE complied with its policies and compensated employees at the appropriate ranges and steps from fiscal years 2013–14 through 2015–16. Further, our review identified that several of the management employees had longer tenures with ICOE, meaning that they are at or near the top of their respective salary schedules. In addition, the management salary schedule has fewer steps, resulting in employees’ reaching the upper ranges in a shorter amount of time.
ICOE’s Memorandum of Understanding With Butte County Office of Education
  • ICOE also has a memorandum of understanding with the Butte County Office of Education (Butte) for the services of an E‑Rate administrator, who works in collaboration with ICOE and Education to provide E‑Rate and Teleconnect Fund expertise and training to K12HSN, county offices of education, and school districts throughout the State. The scope of services originally involved multiple individuals performing Butte’s functions, but has since been reduced to the current individual E‑Rate administrator position.

  • For fiscal year 2016–17, in addition to the budgeted amounts of $150,000 in salary and benefits and $32,534 for direct expenditures such as materials and supplies, travel, conferences, and dues and memberships, the memorandum of understanding requires ICOE to reimburse Butte for $23,551 in other overhead and indirect costs. However, we noted that the E‑Rate administrator does not appear to rely on Butte’s office resources in performing her duties, as her work involves a combination of telecommuting and traveling throughout the State. Therefore, we question the appropriateness of ICOE paying some of these overhead and indirect costs when it could consider adding this position to its complement of existing staff and contractors, resulting in a lower total cost to the State.

Recommendation

To ensure efficient use of state funds, ICOE should conduct a cost and benefit analysis of its memorandum of understanding with Butte to determine whether it represents the most cost‑effective approach to providing the program activities it covers.
Ongoing Use of ICOE
  • The current and former directors of the Educational Data Management Division at Education confirmed that Education has never rebid the K12HSN grant since awarding it to ICOE in 2004, nor has it sought proposals from other local educational agencies. The former director informed us that during her tenure she considered reopening the bid, but very few counties expressed interest. In addition, the directors and their staff explained that the risk of interruptions to the network that would result from changing local agencies outweigh any potential benefit, and they questioned the need to make any changes, in large part because they believe ICOE has been running the program effectively.

  • Nevertheless, Education cannot justify its position because of the lack of consistent reporting on the performance of the network, which we discuss in the previous section of the report. Our recommendation related to improved performance monitoring for the K12HSN program will assist Education in assessing the potential risks and benefits of rebidding the contract.
BIIG Funding
  • Our review of legislatively mandated reports on the planning and progress of the BIIG implementation leads us to conclude that ICOE’s process for administering the grants and its methodology for identifying grant recipients has generally been consistent with the requirements the Legislature implemented as part of the BIIG program’s funding.

  • According to the December 2016 connectivity report prepared by ICOE, 182 sites had been funded with BIIG 1.0 funds, with about $19 million of the $26.7 million in total funds spent. In February 2017, the program’s chief executive officer reported to the advisory board that a total of 363 active projects pertained to BIIG 1.0 and 2.0. He reported that approximately $21 million in BIIG 1.0 funds and about $6.9 million of the total $50 million in BIIG 2.0 funds had been spent thus far.


Back to top




Scope and Methodology

The Audit Committee directed the California State Auditor to conduct an audit of ICOE’s operation of the K12HSN program and of the funding for the program. The audit analysis that the Audit Committee approved contains 11 objectives. Table 5 lists the objectives and the methods we used to address them.


Table 5
Audit Objectives and the Methods Used to Address Them
Audit Objective Method
1 Review and evaluate the laws, rules, and regulations significant to the audit objectives. Reviewed relevant laws, rules, regulations, and procedures.
2 Identify and evaluate the roles and responsibilities of Education, ICOE, CENIC, the K12HSN advisory board, and any other state or local agencies or contractors that may be involved in managing or administering K12HSN.
  • Identified and evaluated the roles and responsibilities of Education, ICOE, CENIC, and the K12HSN advisory board as prescribed in the Education Code. We also reviewed roles and responsibilities defined in contracts and other relevant agreements between ICOE and third parties.

  • Reviewed the meeting frequency and attendance of the K12HSN advisory board over the past five years.

  • Determined that the K12HSN advisory board’s meeting frequency over the past five years complied with state law but that meetings consistently included fewer than the advisory board’s 12 members. The K12HSN program’s chief operating officer told us that the district superintendents on the advisory board often have other commitments that prevent their attending advisory board meetings.
3 Determine what services K12HSN contracts for with CENIC or others. Ascertain what performance measures Education and ICOE have developed for ensuring that K12HSN’s benefits are maximized, including whether contracting with CENIC represents the most cost‑effective option for the State.
  • Reviewed ICOE’s master contract with CENIC, as well as all amendments and addenda to that contract. We also determined whether ICOE’s contracts with CENIC contained any specific performance measures for contracted services CENIC provides to ICOE.

  • Interviewed staff at ICOE and Education about the performance measures they review to ensure that K12HSN’s benefits are maximized through the agreement with CENIC. Although ICOE has reported on certain measures of network performance and has broadly referenced the financial benefit of contracting with CENIC, it has not quantified the benefits or measured them against costs of the contract.

  • Reviewed a selection of CENIC’s service‑level agreements with individual Internet service providers. We also identified whether those agreements contain protections in the event of service outages or interruptions.

  • Determined that the provisions in the contract satisfy requirements in state law with respect to intellectual property rights and interest earned on state funds.
4 Assess how ICOE ensures that Internet subsidies are maximized and fully and properly credited to K12HSN.
  • For a judgmental selection of 20 upgrade projects from fiscal years 2012–13 through 2015–16, reviewed E‑Rate application information to ensure that CENIC claimed E‑Rate discounts for the projects.

  • Reviewed the projects’ application forms and determined whether the applications complied with USAC’s deadlines.

  • For a selection of 26 school districts served by the upgraded circuits, confirmed that ICOE complied with E‑Rate subsidy requirements to obtain authorization from the school districts included in ICOE's E‑Rate applications for those projects.

  • For each of the past three fiscal years, reviewed a judgmental selection of five E‑Rate deposits to ensure that funds were properly credited to the K12HSN program.
5 Identify all of K12HSN’s funding sources for the most recent three fiscal years and do the following:  
a. Determine K12HSN’s major categories of expenditures and evaluate whether its spending practices are reasonable, cost‑effective, and consistent with state policies.
  • Obtained and reviewed K12HSN’s audited financial statements and accounting data for fiscal years 2013–14 through 2015–16. In addition, we determined categories of revenue and expenditures for each of the past three fiscal years and identified any trends.

  • Calculated and compared the proportion of E‑Rate and Teleconnect Fund subsidies advanced to K12HSN for fiscal years 2013–14, 2014–15, and 2015–16.

  • For a judgmental selection of 20 network circuit upgrades and 10 additional upgrades to related circuits ICOE approved for fiscal years 2015–16 through 2017–18, reviewed and determined whether the selected service provider and selected level of service or equipment were the most cost‑effective options available.

  • Interviewed ICOE staff regarding rationales for upgrade selection decisions.
b. Assess whether ICOE’s administrative costs for K12HSN are reasonable and in accordance with its grant agreement. From our review of expenditures, identified any major administrative costs besides staffing costs. We also determined whether the activities underlying those costs were reasonable and necessary for K12HSN’s mission.
6 Review ICOE’s and Education’s budgetary planning practices for K12HSN, including how they determine the need for capital investments and operating reserves. Assess whether the entities’ budgetary planning activities follow best practices and whether the entities use consistent methodologies.
  • Identified the Government Finance Officers Association’s Recommended Budget Practices: A Framework for Improved State and Local Government Budgeting as a source for best practices.

  • Reviewed ICOE’s budget documents for fiscal years 2013–14 through 2016–17 and proposed budget documents for fiscal year 2017–18. Further, we reviewed both internal planning documents and the budget documents that were submitted to the State. We determined whether these documents and ICOE’s process for creating them conformed to best practices.

  • Interviewed relevant individuals to determine the State’s processes for reviewing ICOE’s budget submissions and for setting state funding levels for the program.

  • Identified ICOE’s methodology for identifying network circuits that require capacity upgrades and determined the reasonableness of that methodology.

  • Determined whether ICOE has developed a long‑term plan for the network structure and whether that plan is reasonable.

  • For the selection of 20 upgraded network circuits and 10 other upgraded circuits connected to them reviewed under Objective 5, documented available maximum usage data for those circuits. We estimated the new costs these upgrades will incur for the K12HSN program as compared to the costs of less expensive alternatives ICOE did not pursue. We also determined whether network usage at those circuits warranted the upgrades ICOE selected.

  • Interviewed ICOE’s staff to determine their reasons for approving upgrades.

  • Reviewed the capacity of the surrounding network sites at the time of the upgrade and identified how they were affected by the upgrade.

  • Reviewed K12HSN advisory board meeting minutes and determined the extent to which ICOE reviewed the upgrade with the K12HSN advisory board, as well as the timing of any such review and the board’s input on the upgrade. We concluded that any discussion of network upgrades with the advisory board occurs at a general level and does not include specific costs of and justifications for those upgrades.

  • Identified any network circuits with maximum traffic levels substantially higher than the 50 percent threshold ICOE set for pursuing capacity upgrades during ICOE’s project planning stages for fiscal year 2016–17 and determined that ICOE has consistently taken steps toward upgrading those circuits.
7 Determine K12HSN’s operating reserves for the most recent three fiscal years. To the extent possible, do the following: Reviewed the program’s audited financial statements to identify operating reserve levels for the K12HSN program for the period from fiscal years 2010–11 through 2015–16.
a. Calculate K12HSN’s operating reserve at the end of fiscal year 2016–17.
  • Using those financial statements and ICOE’s accounting data for the K12HSN program, as well as its projected expenditures for fiscal year 2016–17, estimated the program’s operating reserve as of June 30, 2017. Using that information and ICOE’s projected expenditures for fiscal year 2017–18, we estimated the program’s operating reserve level as of June 30, 2018, under different funding scenarios.

  • Reviewed CENIC’s records to identify the timing of E‑Rate subsidy reimbursements from service providers during the past three fiscal years. We also determined when K12HSN received those funds.
b. Determine what a prudent annual operating reserve should be.
  • Interviewed staff and documented ICOE’s methodology or rationale for maintaining its level of operating reserves.

  • Interviewed responsible parties at Education to obtain their perspective about what they believe a prudent reserve should be.

  • Using the above procedures, determined whether ICOE’s proposed reserves for K12HSN are necessary and prudent.
8 Determine the core duties of K12HSN staff. Further, determine K12HSN staffing levels and costs for the most recent three fiscal years and perform the following:  
a. Ascertain whether staffing levels have increased and the reasons why. Reviewed K12HSN personnel and payroll information for the past three fiscal years and determined staffing levels and compensation for each year. We also determined the reasons for any increases or decreases in compensation and documented any changes in staffing levels.
b. Review staffing plans and assess whether these plans are adequate. Reviewed current budgetary staffing documents and job descriptions documenting the responsibilities of K12HSN staff for the past three fiscal years.
c. Determine how staff compensation rates are set and whether those rates are commensurate with their responsibilities.
  • Compared actual compensation for K12HSN staff to ICOE’s salary schedules for the past three fiscal years.

  • Determined ICOE’s methodology for setting compensation levels, including setting compensation for ICOE positions unique to the K12HSN program. Further, we determined whether ICOE followed those policies when setting compensation for K12HSN employees.
9 Review the grant agreement between Education and ICOE and do the following: Determined from interviews with ICOE and Education staff that no overarching grant agreement or specific program requirements exist outside of the Education Code.
a. Determine what process Education follows in awarding the K12HSN grant. Obtained grant award letters issued to ICOE and interviewed staff at Education about its selection process and whether it has considered other potential grantees since first awarding the grant to ICOE. We then evaluated the reasonableness of Education’s decision.
b. Establish how Education calculates the annual grant award and whether that calculation takes into consideration any operating reserves. Interviewed staff at Education to determine the type of information it reviewed each year for fiscal years 2013–14 through 2015–16 before providing the grant and whether that information included consideration of ICOE’s operating reserve.
c. Ascertain whether Education has measured ICOE’s performance against the grant agreement. Determine whether opportunities exist for Education to improve its monitoring.
  • Obtained and reviewed K12HSN’s annual reports, advisory board meeting minutes and presentation documents, and program updates to determine the extent to which ICOE reported to Education and the K12HSN advisory board on performance measures outlined in the Education Code.

  • Determined whether more specific guidance from Education is needed to improve the consistency and detail of reporting on the K12HSN program’s performance and cost‑effectiveness.
10 Determine how ICOE and/or Education monitor CENIC’s performance and whether these entities are providing all necessary oversight.
  • Interviewed staff at ICOE and Education to determine what service metrics they monitor with respect to CENIC’s performance in managing K12HSN’s physical network.

  • Determined whether ICOE’s contract with CENIC should be amended to require reporting on specific measures related to the performance of the network, such as the network’s reliability.
11 Review and assess any other issues that are significant to the audit.
  • Reviewed relevant state laws that established the funding for BIIG 1.0 and 2.0 to identify roles and responsibilities for administering the grants.

  • Interviewed ICOE staff regarding ICOE’s methodology for awarding the grant funds to California public schools and school districts.

  • Obtained and reviewed legislatively mandated reports related to these grants. We determined that ICOE’s reported process for administering the grants and its methodology for identifying recipients were in line with funding requirements.

Sources: California State Auditor's analysis of the Audit Committee's audit request number 2016‑129 and analysis of information and documentation identified in the table column titled Method.


Assessment of Data Reliability

In performing this audit, we relied on various electronic data files that we obtained from the entities listed in Table 6. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of computer-processed information that we use to support our findings, conclusions, or recommendations. Table 6 describes the analyses we conducted using data from these systems, our methodology for testing them, and the limitations we identified in the data. Although we recognize that these limitations may affect the precision of the numbers we present, in total there is sufficient evidence to support our audit findings, conclusions, and recommendations.


Table 6
Methods Used to Assess Data Reliability
INFORMATION SYSTEM PURPOSE METHOD AND RESULT CONCLUSION
ICOE's Orion software To determine the usage rates for the K12HSN program's circuits monitored by Orion as part of audit procedures examining ICOE's network upgrade decisions. We performed reliability testing by comparing usage outputs from the Orion Software to an independent monitoring system used by CENIC known as Cricket. We compared equivalent timeframes for the same parts of the network and found that the two systems were consistent with one another. We found the Orion data to be sufficiently reliable for these purposes.
ICOE's Imperial County Schools Information System (ICSIS) To identify and calculate major sources of revenue and expenditures for the K12HSN program for fiscal years 2013–14 through 2015–16. We reconciled major categories of revenue and expenditures from ICSIS with ICOE’s external audited financial statements for fiscal years 2014–15 and 2015–16, which include a separate opinion on compliance requirements for the K12HSN program.

The fiscal year 2013–14 ICOE audited financial statements do not include specific information for K12HSN. As a result, we interviewed the external auditor to determine the extent and detail of audit testing performed for K12HSN in that year, as well as the subsequent two fiscal years, and determined that level of testing supports the detail with which we present the ICSIS information in our report.		 We found the ICSIS data to be sufficiently reliable for these purposes.
  To determine the value of Internet subsidies the K12HSN program accrued in fiscal year 2015–16 in order to project future program revenue. Fiscal year 2015–16 was the first year ICSIS tracked the accrued E-Rate and Teleconnect Fund subsidies specifically for the K12HSN program. K12HSN’s audited financial statements for that fiscal year indicate that the subsidy amounts were unaudited. Because the K12HSN program has not yet fully received the reimbursements associated with the accrued subsidy amounts, we cannot determine the preciseness of those amounts. Undetermined reliability.

Although this determination may affect the precision of the numbers we present, there is sufficient evidence to support our audit findings, conclusions, and recommendations.

Sources: California State Auditor’s analysis of various documents, interviews, and data from ICOE.


We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Respectfully submitted,

ELAINE M. HOWLE, CPA
State Auditor


Date:
May 25, 2017

Staff:
Linus A. Li, CPA, CMA, Audit Principal
Mark Reinardy, MPP
Ryan T. Canady
Sean D. McCobb, MBA
Bridget Peri, MBA
Karen Wells

Legal Counsel:
Stephanie Ramirez-Ridgeway, Assistant Chief Counsel

For questions regarding the contents of this report, please contact
Margarita Fernández, Chief of Public Affairs, at 916.445.0255.



Footnote

1 Doubling, tripling, or quadrupling a circuit’s capacity may in some circumstances be more expensive than increasing capacity tenfold, as when dark fiber upfront costs are low. Additionally, according to K12HSN’s chief executive officer, depending on the equipment already in use at a network site, the site may also require new equipment for these incremental increases. Go back to text



Back to top