As reported by Oarnet, Denison’s Internet Service Provider: 

A number of OARnet customers saw network slow downs and reachability issues starting late in the day on September 2 and extending intermittently into September 4, 2013.

Root Causes

In many ways this was “the Perfect Storm”. There were multiple issues, which compounded the situation because they were occurring simultaneously.  From an OARnet perspective they are:

1.            The 100Gbps link failure in Cleveland – A simple Hardware failure

2.            Amazon reachability issues due to issues between I2 and AWS. This manifested only to users reaching Amazon over the R & E space. For example, KSU saw this.

3.            General slowdowns caused by congestion on TR-CPS – See Question for Community at the end of this document

4.            Intermittent slow downs caused by flapping of AL2S. Some members saw traffic slow down then speed up, this was caused by the oscillating between links as traffic rerouting between links inside of I2.

5.            On top of all of this, some Blackboard users were seeing issues, which as of this writing appear to be an issue at the provider campus and was not related to any of the above.