The upgrade scheduled for this morning was successful. All equipment was up as of 6:30. Early indicators suggest improvement/resolution of:

• Connection to Internet or U: and S: drives impeded (user gets an “unable to connect to server” message)

Denison’s network will be receiving critical code updates on Friday, Sept 6 between the hours of 5:00AM and 7:00AM.  Expect services to be unavailable, including access to the Internet, during this time. The update is expected to resolve the issue:

• Connection to Internet or U: and S: drives impeded (user gets an “unable to connect to server” message)

As reported by Oarnet, Denison’s Internet Service Provider: 

A number of OARnet customers saw network slow downs and reachability issues starting late in the day on September 2 and extending intermittently into September 4, 2013.

Root Causes

In many ways this was “the Perfect Storm”. There were multiple issues, which compounded the situation because they were occurring simultaneously.  From an OARnet perspective they are:

1.            The 100Gbps link failure in Cleveland – A simple Hardware failure

2.            Amazon reachability issues due to issues between I2 and AWS. This manifested only to users reaching Amazon over the R & E space. For example, KSU saw this.

3.            General slowdowns caused by congestion on TR-CPS – See Question for Community at the end of this document

4.            Intermittent slow downs caused by flapping of AL2S. Some members saw traffic slow down then speed up, this was caused by the oscillating between links as traffic rerouting between links inside of I2.

5.            On top of all of this, some Blackboard users were seeing issues, which as of this writing appear to be an issue at the provider campus and was not related to any of the above.

This is a general notice that a number of technology anomalies causing disruption have been occurring throughout Denison’s network infrastructure (and beyond) since the start of the semester. IT staff, in conjunction with several outside consultants, continue to work around the clock to bring stability and expected performance to all systems and services. Instability and poor performance with a variety of services is not wide spread or consistent across the campus areas but the incoming Trouble Call volume of the Help Desk is steady. Technicians are resolving problems as quickly as possible. Electronic classroom technology issues will be resolved as a first priority, followed by faculty, staff, and student wireless connectivity reported issues.  If you have any problem associated with connecting to the network, wired or wireless, please report it to the ITS Help Desk immediately at 740.587.6395 or helpdesk@denison.edu

Pending and resolved issues:

• Blackboard performance (slow): Believed to be largely resolved; intermittent slowness was experienced over the past week. This was primarily experienced by those on-campus but reports also were taken of slowness when connecting off-campus. There was a problem with our Internet connection as reported by Oarnet, occuring through Wed afternoon, that may have contributed to the problem. The Blackboard platform is hosted by the University of Cincinnati.  It was upgraded to a new version during the summer.
• Internet performance (slow or unavailable): Believe to be resolved; reports were individual (not wide spread) and sporadic; Denison’s ISP Oarnet identified an issue in the Ohio fiber ring and corrected; the holiday delayed the resolution.
• Wireless (cannot connect or seemingly unavailable): Believed to be largely resolved although can re-occur; IT is monitoring affected areas closely. Affected approximately 30% of student residences and two academic buildings and Slayter. Meru access point radios are sporadically going off-line. IT has put in place a work-around to re-connect radios while awaiting a permanent fix from the manufacturer. Radios may continue to experience interruption.

• Connection to Internet or U: and S: drives impeded (user gets an “unable to connect to server” message): Not resolved; IT is actively working on this problem with the core infrastructure vendor support. Diagnosis has been tedious but a fix is on the horizon!  Progress continues.  If reported to the help desk, technicians are able to resolve for individual machines.  Please call if you have trouble.

   

A misconfiguration would have caused wireless connectivity problems in the following buildings this morning:

Update:  As of 00:30 on 9/3, our provider has put in a work-around to address this issue while troubleshooting continues. Service speeds should be normal again.

Original Post:

Denison’s Internet provider, Oarnet, is experiencing difficulties which has resulted in extremely slow connection time for some of our services, including Blackboard.

We will be performing maintenance during the outage window tomorrow morning which may result in a brief outage in wired and wireless connectivity.

There will be a brief Internet outage this morning while we update and reload a network appliance. This work will begin at 10:00 a.m.