On Thursday, December 9th, ITD noticed intermittent problems with authentication of our wireless users. These problems affected all wireless STAGEnet users and guests, effectively stopping wireless traffic for those people. While this was an issue, it was not insurmountable. We were able to logically move traffic from the affected controller to one of the other two operational controllers. Shortly thereafter, one of the two remaining controllers failed.
Our wireless system is designed to survive the loss of any one of the three controllers; at this point, we have two failing controllers.
ITD has made the necessary adjustments to keep wireless service available to as many customers as possible. However, not everyone can be provided with wireless connectivity while we fix the two failing controllers.
In the process of troubleshooting:
- A priority-one, critical-status incident has been opened with our vendor.
- The vendor has escalated the issue to their engineering department.
- All of the controllers and access points were upgraded to the latest version of code available.
- When the issue persisted, all of the code versions were rolled back to the last known working version.
- Soft-resets and power-cycles have been done on the controllers.
- Access-lists have been implemented to limit traffic passing through the controllers.
None of these efforts have been successful. We have been providing all possible information to the vendor, and they are attempting to find a solution. We are in contact with them multiple times per day. In short, while the vendor is trying to resolve the errors, we are also trying all avenues internally.
Thank you for your patience as we work through this. ITD will let you know as soon as it is resolved. If you have any questions, please contact the Service Desk.