PKI Router under investigation

Incident Report for Network Nebraska

Postmortem

On April 17th just after 5:00pm, the PKI WAN router lost power due to a power fault with the primary UPS. The outage was caused by a failure in a power module on the router, which caused the UPS to go into short circuit protection. When the router came back online, the configuration didn’t load correctly and had to be failed over to a second management card to bring up some of the routes. After working with Juniper TAC, the team did a full reboot of the router, and all networks came back online. The UPS and faulty power module were replaced April 20th, bringing the system back to a fully redundant state.

Posted Apr 30, 2024 - 11:03 CDT

Resolved

All remaining issues have been rectified and all services are restored.
Questions 402-472-7656 Toll Free 888-638-6327 (1-888-NET-NEBR) or noc@nebraska.edu

Posted Apr 20, 2024 - 09:29 CDT

Update

The emergency maintenance to the PKI Core Router has been completed. Engineers are continuing to work with vendor support on issues that are non-service impacting.

Posted Apr 17, 2024 - 23:01 CDT

Update

Network Nebraska Engineer will be performing emergency maintenance to the PKI Core Router. All traffic traversing Omaha will be impacted momentarily.

Posted Apr 17, 2024 - 22:18 CDT

Update

Vendor support is currently being engaged to further troubleshoot.

Posted Apr 17, 2024 - 19:00 CDT

Monitoring

A fix has been implemented, and we are monitoring the results.

Posted Apr 17, 2024 - 18:48 CDT

Update

Power restored at PKI, internet peering in Omaha impacted. Restoration efforts ongoing.

Posted Apr 17, 2024 - 18:47 CDT

Update

We are continuing to work on a fix for this issue.

Posted Apr 17, 2024 - 18:02 CDT

Identified

A UPS handling power for equipment at the PKI location has failed. Troubleshooting efforts are underway.

Posted Apr 17, 2024 - 17:51 CDT

Update

We have identified the issue to be related to a power outage on site, and we are still investigating.

Posted Apr 17, 2024 - 17:45 CDT

Update

We are continuing to investigate this issue.

Posted Apr 17, 2024 - 17:31 CDT

Investigating

Potential problem with power at PKI is under investigation. Engineers are investigating the issue.
Reported by Network Nebraska
Submitted by Network Nebraska
Questions 402-472-7656 Toll Free 888-638-6327 (1-888-NET-NEBR) or noc@nebraska.edu

Posted Apr 17, 2024 - 17:17 CDT

This incident affected: Educational Service Units (ESU 1, ESU 2, ESU 3, ESU 7, ESU 8, ESU 9, ESU 15, ESU 17, ESU 19), State Colleges (Chadron State College), Internet / Peering (CenturyLink, Hurricane Electric, Omaha Internet Exchange (OmahaIX), Google), Public Libraries (Fairmont Public Library, Omaha Public Library System), Private Non Profit Colleges / Universities (College of St. Mary, Doane University, Creighton University), Core Services (Router NN-OMA-PKI-MX960), and Community and Tribal Colleges (Little Priest Tribal College, Nebraska Indian Community College).