Customer issues nationwide - Issue resolved
We are aware of an issue causing instability for a number of our customers nationwide.
Our senior engineering team are currently investigating and an update will follow shortly.
Apologies for the inconvenience.
Please note this issue is also affecting our phone lines.
Regards
Metronet Support
Following our investigations, please see below for the RFO,
Timeline:
12/02/2015 12:18 – Metronet (UK) NOC alerted to packet loss and routing changes on the Manchester core network. This coincided with a large increase in traffic being received at the edge of the Metronet (UK) network. This was immediately escalated to the core network team.
12:20 – Metronet (UK) invoke emergency support procedures and assign engineers to investigating the issue and placing an entry on the status page.
12:22 – The core network team identifies the cause of the issue as a DDoS and proceeds to identify the sources and destination.
12:25 – Status page submitted at http://status.metronet-uk.com/issue/191
12:28 – Incoming traffic levels decrease and routing instability clears. Investigations continue to identify the sources and destination of the DDoS.
12:35 – Multiple sources are identified as transmitting large amounts of traffic from UDP port 123 (NTP) to the Metronet (UK) network. This confirmed the DDoS as being an NTP reflection attack. Further investigations continue into the sources of the attack.
12:50 – All services remain stable and an update is added to the status page with this information.
Additional Details:
Over 500 sources were identified as each transmitting 100MB of data resulting in a total of 100GB of data being received and routed across the Metronet (UK) network. This caused some routing instability as our core made routing calculations and changes to re-route traffic to lower utilised paths.
The core network team continued to monitor all edge links for irregular traffic patterns throughout the day and worked to implement controls to mitigate against any further NTP reflection attacks. This configuration was approved and implemented on 13/02/2015.
We apologies for any inconvenience caused by this incident.
Regards,
Metronet (UK) Support
The network has remained stable since 12:30. We will continue investigations and issue a full RFO within 5 working days. This incident will be marked as resolved however will have further details posted when available.
Regards
Metronet Support
The network is currently stable and has been since 12:35. All customer services should be back online - if you are still experiencing service problems, please contact our support team.
Investigations into the cause are continuing.
Regards
Metronet Support