Intermittent Connectivity - Notice
We are currently investigating an issue resulting in intermittent connectivity to some Internet destinations. Further updates will be posted shortly.
Regards
Metronet Support
Further to the previous updates, please find below details of the RFO:
Incident Start: 18/02/2015 14:40
Incident End: 18/02/2015 14:47
Reason for Outage: Transit routing re-convergences
Associated Status Entry: http://status.metronet-uk.com/issue/194
Timeline:
18/02/2015 14:40 – Metronet (UK) NOC alerted to a drop in the BGP session to one of Metronet (UK) transit providers. This is followed by packet loss alarms. This was immediately escalated to the core network team.
14:42 – Metronet (UK) invoke emergency support procedures and assign engineers to investigating the issue and placing an entry on the status page.
14:44 – The core network team identify that the BGP session to the transit provider had disconnected and then re-connected again. The team monitor the session directly from the connecting boarder router.
14:46 – Status page submitted at http://status.metronet-uk.com/issue/194
14:47 – The BGP transit session drops again resulting in further routing re-convergences. The core network team take the decision to administratively shutdown the session to avoid further instability.
Additional Details:
Metronet (UK) connect to eight upstreams for internet connectivity. On the event of a session going down the routes from this session are removed. Our border routers then re-calculate their routing table based on the removal of these routes. When the session reconnects, these calculations take place again to account for the introduction of new routes. As the session to this transit provider disconnected, connected and then disconnect again, there were three routing convergence events within a short period of time and as such customers may have experienced packet loss or intermittent connectivity to certain destinations as certain routes changed based on these events. The administrative shutting down of the BGP session stopped further re-calculations and so stabilised the issue.
We have engaged with the transit provider support team to ascertain the reason for the session disconnections and we will be re-enabling under a controlled maintenance window following the conclusion of this investigation. Details of this will be available on the status pages.
We apologies for any inconvenience caused by this incident.
Regards,
Metronet (UK) Support
No further issues have been experienced while the affected transit provider has remained shutdown from our transit edge. As all services are stable the incident will be marked as "Informational". An RFO will be issued shortly.
Regards
Metronet Support
Investigations indicates that an issue with one of our three transit providers caused intermittent connectivity to parts of the internet. During this period customers may have experienced connectivity issues on VoIP systems, across VPNs, some web pages and to other systems which are hosted outside of the Metronet (UK) network that route across this transit provider.
Due to the routing instability, we took the action to shut this transit session down and remove this from our global routing mix. All internet traffic which was routing over this transit provider has now re-balanced across our other two transit providers and 4 peering LANs.
We will now be liaising with the transit provider to investigate the cause of this incident and will only re-enable the session to them under a controlled maintenance window following the conclusion of the investigation.
All connectivity is now stable and no customers should be experiencing any issues. Any customers experiencing connectivity issues should contact Metronet (UK) Support on 01618222580 or by emailing ticket@metronet-uk.com.
No further issues are expected. We apologies for any inconvenience caused.
Regards
Metronet Support