On Thursday, January 13th 2018, we experienced difficulties when upgrading our network controllers. These difficulties resulted in interruptions of external network connectivity beyond our maintenance window. This was due to an incidental extremely slow upgrade process which we are unable to reproduce and a complete redeployment of the network stack which took longer than we anticipated. We found a bug in a networking subcomponent and applied a fix which resolved the networking issues. Procedures have been updated to prevent this from happening in the future.
After the upgrade was completed we encountered an unforeseen bug in one of our message queues. Internal service communication was unstable causing partial API fallout. Most noticeably DHCP services in virtual routers started to malfunction which affected the instances' network configuration. No user data was lost but the incident resulted in service degradation for most of our users.
In order to prevent delays to maintenance windows in the future, we will upgrade platform critical services separately preventing service upgrades interfering with one another.