This issue is resolved and we will continue to monitor.
We are also investigating the underlying cause further. GCP is currently experiencing a network issue in the same region we are experiencing the issue in: https://status.cloud.google.com/incident/cloud-networking/19002
However a lack of monitoring a particular internal service exacerbated this issue. We'll be adding additional monitoring coverage to address this.
Apologies for any inconvenience.