Sandboxes loading issue
Incident Report for Next Tech
Postmortem

Summary

As suspected, this incident was attributable to an abuse of our sandbox cluster by Bitcoin miners who were able to access our sandboxes via an unauthenticated embed by a customer.

We have also identified the systematic consumption of sandbox host resources over the last week which we attribute to the same root cause. Having identified this issue and taken steps to prevent it will likely result in an increase in overall performance.

To be clear, this was not a security breach. Each sandbox is an isolated environment and the virtualization software we use is battle tested and trusted industry-wide. No unauthorized access to code or any other data was made.

Next Steps

Going forward, we will be:

  1. Conducting a thorough review of our resource limit policies to further prevent the ability to abuse sandbox host resources.
  2. Adjusting our alerting policies to better detect CPU spikes like this in the future.
  3. Exploring a number of infrastructure orchestration techniques to further stabilize our infrastructure and offer exciting use cases for our customers.

Additionally starting in January 2019 we will require that all users of sandboxes (including those accessing via 3rd party websites) to be authenticated via at least a GUID. We'd been considering this and in light of today's abuse we feel it is absolutely necessary.

Thank you for your patience as we resolved this issue. If you have any questions please do not hesitate to reach out to support@next.tech.

Posted 11 months ago. Nov 19, 2018 - 14:21 PST

Resolved
The issue has been resolved. Please see the postmortem for additional comments.
Posted 11 months ago. Nov 19, 2018 - 13:31 PST
Monitoring
We have taken steps to discontinue the current abuse and are monitoring the sandbox cluster as it recovers.
Posted 11 months ago. Nov 19, 2018 - 12:43 PST
Identified
We have identified the issue as an abuse of our sandbox environments by Bitcoin miners and are taking steps to address the issue.
Posted 11 months ago. Nov 19, 2018 - 12:35 PST
Investigating
We are investigating an issue causing some sandbox environments to load slowly or not at all.
Posted 11 months ago. Nov 19, 2018 - 12:21 PST
This incident affected: Project API.