As suspected, this incident was attributable to an abuse of our sandbox cluster by Bitcoin miners who were able to access our sandboxes via an unauthenticated embed by a customer.
We have also identified the systematic consumption of sandbox host resources over the last week which we attribute to the same root cause. Having identified this issue and taken steps to prevent it will likely result in an increase in overall performance.
To be clear, this was not a security breach. Each sandbox is an isolated environment and the virtualization software we use is battle tested and trusted industry-wide. No unauthorized access to code or any other data was made.
Going forward, we will be:
Additionally starting in January 2019 we will require that all users of sandboxes (including those accessing via 3rd party websites) to be authenticated via at least a GUID. We'd been considering this and in light of today's abuse we feel it is absolutely necessary.
Thank you for your patience as we resolved this issue. If you have any questions please do not hesitate to reach out to firstname.lastname@example.org.