(((The cloud was too big to back up onto another cloud, and besides, the cloud was growing so fast that they didn't have the time or money to build another cloud.)))
http://www.theregister.co.uk/2008/08/28/flexiscale_outage/
Link: Engineer accidentally deletes cloud | The Register .
(...)
As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes.
"The problem was caused by human error," Lucas told us.
"We've been having some capacity issues - FlexiScale has been growing at about 30 per cent a month in terms of usage. We've been adding capacity and adding capacity and we were in the process of adding even more, when one of the engineers who was tidying things up on the disk architecture made a mistake."XCalibre immediately took its entire disk structure offline - in order to "preserve the integrity of the data on the system." And with storage down, FlexiScale's processing and networking services are dark as well. But the platform should be returned to normal, once engineers restore the lost data to a brand new disk structure.
"After consulting with our storage vendor it was agreed the most sensible option would be to copy the entire volume to a new disk structure (still maintaining it's integrity and structure), from where we could re-mount it correctly," Lucas' email reads."Unfortunately, due to it's size, we didn't have spare capacity on the platform to create a complete duplicate of it." (...)
