*Cloud trouble.
*Okay, we've got the metaphor of the "Cloud," and we're kind of stuck with that word.
"Backbone, Highway, Web, Cloud." History of the Internet in four words.
*From a bony skeleton to an oozing, omnipresent,
wireless fog.
*But, one of the larval clouds, the Microsoft T-Mobile Danger Sidekick cloud,
just failed. We now need a verb to go with that noun. The Cloud "crashed"?
How can a cloud "crash?" The cloud "failed"? Did you ever see a cloud fail?
A cloud "service disruption"? Would the term "drought" do for that?
*Clouds blow away with the winds, evaporate with a hot sun, they thunderstorm,
they spew tornadoes, they dissipate... "the dark side of cloud computing
has no silver linings..." Well, this isn't the last we're going to hear from
clouds and their many new woes, so here comes more than you'd like to know
about it, from RISKS DIGEST and SANS – hardened cynics who've got
it together with when it comes to verbal fog.
RISKS-LIST: Risks-Forum Digest Monday 12 October 2009 Volume 25 : Issue 81
ACM FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS (comp.risks)
Peter G. Neumann, moderator, chmn ACM Committee on Computers and Public Policy
***** See last item for further information, disclaimers, caveats, etc. *****
This issue is archived at as
The current issue can be found at
Contents:
Microsoft's Danger Data Service disrupts users (John F. McMullen)
Microsoft's Danger SideKick and cloud computing (Daniel Eran Dilger via
Monty Solomon)
Microsoft's Sidekick due to dogfooding/sabotage (Daniel Eran Dilger via
Monty Solomon)
Cloud Danger, literally... M$ loses T-mobile data (David Lesher)
Excess CAT scan radiation – the return of Therac 25? (David Lesher)
A Time Machine time bomb (Ron Garret)
Why E-mail No Longer Rules (Jessica E. Vascellaro via Monty Solomon)
Re: Airline status display follies (Peter R Cook, Arthur Flatau)
Re: The risks of being cute (Rob Seaman, Ken Knowlton)
Re: The computers did it – differently (Wendell Cochran)
Re: Software never fails, people decide that it does (Martyn Thomas,
Michael Smith, Geoffrey Brent, Dimitri Maziuk)
Abridged info on RISKS (comp.risks)
———————————————————————-
Date: Mon, 12 Oct 2009 18:54:31 -0400
From: "John F. McMullen"
Subject: Microsoft's Danger Data Service disrupts users
[From Johnmac's blog: ]
T-Mobile's Sidekick Smart Phone Service, powered by Microsoft's Danger
Data Service has been out of commission for over a week and now the users
are warned that their data, stored on Danger's Servers, may have been lost
and that the data that remains on their Sidekick devices is at jeopardy,
putting customers contact and calendar information at risk to disappear.
Some johnmac comments:
1. There was never a problem like this prior to the Microsoft acquisition
of Danger.
2. There has been little media coverage of this problem although I suspect
that multi-thousands of users are affected.
3. It would seem that, given all of its technical expertise, Microsoft could
come up with some way to replicate the original Danger SideKick to Danger
backup. Failing that, it should be able to provide a USB backup to
Outlook.
4. Perhaps Google can jump in with a Sidekick to G-Mail, G-Calendar, etc. If
so, game over and a lot of Androids get sold.*
- - - - - - - - - - - -
The latest missive:
Sidekick customers, during this service disruption, please DO NOT remove
your battery, reset your Sidekick, or allow it to lose power.
Updated: 10/10/2009 12:35 PM PDT
T-MOBILE AND MICROSOFT/DANGER STATUS UPDATE ON SIDEKICK DATA DISRUPTION
Dear valued T-Mobile Sidekick customers:
T-Mobile and the Sidekick data services provider, Danger, a subsidiary of
Microsoft, are reaching out to express our apologies regarding the recent
Sidekick data service disruption. We appreciate your patience as
Microsoft/Danger continues to work on maintaining platform stability, and
restoring all services for our Sidekick customers.
Regrettably, based on Microsoft/Danger's latest recovery assessment of their
systems, we must now inform you that personal information stored on your
device - such as contacts, calendar entries, to-do lists or photos - that is
no longer on your Sidekick almost certainly has been lost as a result of a
server failure at Microsoft/Danger. That said, our teams continue to work
around-the-clock in hopes of discovering some way to recover this
information. However, the likelihood of a successful outcome is extremely
low. As such, we wanted to share this news with you and offer some tips and
suggestions to help you rebuild your personal content. You can find these
tips in our Sidekick Contacts FAQ. We encourage you to visit the Forums on a
regular basis to access the latest updates as well as FAQs regarding this
service disruption.
In addition, we plan to communicate with you on Monday (Oct. 12) the status
of the remaining issues caused by the service disruption, including the data
recovery efforts and the Download Catalog restoration which we are
continuing to resolve. We also will communicate any additional tips or
suggestions that may help in restoring your content.
We recognize the magnitude of this inconvenience. Our primary efforts have
been focused on restoring our customers' personal content. We also are
considering additional measures for those of you who have lost your content
to help reinforce how valuable you are as a T-Mobile customer. We continue
to advise customers to NOT reset their device by removing the battery or
letting their battery drain completely, as any personal content that
currently resides on your device will be lost. Once again, T-Mobile and
Microsoft/Danger regret any and all inconvenience this matter has caused.
Service Disruption FAQs| Disruption Credit FAQs| Disruption Discussion
Password/Sign-in Text Message FAQs | Password/Sign-in Discussion
[One of my closest associates reacted to this, and said,
"Who would want to use a system called `Danger'?" PGN]
[email protected] [email protected] [email protected]
johnmac@panix. com, [email protected] [email protected]
[email protected] [email protected] [...]
[Johnmac's message also included an incisive item by Robert X. Cringeley,
Microsoft screwup puts T-Mobile users in Danger. PGN]
http://www.infoworld.com/d/adventures-in-it/microsoft-screwup-puts-t-mobile-users-in-danger-482?source=3DIFWNLE_nlt_blogs_2009-10-12
——————————
Date: Mon, 12 Oct 2009 19:22:50 -0400
From: Monty Solomon
Subject: Microsoft's Danger SideKick and cloud computing (Daniel Eran Dilger)
Daniel Eran Dilger, Microsoft's Danger SideKick data loss casts dark on
cloud computing, 11 Oct 2009
http://www.roughlydrafted.com/2009/10/11/microsofts-danger-sidekick-data-loss-casts-dark-on-cloud-computing/
Microsoft has demonstrated that the dark side of cloud computing has no
silver linings. After a major server outage occurred on its watch last
weekend, users dependent on the company have just been informed that their
personal data and photos "has almost certainly been lost."
Microsoft's Danger SideKick data loss casts dark on cloud computing
While occasional service outages have hit nearly everyone in the business,
knocking Google's Gmail offline for hours, plunging RIM's BlackBerrys into
the dark, or leaving Apple's MobileMe web apps unreachable to waves of
users, Microsoft's high profile outage has impacted users in the worst
possible way: the company has unrecoverable lost nearly all of its users'
data, and now has no alternative backup plan for recovering any of it a week
later.
The outage and data loss affects all SideKick customers of the Danger group
Microsoft purchased in early 2008. Danger maintained a significant online
services business for T-Mobile's SideKick users. All of T-Mobile's SideKick
phone users rely on Danger's online service to supply applications such as
contacts, calendars, IM and SMS, media player, and other features of the
device, and to store the data associated with those applications.
When Microsoft's Danger servers began to fall offline last Friday October 2,
users across the country couldn't even use the services; even after
functionality was beginning to be brought back on Tuesday October 6, users
still didn't have their data back. This Saturday, after a week of efforts to
solve the crisis, T-Mobile finally announced to its SideKick subscribers:
"Regrettably, based on Microsoft/Danger's latest recovery assessment of
their systems, we must now inform you that personal information stored on
your device - such as contacts, calendar entries, to-do lists or photos -
that is no longer on your Sidekick almost certainly has been lost as a
result of a server failure at Microsoft/Danger."
A new report from Engadget says that T-Mobile has suspended sales of its
SideKick models and is warning: "Sidekick customers, during this service
disruption, please DO NOT remove your battery, reset your Sidekick, or allow
it to lose power." ...
[Also noted by Ben Moore. PGN]
——————————
Date: Mon, 12 Oct 2009 23:01:13 -0400
From: Monty Solomon
Subject: Microsoft's Sidekick due to dogfooding/sabotage (Daniel Eran Dilger)
Daniel Eran Dilger, Microsoft's Sidekick/Pink problems blamed on dogfooding
and sabotage, 12 Oct 2009
Additional insiders have stepped forward to shed more light into Microsoft's
troubled acquisition of Danger, its beleaguered Pink Project, and what has
become one of the most high profile Information Technology disasters in
recent memory.
The sources point to longstanding management issues, a culture of
"dogfooding," and evidence that could suggest the issue was a deliberate act
of sabotage.
AppleInsider previously broke the story that Microsoft's Roz Ho launched an
exploratory group to determine how the company could best reach the consumer
smartphone market, identified Danger as a viable acquisition target, and
then made a series of catastrophic mistakes that resulted in both the
scuttling of any chance that Pink prototypes would ever appear, as well as
allowing Danger's existing datacenter to fail spectacularly, resulting in
lost data across the board for T-Mobile's Sidekick users. ...
http://www.roughlydrafted.com/2009/10/12/microsofts-sidekickpink-problems-blamed-on-dogfooding-and-sabotage/
——————————
Date: Sun, 11 Oct 2009 15:43:17 -0400 (EDT)
From: "David Lesher"
Subject: Cloud Danger, literally... M$ loses T-mobile data
T-Mobile's "Sidekick" mobile service uses a backend system provided by
Microsoft, and seemingly aptly named "Danger." [Will Robinson was not
mentioned, but...]
Danger has lost ALL the customers' stored data. The only copy remaining
is that remaining on the mobile device itself.
"our teams continue to work around-the-clock in hopes of discovering some
way to recover this information. However, the likelihood of a successful
outcome is extremely low."
RISKS:
Backups are good, working backups *far* better.
If you run a cloud-based service, you can ruin *many* more people's days
than anyone with a mere departmental failed server ever can.
——————————
((((From SANS:)))
TOP OF THE NEWS
–Sidekick Outage Causes Data Loss and Outrage
(October 10 & 12, 2009)
A server failure appears to be responsible for a massive data loss
affecting T-Mobile Sidekick customers. The outage occurred at Danger,
a Microsoft subsidiary, which is the Sidekick data service provider.
Users lost contacts, pictures, and saved email messages. While it is
possible that some data could be restored from a backup system, most is
likely gone forever. T-Mobile has suspended sales of Sidekicks for the
time being. The company is offering customers a one-month credit to
their accounts to compensate for the data loss. The data loss affects
customers who conducted a hard reset - removing their phones' batteries
or pressing a reset button. The customers attempted the hard reset
because of outages affecting the devices all last week.
http://www.msnbc.msn.com/id/33278150/ns/technology_and_science-security/
http://www.computerworld.com/s/article/9139261/T_Mobile_sidelines_Sidekick_in_wake_of_data_debacle?taxonomyId=1
http://www.usatoday.com/tech/wireless/phones/2009-10-12-sidekick-data_N.htm
http://www.informationweek.com/news/personal_tech/smartphones/showArticle.jhtml?articleID=220600351
http://voices.washingtonpost.com/fasterforward/2009/10/sidekick_users_see_their_data.html
http://www.washingtonpost.com/wp-dyn/content/article/2009/10/11/AR2009101100109_pf.html
http://www.cnn.com/video/#/video/tech/2009/10/12/tsr.tmobile.loses.data.cnn
Editor's Note (Ullrich): So much for storing your data "in the cloud".
A local backup sounds like a great idea again.
(Pescatore): Ah, the monthly reminder that consumer grade services do
not live up to business class needs....
(((This just in: the Danger cloud managed to claw some missing data back.
If "clouds" have "claws," that is.)))
*Via RISKS DIGEST:
——————————
Date: Thu, 15 Oct 2009 12:16:00 -0400
From: "John F. McMullen"
Subject: Re: Microsoft's Danger Data Service (Re: RISKS-25.81)
John Murrell
Sidekick depression eases; Microsoft says recovery under way
The prospects for recovering the personal data lost by T-Mobile Sidekick
customers in a server snafu at Microsoft's Danger unit have gone from bleak
to hazy to substantially brighter.
In a post early today, Roz Ho, Microsoft's VP for (ideally) Premium Mobile
Experiences, said "We are pleased to report that we have recovered most, if
not all, customer data for those Sidekick customers whose data was affected
by the recent outage. We plan to begin restoring users' personal data as
soon as possible, starting with personal contacts, after we have validated
the data and our restoration plan. We will then continue to work around the
clock to restore data to all affected users, including calendar, notes,
tasks, photographs and high scores, as quickly as possible. We now believe
that data loss affected a minority of Sidekick users."
She went on: "We have determined that the outage was caused by a system
failure that created data loss in the core database and the back-up. We
rebuilt the system component by component, recovering data along the way.
This careful process has taken a significant amount of time, but was
necessary to preserve the integrity of the data. ... We have made changes to
improve the overall stability of the Sidekick service and initiated a more
resilient backup process to ensure that the integrity of our database
backups is maintained."
http://click1.newsletters.siliconvalley.com/wsqfqmtdr_ohmctgnpjnp_myfvsqln.html
That said, Microsoft continued to run away from Danger lest its other cloud
computing efforts be injured. Microsoft spokeswoman Tonya Klause said
Wednesday, "The Danger Service platform, which experienced the outage, is a
standalone service operating on non-Microsoft technologies, and is not
related to Microsoft's cloud services platform or Windows Live. Other and
future Microsoft mobile products and services are entirely based on
Microsoft technologies and Microsoft's cloud service platform and software."
The good news on the recovery front arrived too late to stop the first wave
of the inevitable lawsuits including a pair in Northern California that seek
class action status and assert negligence and false claims by Microsoft and
T-Mobile.
[Source: MediaNews Group, 1560 Broadway, Ste. 2100, Denver, CO 80202]
