Folks, I'm not going to bag on Amazon.com too much for their Elastic Compute Cloud (EC2) failure; I'm sure they're getting enough flack from their customers. However, this is why I dislike any absolute statements when we're dealing with this type of technology. Technically, they're right. You don't have to worry about the cloud. You do have to worry about your cloud.
The cloud may have a backup plan for you. Do you have a backup plan for your cloud?
Following California's introduction of an online privacy bill, Senators Kerry and McCain have introduced one at the federal level. Formal name: Commercial Privacy Bill of Rights Act of 2011. At first glance, the federal bill seems watered-down compared to the California version. This isn't necessarily surprising, since the same can be said about dueling spam legislation between California and the Feds.
However, the federal bill is lacking a prime component of the proposed California bill; a "Do Not Track" provision. Also, the federal bill would vest a lot of the rulemaking power in the FTC.
I did note that some of Silicon Valley's heaviest hitters came out in favor of the federal bill, but seemed more luke-warm to the California bill. What should we glean from that?
The FCC adopted Net Neutrality rules in December 2010. The House voted to block them in April 2011.
You see why I don't race to analyze these items?
Risk. What is it, exactly? Here are a few good definitions:
"The possibility of suffering harm or loss; danger."
"Product of the impact of the severity (consequence) and impact of the likelihood (probability) of a hazardous event or phenomenon."
"The quantifiable likelihood of loss or less-than-expected returns."
"The amount of statistical gamble that someone (usually management) is willing to take against a loss. That loss can for example be either in profits, reputation, market share, or franchise."
Ok…so, knowing that all of this is on the line, why do I keep hearing words like, "Underestimated", "Miscalculated", "Unexpected" and "Unanticipated" every time something goes wrong?
I'm unimpressed.
The latest is the crack (or should I say, sun roof) that appeared in the fuselage of a Boeing 737. As either lawyers or techies, our miscalculations are bad enough, but when these people miscalculate (as in our previous study of Japan's nuclear mishap), other people die!
As a human being, I have to at least give Boeing credit for stepping up to the plate and publicly acknowledging their mistakes. As an attorney? Well…that's another matter, entirely.
A company I worked with many years ago hadn't implemented any disaster-planning. When their catastrophic event occurred (prior to my arrival), they were essentially out of business for close to three weeks before the systems could be rebuilt. In another incident, one of my direct reports was fooling around with an Exchange server and ended up accidentally deleting one of the accounts. Too bad it happened to belong to the CEO…
Yep, disregarding risk may result in a death…but in our disciplines, it's more likely to be one of us!
"For every eDiscovery action, there is an equal and opposite reaction."
— Sir Isaac Newton
As I've mentioned before, my litigation hold letter – that you see over on the right sidebar – is still the most popular link on the blog (next to the actual posts, that is…). I've also provided this template to attendees of my presentations. A question comes up regularly:
"If we send out a letter like that, our adversary will simply replace their name with ours and send it right back to us. We don't want that to happen!"
It's a good point. And as I've also mentioned before, what's good for you may also be good for your adversary. Furthermore, there's absolutely no fault in thinking about this strategically, for example, keeping your clients' advantages/disadvantages in mind when you craft your demands and responses (which, hopefully, you're doing anyway).
There are times when you want everything but the kitchen sink, but sometimes, the sink itself will do nicely. After all, if both sides produce a gargantuan amount of product, somebody's going to have to review it – and pay for it.
Be careful what you wish for in litigation; you might get it – and get nothing.
An adversary may produce reams of product, the sole purpose of which is to either make it next to impossible to find relevant needles in the document haystack, or worse, obfuscate the fact that they didn't produce relevant documents at all. Oracle's Larry Ellison knows a bit about this [In re Oracle Corp. Sec. Litig., 627 F.3d 376 (9th Cir. 2010)].
That's why my template is meant only as a guide. It may be appropriate to issue a letter that simply says, "You're on notice and we expect you to preserve relevant data." And here's the other elephant in the room to consider; are you absolutely certain that your client is completely forthcoming about their own data? If not, we already know who'll be on the hook for it.
As for attorneys who are complicit in assisting their clients with 'hiding the football'? As they said in the movie, Airplane, "…they bought their tickets, they knew what they were getting into. I say, let 'em crash!"
Huh? Wrong Newton? Don't look at me. Go back and redo your keyword & concept searches!
We started with a premise: A disaster has occurred. What now?
We segued into a limited examination: Were we properly prepared? Why or why not?
Now, comes the all-too-familiar Watergate-esque finale: What did we know; and when did we know it?
According to this comprehensive report, officials were warned that there was a 10% risk within a 50-year span of a tsunami swamping the protective barriers of the Fukushima Dai-ichi nuclear power plant – and disregarded it. What result?
I could go on, but you get the idea. Now for the bad news. That's not the worst of it. How about:
You know what? I have to stop now. This feels ghoulish.
The point I'm making is, certainly, this is about as bad as a disaster gets, but we can all learn from it because there's only one item we need to change – scale. Plus, the most important thing relevant to us in the real-life case study we're now seeing is what happens when we're wrong.
Worried eDiscovery clients always ask me how they're ever going to do everything right. I tell them, there is no such thing. It's impossible to anticipate everything, but as a rule of thumb, the fallback position is the basic negligence standard:
Knew, or should have known.
If they acted in good faith based on what they knew or should have reasonably anticipated at a given point in time – and present a defensible position as to why they acted – they'll likely preserve safe harbor. Naturally, one can never completely account for the odd rogue judge. The day all judges rule alike is the day I give a specific answer. In the meantime, you do the best you can.
The key is in making sure you have the appropriate harbor pilot.
It's been roughly two weeks since the devastating events in Japan. As I mentioned in my initial post regarding their disaster-recovery efforts, we weren't going to know all of the elements we needed to know at that time in order to make an assessment – and we don't know them now. On the other hand, we know enough to put them under a magnifying glass. If you're part of a disaster-preparedness team, a cursory examination of their nuclear mess is a true 'teachable moment'.
Why do I keep harping on this? Because litigation may take on all of the elements of a disaster-recovery operation in that out of nowhere, you're tasked with finding, restoring and producing massive quantities of information – possibly from several sources and/or geographic locations. And, somebody has to pay for it (Zubulake, Toshiba, et al). Oh, and tic-toc – the clock is ticking…
Let me preface this by saying that armchair quarterbacking is easy – and this is not a 'bash Japan' post. You don't kick someone when they're down (but you do try to learn from their mistakes). Nor is it an "I told you so" post – at least, not by me. Let's be honest, for a moment. Sometimes, when a person says "I told you so", they really did tell you so. So what? The issue isn't what they told you, the issues are:
[Add your own questions here]
What are questions nine and eleven about? You should always be prepared to justify and/or defend your position. After all, you may have to persuade your bosses today, but you never know who you might have to persuade tomorrow (I'm thinking…a judge? A jury?)
Last night I read this article from the Washington Post (and others over the past few days) regarding how the Japanese authorities considered risk when assessing how to protect their nuclear plants. In my opinion, if you commit to the short amount of time necessary to read the entire story, you'll learn more about disaster-preparedness than you ever could in a classroom; unless, of course, they're studying this disaster.
In an island nation, surrounded by volcanic activity, "experts" didn't even consider a major tsunami as part of the plan for the Fukushima Daiichi power plant because it was considered "unlikely". But, here's an even better question, raised at the conclusion of the story:
"To what degree must regulators design expensive safeguards against once-a-millennium disasters, particularly as researchers learn more about the world’s rarest ancient catastrophes?"
Which leads me to the obvious follow-up:
Two weeks ago, the experts may have thought that the risks were worth it. But now that radiation is showing up in drinking water as far away as Tokyo? My guess is, they wish they'd have built the retaining walls a few feet higher.
"Nobody anticipated…"
What if? Those two words form the initial basis of a disaster-recovery conversation. Like you, I've seen the heartbreaking pictures from Japan and what gets me is, a country that is known for having the best earthquake-disaster-preparedness in the world has suffered tremendous losses in spite of that fact.
The best laid plans…
Japan's nuclear facilities prepared for a monstrous earthquake, but not an 8.9. Is there any way to plan for an 8.9? And if so, at what cost? Obviously, when contrasted with the devastation we've seen – and may yet see – I wouldn't blame you if you said money is no object. But in reality, we're rarely given a blank check. We're required to work within parameters; sometimes very constrictive ones.
Lessons learned: No matter how thoroughly you plan, it's impossible to prepare for absolutely every contingency that may befall you. In the future – when memory of this disaster has faded and the passage of time blunts the impact – when envisioning a worst-case-scenario for your disaster-recovery program, if those around you are prone to cut corners, remember Japan.
…not the e-Discovery Army you might want or wish to have."
– Perry Rumsfeld
I don't think I've seen a story re-blogged, re-tweeted and/or re-everything'ed – at least among those who I monitor – as much as the "Armies of Expensive Lawyers, Replaced by Cheaper Software" article that appeared recently in the New York Times.
First of all, we need to remember what headlines are; short statements designed to entice one to inquire further, also known as 'teases'. Mission accomplished! Nothing like combining the words, "Expensive Lawyers", "Replaced by" and (horrors!) "Cheaper" in a headline to achieve a visceral reaction. And the tagline (ouch!), "Smarter Than You Think"? Really, New York Times? Without the guidance of humans?
Perhaps you should have looked behind the set to see how Watson was able to apply context to Jeopardy questions. Coming from a programming background (and I mean, waaaaaaaay back), I was more fascinated by the fact that it got the 'simple' question wrong. What I mean is, on a binary (0/1, pass/fail or right/wrong, if you prefer) issue, Watson 'thought' Toronto was in the United States – even when the question clearly stated that the correct answer was a city in the United States.
My 2nd-grade teacher, Mrs. Blythe, said, "Always read (or listen to) the entire question (or instruction) and make sure you understand it before proceeding (or buzzing in). Ok, she never mentioned anything about buzzing in. How did she illustrate this lesson? When we walked into class one day, we saw that she had placed a long list of tasks on a chalkboard, all numbered. Each task was below the other. She said these were the tasks we were to perform and to simply begin. Everyone did, except for one person, who remained in their chair. Why? Because the last task on the list said, "#10. Do not perform any of the previous nine tasks.", and only one student read them all as opposed to one-at-a-time (No, I'm not going to tell you if I'm the one who read #10.)
When I first saw the NYT article last week, I was going to jump all over it – then I decided to wait. I was curious how my colleagues would react to it. Some simply repeated it through Twitter or their blogs, others added a few comments one way or the other (of course I haven't seen them all) and – not surprisingly – Ralph Losey hit a home run, in my opinion, with his analysis.
By coincidence, the same day the NYT article appeared, the LA Times wrote, "Retail jobs are disappearing as shoppers adjust to self-service". See any parallels?
The issue, as I see it, has nothing to do with jobs. It has everything to do with efficiency. I'd also note that while document review – which is what the NYT article is really talking about – is a component of e-Discovery, e-Discovery encompasses much more than document review. Think strategy, for example. Watson may be a genius, but 'he's' unable to reason when your opponent is hiding evidence.
Years ago, a headhunter offered me some document review work. She was paying $25/hr. At the time, a checker at my local Albertsons supermarket was earning approximately $30/hr and was about to go on strike to demand more. Prior to that, at the nadir of the dot-com bubble, I'd watched as a plethora of good technology jobs were outsourced to India. This is what free markets are all about.
I'm not a fan of the language of progress as expressed in corporate meetings. Matrix, metrix, service-levels, deliverables…ugh…so robotic in nature. Maybe that's why I migrated to consulting. But I'll tell you one phrase I – and employers – do like. Value add.
The focus seems to be, "The machines can now do what I can do!". To me, the focus should be, "What can I do that the machines can't do? How do I add value, the result of which is, I'm needed in addition to the machines?
When you ask yourselves that question, you'll always be one step ahead of technological advances, rather than chasing them.
I'm probably the last to comment on the Gmail/cloud issue – and you already know my opinion of cloud computing – agnostic.
We find that almost anything in life is great…when it works. When it doesn't?
Where the mistake is usually made is in the assumption that things always work. We pick up the landline, expect the dial-tone to be there and are shocked if it's not.
If I wasn't an eDiscovery dude, I'd probably sell insurance like my grandfather. Maybe I learned something about disaster-planning by observing him.
If you're going to use the cloud, institute a backup plan and stick to it – or make sure your provider is doing so.
Oh, and don't forget to test it regularly, in order to avoid 'Chronic' problems – such as getting into a David-and-Goliath war with a 3rd-party like Google.