I've been hearing a lot more pro-cloud chest-thumping lately. One article basically said, "Fears of the cloud are overblown. You need to get off it!"
I don't think the creators of "Binky" would agree.
What I'd like to know is, were there backups? If so, were those deleted, too?
We started with a premise: A disaster has occurred. What now?
We segued into a limited examination: Were we properly prepared? Why or why not?
Now, comes the all-too-familiar Watergate-esque finale: What did we know; and when did we know it?
According to this comprehensive report, officials were warned that there was a 10% risk within a 50-year span of a tsunami swamping the protective barriers of the Fukushima Dai-ichi nuclear power plant – and disregarded it. What result?
I could go on, but you get the idea. Now for the bad news. That's not the worst of it. How about:
You know what? I have to stop now. This feels ghoulish.
The point I'm making is, certainly, this is about as bad as a disaster gets, but we can all learn from it because there's only one item we need to change – scale. Plus, the most important thing relevant to us in the real-life case study we're now seeing is what happens when we're wrong.
Worried eDiscovery clients always ask me how they're ever going to do everything right. I tell them, there is no such thing. It's impossible to anticipate everything, but as a rule of thumb, the fallback position is the basic negligence standard:
Knew, or should have known.
If they acted in good faith based on what they knew or should have reasonably anticipated at a given point in time – and present a defensible position as to why they acted – they'll likely preserve safe harbor. Naturally, one can never completely account for the odd rogue judge. The day all judges rule alike is the day I give a specific answer. In the meantime, you do the best you can.
The key is in making sure you have the appropriate harbor pilot.
It's been roughly two weeks since the devastating events in Japan. As I mentioned in my initial post regarding their disaster-recovery efforts, we weren't going to know all of the elements we needed to know at that time in order to make an assessment – and we don't know them now. On the other hand, we know enough to put them under a magnifying glass. If you're part of a disaster-preparedness team, a cursory examination of their nuclear mess is a true 'teachable moment'.
Why do I keep harping on this? Because litigation may take on all of the elements of a disaster-recovery operation in that out of nowhere, you're tasked with finding, restoring and producing massive quantities of information – possibly from several sources and/or geographic locations. And, somebody has to pay for it (Zubulake, Toshiba, et al). Oh, and tic-toc – the clock is ticking…
Let me preface this by saying that armchair quarterbacking is easy – and this is not a 'bash Japan' post. You don't kick someone when they're down (but you do try to learn from their mistakes). Nor is it an "I told you so" post – at least, not by me. Let's be honest, for a moment. Sometimes, when a person says "I told you so", they really did tell you so. So what? The issue isn't what they told you, the issues are:
[Add your own questions here]
What are questions nine and eleven about? You should always be prepared to justify and/or defend your position. After all, you may have to persuade your bosses today, but you never know who you might have to persuade tomorrow (I'm thinking…a judge? A jury?)
Last night I read this article from the Washington Post (and others over the past few days) regarding how the Japanese authorities considered risk when assessing how to protect their nuclear plants. In my opinion, if you commit to the short amount of time necessary to read the entire story, you'll learn more about disaster-preparedness than you ever could in a classroom; unless, of course, they're studying this disaster.
In an island nation, surrounded by volcanic activity, "experts" didn't even consider a major tsunami as part of the plan for the Fukushima Daiichi power plant because it was considered "unlikely". But, here's an even better question, raised at the conclusion of the story:
"To what degree must regulators design expensive safeguards against once-a-millennium disasters, particularly as researchers learn more about the world’s rarest ancient catastrophes?"
Which leads me to the obvious follow-up:
Two weeks ago, the experts may have thought that the risks were worth it. But now that radiation is showing up in drinking water as far away as Tokyo? My guess is, they wish they'd have built the retaining walls a few feet higher.
"Nobody anticipated…"
So, my PDA was working fine this morning…then it wasn't. A total and complete software crash. I traced it down to an offending app, but it completely hosed (thank you, Canada) the O/S. And of course, I'm out of town! So, I located my carrier's nearby retail outlet and headed on over. They couldn't repair it, even with their "push" software. However, due to my predicament (out of town and desperate), they were able to swap me over to a new device. Awesome!
How did I reward them for their excellent customer service? By hounding the poor tech, first by insisting that he had to wipe my prior device clean and second, by insisting that he show me how he was going to do it, then let me watch. The store was full, they were busy and I caught a person or two rolling their eyes at my request, but I persevered.
Quite frankly, the tech understood when I explained that as an attorney, there was confidential client info on the device and our ethics rules compel us to protect it (that, or he thought I was just making it all up so he would delete porn, or something…hey, as long as I get it done, let him imagine whatever he wants!) But take note; I protected client information – as I'm obligated to do – and, I had a recent backup available on my laptop, so I lost virtually nothing. Good for me; good for my clients!
Sadly, this is the eDiscovery equivalent of voyeurism.
e-Voyeurism? Nahhhh…
What if? Those two words form the initial basis of a disaster-recovery conversation. Like you, I've seen the heartbreaking pictures from Japan and what gets me is, a country that is known for having the best earthquake-disaster-preparedness in the world has suffered tremendous losses in spite of that fact.
The best laid plans…
Japan's nuclear facilities prepared for a monstrous earthquake, but not an 8.9. Is there any way to plan for an 8.9? And if so, at what cost? Obviously, when contrasted with the devastation we've seen – and may yet see – I wouldn't blame you if you said money is no object. But in reality, we're rarely given a blank check. We're required to work within parameters; sometimes very constrictive ones.
Lessons learned: No matter how thoroughly you plan, it's impossible to prepare for absolutely every contingency that may befall you. In the future – when memory of this disaster has faded and the passage of time blunts the impact – when envisioning a worst-case-scenario for your disaster-recovery program, if those around you are prone to cut corners, remember Japan.
I'm probably the last to comment on the Gmail/cloud issue – and you already know my opinion of cloud computing – agnostic.
We find that almost anything in life is great…when it works. When it doesn't?
Where the mistake is usually made is in the assumption that things always work. We pick up the landline, expect the dial-tone to be there and are shocked if it's not.
If I wasn't an eDiscovery dude, I'd probably sell insurance like my grandfather. Maybe I learned something about disaster-planning by observing him.
If you're going to use the cloud, institute a backup plan and stick to it – or make sure your provider is doing so.
Oh, and don't forget to test it regularly, in order to avoid 'Chronic' problems – such as getting into a David-and-Goliath war with a 3rd-party like Google.
I'm pretty sure most of you are already aware that you can be tracked through your PDA (for those who don't, start by disabling GPS functionality or – horrors! – turn the device off). But what other methods are companies using to track you; and while we're at it, how many of them are doing so? The answer might surprise you. The Wall Street Journal did some testing on iPhone and Android devices (Blackberrys weren't included). According to the WSJ, roughly half of the apps sent differing degrees of personal information to 'someone' without the user's consent.
First of all, this is supposed to be a violation of most privacy policies, but in many cases, the policy either doesn't exist or isn't enforced if it does exist. Second, although many of the culprits insist that they only compile general information, that explanation doesn't hold water. PDAs have unique identifiers (aka UDID, PIN, etc.) that cannot be masked. The article likens them to a "supercookie", but they remind me of the days of static IP numbers (for you non-techies, I usually describe an IP number as being similar to a telephone number in which the sequence can be used to pinpoint someone's specific location in the way one would use an area code and a prefix).
Why is this such a big deal? Because I can mask my IP number by placing it behind a firewall (for you non-techies, a firewall is…oh, just look it up…) but I can't do so with the identifier. Once someone has that identifier, it wouldn't take them too long to scour the Internet to retrieve your personal information and build a profile of your specific habits.
Do you see the implications? This goes far beyond advertisers. This is bad enough on a personal-exposure level, but then add the corporate dynamic. Suppose Bob Smith, CEO of a publicly-traded concern, has been identified by a 3rd-party and is being tracked. And suppose Mr. Smith is shown to be visiting the location of a competitor on three occasions. What's Bob doing?
You get the idea. Paranoid? You bet! A true disaster-recovery and/or security specialist deals in the realm of the possible, not just the probable. Possibility is the primary risk; probability is the degree of risk (does that make sense?).
To put it another way, probability assists a client in making an informed decision about which risks they wish to defend against (or which risks may be accounted for in the budget) after all possible risks have been identified and scaled.
How many corporations, do you think, account for these risks? Start by asking how many of them have developed a policy that prevents the keeper of a corporate PDA from installing apps without some sort of controls in place.
"Today, more
organizations have a policy than ever before, but only one-third have tested
their policies and nearly half do not know if their policies have been tested."
~ Kroll Fourth Annual ESI Trends Report
I hadn't even opened my copy of Kroll's new report yet; that little tidbit was in their preamble. It's an excerpt from their section, "A Decade of Discovery". [The report is free, but you're required to register]
What else disturbs me? Only 53% of companies have a litigation hold tool in place. 47% either don't have – or don't know if they have – a litigation hold methodology in place. 62% either haven't – or don't know if they haven't – tested their ESI policies. 62%. Unbelievable!
That's a lot of "I don't knows". All I keep thinking is, did the survey-respondent ask anybody before they answered these questions? If not, they're basically admitting they're part of the problem! Where's the communication!?
The other buzzword you're going to be hearing a lot more of is "ECA", aka early case assessment. A lot of my colleagues have blogged about it. You'll see it visually represented as the "ECA Funnel". The short description is a review of a particular case to determine whether it's worth prosecuting – or defending; usually based on cost analysis and/or drag on resources.
Do you hear that sound? That's the creaky door of the e-Discovery Insights vault opening to two posts from November of 2008 about proper testing. Part I covered identification & preservation. Part II covered collection.
This just goes along with my premise; eDiscovery issues are solved at
the beginning, not the end.
Q.E.D.
Theory is usually easier than practice. You project managers know exactly what I'm talking about. Courses like the Project Management Body of Knowledge (PMBOK) have value, but one item tends to be underestimated; the human element. Projects always look great on paper but unfortunately, they're not executed by robots. They're executed by people with varying talent, ambition, health and – dare I say it – competence levels. Add to that the other human elements; management support or lack thereof, other duties of the team (distractions), unexpected emergencies ("Hey, I need to borrow Steve for a few hours…"), predictive miscalculations and – dare I say it, part II – the competence of the project manager.
With this in mind, it comes as no surprise that Google has missed a deadline to convert the City of Los Angeles email system to the cloud due to security concerns with the L.A.P.D.'s data. Tha-a-a-a-a-t's gonna cost 'em. Worse, they beat out Microsoft for the contract.
Ultimately, the issue will be resolved, but it begs the question – what happens when L.A. requests to retrieve data? Another cautionary tale about 3rd-party vendors…
Canada Day is tomorrow, July 1st. In honor of this annual event, I'd like to highlight two stories of interest:
The 1st item is a tip of the hat to firms who take preservation of critical data seriously. Roebothan McKay and Marshall suffered a catastrophic loss when the practice was destroyed by fire. Yet, it'll be business as usual for the firm. Partner Steve Marshall said that the hard drive containing client data was retrieved and that back-up
data files were also contained off-site.
What if this happened to your firm?
The 2nd item is a link to an excellent article by Clifford F. Schnier (who is also on my blogroll) about the "state of the union" of e-discovery in Canada. Of course, when Cliff learns that I used "Canada" and "state of the union" in the same sentence, I suspect he'll be in touch to give me a history lesson…