Category Archives: Disaster Recovery

FINRA Fines Firm for Failure

J0442430 I plucked this particular headline out of many as an excellent real-world illustration of the downside of failing to properly retain and archive data.  In this particular instance, Piper Jaffray paid FINRA (the Financial Industry Regulatory Authority Inc.) a $700,000 fine because they were unable to produce an electronic copy of a single email message for an investigation.

Unfortunately, that's when Piper "informed" FINRA about the other 4.3 million emails they failed to retain over a period of six years.  Obviously, I don't have the inside scoop as to why this occurred, but based on experience, I can tell you that $700,000 buys a lot of data protection.

As I've pontificated on many occasions, what if this involved a lawsuit or a tax issue; or a host of other issues?  Does it really matter?  Proper data retention and management may involve a large capital outlay, but pays for itself over and over in the long run.

If you're the person tasked with getting this done, forward this story to your bosses – or present copies at the next meeting.  Eventually, someone will realize that avoiding the issue is simply penny-wise and pound-foolish.

CA Lawyer: IT from Mars, Lawyers from Venus

CA Lawyer May 2010

Folks, I'm going to trial Monday so forgive the sparse posts.  I wanted to point you to this article in CA Lawyer magazine from my colleague, Robert Brownstone.  He and I are both on the CA State Bar's Law Practice Management & Technology executive committee.  We were amused to discover that his article would be published this month, and my next article (on the intersection of e-discovery and privacy) is scheduled for July's issue.

We're also presenting a CLE seminar along with a third colleague (for you non-lawyers, that's "continuing legal education") at the CA State Bar annual meeting in September.

In the same issue is this blurb about e-discovery coming back in-house.  It's funny because I recently observed that more corporations are attempting to handle e-discovery internally to save costs and that lately all of my new clients are law firms.

Redux: Orly gets Sidekicked!

Einstein Time is Money

We're revisiting two former posts today to see how things are working out…

First, we have birther Orly Taitz.  Her frivolous filings have resulted in a $20,000 sanction from judge Clay Land.  She has a lot of supporters who I'm sure will raise the cash for her, so I don't think it'll serve as much of a deterrent.  But maybe this will; the judge has also referred his order to the State Bar of California.

If you want to read a PDF of the judge's order, click here.

Second, we have the Sidekick smartphone and all of the lost user data.  I've been following this story with interest.  Microsoft has issued a recovery tool, but it only applies to contacts, not all of the other data such as photos and notes.  As expected, class-action lawsuits are flying, but many will fizzle out if the recovery tool works.

At this point, there's no solid confirmation that users have recovered data – or what particular data has been recovered – but there's a moral to this story.  It originally hit the news wires around October 10th and the recovery tool was to be available yesterday. 

Moral #1:  Can you afford to be out of commission for two weeks?  When you trust your data to the cloud, make sure it isn't the kind made up mostly of vapor.

Moral #2:  Always, always, back up your own data whenever possible.

Cloud BURST!

J0438348 The lead paragraph from this Washington Post article says it all:

"A server meltdown over the weekend wiped out the master copies of
personal data — including address books, calendars, to-do lists and
photos — accumulated by users of T-Mobile's formerly popular Sidekick smartphone."

It's a little out of our element, but I touched briefly on the risks of cloud computing back in April.  Well, T-Mobile users are getting a front-row seat to what happens when those entrusted with your data don't endeavor to properly protect it.  And this isn't just any old vendor – this is Microsoft subsidiary Danger, Inc.

There isn't really much to add, except that had this happened to one of my clients under my watch, I would have been fired!

I’m Good Enough, I’m Smart Enough & Doggonit, People Like Me…

Cute Sheep…and eventually, I’ll write an in-depth analysis of California AB 5 and contrast it with the Federal Rules.  But, take a look at this fine analysis from Law.com regarding how California will deal with Zubulake ‘accessible vs. inaccessible’ ESI and how it contrasts with the Federal rules.

Finished reading?  Great.  Now let me tell you why nothing in the analysis rattles me.  You should have been treating your ESI as accessible all along.  Here’s why:

The law is all about exceptions.  Everyone knows the general rule, but ultimately the facts dictate whether an exception is in order.  At Sony Pictures in 1997, we successfully fought off a request for ESI in a California court because we made the case that it was accessible, but at punitive cost because we’d long since retired that particular backup system and complying would have required a $250,000 outlay (there was no product like Index Engines around in 1997).

I had a very smart professor in law school.  His advice was to always assume the worst-case scenario, then work backwards.  It’s actually a very logical approach.  Yes, the flood may be that bad, yes, the stock market may fall that much and yes, your adversary may make a persuasive argument to the judge.

Be an actuary.  They look at everything as ‘cost vs. risk’.  Which will likely cost more, paying to protect or paying after the fact?  A young PC user asked me the other day, “Why do I need a firewall?  What are the chances that I will be the one that will be hacked?”  My answer was, “If you want to gamble, that’s your call, but here’s a short list of things that could happen if you’re the unlucky one.”

Gamble if you like; but be prepared to face the consequences.

Observations on Off-Site, On-Site, Outsourcing & Ownership

j0438776Seems to me there are a lot of companies selling data & e-discovery services with the attitude of, “Place your data responsibilities with us, then sleep well at night”.  Hardware and software are offered in-house, SaaS, appliance, off-site…anything you want can be provided.

My personal opinion; before you start relinquishing responsibility to others, keep one thing in mind – it won’t matter.  You’ll ultimately be responsible in the eyes of the law.

I blogged about this before in my ‘Hot Potato‘ post, among others.  The instinctive thing to do – especially with the added complexity of the e-discovery rules hanging over you – is to contract out and make it someone else’s problem.  Heck, I get that.  I’m a Contractor!  Only thing is, in this case…it won’t work.  Good-faith won’t be enough.

I’m not living in a fantasy world.  Some companies have so much data – including ones I’ve consulted with – it would be virtually impossible to manage in-house.  If I said “Don’t do it!” I would expect you to laugh me out of the room (which would be difficult, since none of you know where my ‘room’ is, exactly).  All I’m suggesting is, before you consider outsourcing data management; whether it be on-site, off-site or a combination of both – or even if you’ve already done so – think about all the risks, especially in these difficult economic times.  Do you have a contingency plan in place?

These are the items I’d be including in a checklist (order of preference is up to you):

Hold on a second.  Let’s begin by answering a fundamental question.  Who will manage this?  You?  The Vendor?

Rural Road from a Car

I know.  Some of you are asking what that means?  After all, regardless of how you proceed, somebody representing the company will be responsible for managing this or serving as liaison, right?  Yes and no.

If you don’t know your ‘stuff’, then aside from serving as liaison, you’ve relinquished your ability to make decisions in the best interests of the company.  Essentially, the Vendor will be advising you, and their interests may conflict with yours – especially if litigation arises.  In the alternative scenario, if you’ve educated yourself – or have hired a knowledgeable representative in-house – you’ll be advising them.

Think this is a distinction without a difference?  Take a look at my checklist and see what you think:

  1. Does the Vendor handle backup, restore, disaster-recovery and/or e-discovery services?
  2. Are all of their products integrated?  (Many Vendors acquired other Vendors to stake a presence in the e-discovery field; it doesn’t mean their products integrate well).
  3. What if the Vendor goes bankrupt?
  4. How will the Vendor respond if/when they’re served with a subpoena as a 3rd-party?
  5. Does the Vendor have their own legal representation?
  6. Who will be responsible for managing the retrieval of data?
  7. How quickly can/will the Vendor respond to a request?
  8. Does the Vendor subcontract any services?
  9. Will an additional Vendor be needed for e-discovery if the 1st Vendor doesn’t have that capability?  Do they already have a secondary Vendor in place?
  10. As we expand – including to other countries/continents – how will the Vendor handle it?

I realize this is a ‘macro’ view.  The list above should open up several more questions, such as how are they backing up your backups?

I would think it would be very important to instruct the Vendor about what you expect, rather than rely on the Vendor to tell you what they’re going to do for you.  There’s no room for ambiguity where e-discovery in concerned.

Alliteration always assists attorneys acting as authors…

I Have some Good News & some Bad News…

*** My n/w and phones are out, so I’m coming to you live from the Redondo Beach Public Library, courtesy of their free wireless service…THANK YOU!!! ***

j0178660

Why did the goose cross the road?  Let’s take a gander…

I.T. to the Attorneys and Management:  “Great news!  We can leverage our existing ESI backup and/or disaster recovery systems to solve many of our e-discovery challenges and simultaneously cut costs!”

The Attorneys to I.T. and Management:  “Terrible news!  You can leverage your existing ESI backup and/or disaster recovery
systems to solve many of your e-discovery challenges and simultaneously cut costs!”

Why both?  Sauce for the goose is good for the gander – anything that makes it easier for you to access ESI, also makes it easier for your adversary.  But is it that simple?

Key phrases to keep in mind; ‘accessible’, ‘not reasonably accessible’, ‘inaccessible’ and ‘cost-shifting’.  The Federal rule states:

A party need not provide
discovery of electronically stored
information from sources that the party
identifies as not reasonably accessible
because of undue burden or cost. On motion
to compel discovery or for a protective order,
the party from whom discovery is sought
must show that the information is not
reasonably accessible because of undue
burden or cost. If that showing is made, the
court may nonetheless order discovery from
such sources if the requesting party shows
good cause
, considering the limitations of
Rule 26(b)(2)(C).
Fed.R.Civ.P. 26(b)(2)(B), italics added.

j0178039The courts sought to define the parameters in a series of rulings, commonly referred to as ‘Zubulake I, II, III, IV & V‘!  These are not new rulings by any means (2003-2004), and I dealt with a case on this very issue in 1997, but because so many IT groups are ramping up their e-discovery bona fides at this time, this would be a good opportunity to revisit Zubulake and make sure you understand the implications.

In the normal course of business, one might implement a solution, then policy follows.  This is definitely one of those times where you should be thinking about policy – and consulting your legal resources – before you implement the solution or modify your current one.  After all, a lot of IT professionals don’t read cases nor know of their implications.

I can’t count how many times I’ve been asked, “How long do we have to keep this stuff?”  Is it possible 37 days is enough?  Gippetti v. UPS, Inc., 2008 WL 3264483 (N.D. Cal. Aug. 6, 2008)

Think about it; what does “keep” mean, exactly?  What does “stuff” mean, exactly?  Can a single data retention policy apply to “everything” or only certain types of ESI; and should you apply a different retention standard to various forms of ESI, based on their use?

Let’s say you have a policy that you delete ESI after X months.  Do you retain or destroy the backup media?  Do employees thwart you by archiving data to their office PCs – or worse – store it on the internet, a personal PC or a thumb drive?

This should be part of your thinking as you craft policy.  It matters whether you can answer those questions.  If not, be prepared for an unpleasant surprise when your adversary comes looking for this information.

Testing 1-2-3…Are you ‘Really’ Ready for a Litigation Request?

Part II of a two-part series.  Part I appeared 11/24/08.

PART II – ESI COLLECTION

I read a lot of of excellent articles, white papers and documents (as do you)
which present reasonable, astute and prescient approaches to getting a
handle on your company’s ESI (electronically stored information).

However, in virtually all of the materials I see, one important element is missing:

TESTING.

j0433180Buildings run fire drills. Do you run data recovery drills?

Sounds counterintuitive, doesn’t it?  Common sense would tell you that if you’re backing up your data, it should be relatively easy to recover it on demand.  After all, the software “tells” you in your morning report that last night’s run went fine.  But did it?  Is that all that matters?

 Think about it for a moment.  How many spokes are in your hub?  Where are they?  How many people are responsible for protecting the data?  What software do you use?  What hardware?  What media?  Is it easily accessible?  Physically?  Remotely?  Do you handle it in-house or do you depend on outside vendors?  Do you use off-site media storage?  Do you know the time it would require for you to comply with a request to produce data?  Do you have an alternate location to restore it?  It isn’t always restored to the location where it originated, and certainly not when litigation is involved.

Let’s boil it down to one simple question.  What would you do if you received a call with a demand for data – a large quantity of data – that isn’t at your fingertips?

Woman with Headache --- Image by © Royalty-Free/Corbis

It would surprise you how many companies haven’t thought about this.  They do everything right in terms of the front-end of this process, but never anticipate the back-end.  They do a terrific job of thinking about data protection, yet don’t think about more important issues – data integrity and the ability to restore it.

What good is all of this technology if, when the big request comes down, you can’t deliver?  It’s bad enough when this has nothing to do with e-discovery (such as my location in California, where we have to worry about earthquakes), but when it does, there are sanctions on the line – and not just civil sanctions.  Some of the penalties are criminal in nature.

Admittedly, criminal liability would most likely require intentional and/or egregious conduct, but the spectre is out there (I’ll address the facts vs. fictions in a future post).

You don’t want to be the attorney who has to stand in front of the judge and say “I’m sorry, Your Honor.” because you are either experiencing delays in producing the data, produced it very late in the litigation process or are unable to produce it at all.  You might get a response like this one from a Judge in the recent McAfee case – “Heads will have to roll“.

Let’s hope it isn’t your head she’s talking about.

Disaster, Recovery and e-Discovery – What You Don’t Know CAN Hurt You

Part I of a two-part series.  Part II will appear 11/25/08.

PART I – ESI IDENTIFICATION & PRESERVATION

j0439550Perception is reality – or so the saying goes.  With e-discovery, perception cannot be reality.  The divergence of these concepts is illustrated by the following statistics:

When queried, a high percentage of law firms and in-house counsel believe the companies they represent are ready to comply with a litigation request.

Apparently, they didn’t ask the IT department.  A dismal percentage of IT managers believe they are ready to comply.

A lot of this obvious disconnect can be attributed to lack of communication between the parties.  However, another major element is what’s lost in translation.  Do the attorneys understand how IT accomplishes this task – or the difficulty of achieving it?  Does IT understand what the attorneys are asking of them?  Do both groups understand what is encompassed in the term “ESI” (electronically stored information)?

Lawyers are thinking about the litigation hold.  IT is thinking about incremental, differential and full backups.  Never the twain shall meet.

How many times has IT received a call like this?  “I created a document this morning and I accidentally overwrote it this afternoon.  Can you please restore it for me?”  That’s a problem.  Regardless of what day it may be in the rotation, most companies perform a back-up once per evening.  As such, there is no back-up of the caller’s file.  Unless the over-written file can be restored somehow from the disk it was saved to, the caller is out of luck.

Back-ups are not normally a dynamic process; they’re snapshots in time.  Even if you do full backups every night, theoretically, an infinite number of people may “touch” a file between those two periods.  This is something lawyers would easily understand; but many are not aware of it.

What the lawyers need is for the data to not only be located – and restored, if necessary.  The data must also be preserved.  Nobody must touch or modify that snapshot – a line in the sand, so to speak.  Again, this is something IT would easily understand; but many are not aware of it; nor the massive amounts of storage that may be required to accomplish it.

Also, most rotation schemes involve eventually overwriting the media (Grandfather-Father-Son?  Tower of Hanoi?).  What happens if, like in the recent McAfee case, data is requested that is from the year 2000?

Rows of Drawers at Library ca. 2001

Basic definitions also come into play.  Do all of the parties mean the same thing when they use the terminology?  What is a back-up and a restore?  What is disaster-recovery?  Do you have separate processes for each?  Are they considered the same thing at your company?  What is the intent of the process; ready access to the files or worst-case-scenario access?  Is the data stored on-site or off-site?  Both?

IT is thinking about how feasible it is to access the data.  Attorneys are thinking about Zubulake.

Be careful you’re not creating your own homonyms.  Webster’s Dictionary defines them as, “Two words…pronounced or spelled the same way but have different meanings”.

If Legal thinks it’s one thing and IT thinks it’s another, both groups are going to face some very unpleasant realities down the road.  This would be a good time to get those definitions written down.  Then make sure you’re all on the same page.