Category Archives: Backup & Recovery Systems

Baby? Meet Bathwater…

J0437389 Not that I want to pick on Computerworld, but…

I was reading their article, "Why IT should start throwing data away" and, while it's a great document with a lot of astute information, it still made my blood boil.  Why?

This is a classic example of a terrific thought process that only considers one side of the equasion.  The perspective is, 'Let's do what we can to eliminate as much data as legally possible to limit the strain on our resources, thwart adversarial e-discovery and lower costs'.

But here's the thing.  Some of that data may exculpate the company and you may really regret deleting it.  Sure, I've talked to many attorneys who operate under the philosophy that the least amount of data available, the better, and you know what?  I don't agree.  At least not without a case-by-case assessment, anyway.

Sauce for the goose is good for the gander.  Don't be so efficient in your data sweeps that you have a defensible legal hold, but leave yourself with no line of defense.

I’m Good Enough, I’m Smart Enough & Doggonit, People Like Me…

Cute Sheep…and eventually, I’ll write an in-depth analysis of California AB 5 and contrast it with the Federal Rules.  But, take a look at this fine analysis from Law.com regarding how California will deal with Zubulake ‘accessible vs. inaccessible’ ESI and how it contrasts with the Federal rules.

Finished reading?  Great.  Now let me tell you why nothing in the analysis rattles me.  You should have been treating your ESI as accessible all along.  Here’s why:

The law is all about exceptions.  Everyone knows the general rule, but ultimately the facts dictate whether an exception is in order.  At Sony Pictures in 1997, we successfully fought off a request for ESI in a California court because we made the case that it was accessible, but at punitive cost because we’d long since retired that particular backup system and complying would have required a $250,000 outlay (there was no product like Index Engines around in 1997).

I had a very smart professor in law school.  His advice was to always assume the worst-case scenario, then work backwards.  It’s actually a very logical approach.  Yes, the flood may be that bad, yes, the stock market may fall that much and yes, your adversary may make a persuasive argument to the judge.

Be an actuary.  They look at everything as ‘cost vs. risk’.  Which will likely cost more, paying to protect or paying after the fact?  A young PC user asked me the other day, “Why do I need a firewall?  What are the chances that I will be the one that will be hacked?”  My answer was, “If you want to gamble, that’s your call, but here’s a short list of things that could happen if you’re the unlucky one.”

Gamble if you like; but be prepared to face the consequences.

Observations on Off-Site, On-Site, Outsourcing & Ownership

j0438776Seems to me there are a lot of companies selling data & e-discovery services with the attitude of, “Place your data responsibilities with us, then sleep well at night”.  Hardware and software are offered in-house, SaaS, appliance, off-site…anything you want can be provided.

My personal opinion; before you start relinquishing responsibility to others, keep one thing in mind – it won’t matter.  You’ll ultimately be responsible in the eyes of the law.

I blogged about this before in my ‘Hot Potato‘ post, among others.  The instinctive thing to do – especially with the added complexity of the e-discovery rules hanging over you – is to contract out and make it someone else’s problem.  Heck, I get that.  I’m a Contractor!  Only thing is, in this case…it won’t work.  Good-faith won’t be enough.

I’m not living in a fantasy world.  Some companies have so much data – including ones I’ve consulted with – it would be virtually impossible to manage in-house.  If I said “Don’t do it!” I would expect you to laugh me out of the room (which would be difficult, since none of you know where my ‘room’ is, exactly).  All I’m suggesting is, before you consider outsourcing data management; whether it be on-site, off-site or a combination of both – or even if you’ve already done so – think about all the risks, especially in these difficult economic times.  Do you have a contingency plan in place?

These are the items I’d be including in a checklist (order of preference is up to you):

Hold on a second.  Let’s begin by answering a fundamental question.  Who will manage this?  You?  The Vendor?

Rural Road from a Car

I know.  Some of you are asking what that means?  After all, regardless of how you proceed, somebody representing the company will be responsible for managing this or serving as liaison, right?  Yes and no.

If you don’t know your ‘stuff’, then aside from serving as liaison, you’ve relinquished your ability to make decisions in the best interests of the company.  Essentially, the Vendor will be advising you, and their interests may conflict with yours – especially if litigation arises.  In the alternative scenario, if you’ve educated yourself – or have hired a knowledgeable representative in-house – you’ll be advising them.

Think this is a distinction without a difference?  Take a look at my checklist and see what you think:

  1. Does the Vendor handle backup, restore, disaster-recovery and/or e-discovery services?
  2. Are all of their products integrated?  (Many Vendors acquired other Vendors to stake a presence in the e-discovery field; it doesn’t mean their products integrate well).
  3. What if the Vendor goes bankrupt?
  4. How will the Vendor respond if/when they’re served with a subpoena as a 3rd-party?
  5. Does the Vendor have their own legal representation?
  6. Who will be responsible for managing the retrieval of data?
  7. How quickly can/will the Vendor respond to a request?
  8. Does the Vendor subcontract any services?
  9. Will an additional Vendor be needed for e-discovery if the 1st Vendor doesn’t have that capability?  Do they already have a secondary Vendor in place?
  10. As we expand – including to other countries/continents – how will the Vendor handle it?

I realize this is a ‘macro’ view.  The list above should open up several more questions, such as how are they backing up your backups?

I would think it would be very important to instruct the Vendor about what you expect, rather than rely on the Vendor to tell you what they’re going to do for you.  There’s no room for ambiguity where e-discovery in concerned.

Alliteration always assists attorneys acting as authors…

The ‘Missile Command Act’? No, the ‘Internet Safety Act’!

A5200_Missile_CommandI think my career is about to resemble Missile Command.  It was all the rage in the 1980s.  Atari still exists and I was surprised to see that they’re still selling it.

The name of the game is to intercept falling missiles (which have an annoying tendency to split off in multiple directions) with silos on the ground (hint; we’re the silos).

John Cornyn (R) has introduced the “Internet Stopping Adults Facilitating the Exploitation of Today’s Youth Act”, or ‘Internet Safety Act’ (for those of us who can’t fit all that in a catchy blog title).  This bill is actually a regurgitation of a bill introduced in 2006.  I think you get the gist from the bill’s title, but here’s the fine print:

“A provider of an electronic communication service or remote computing service shall retain for a period of at least two years all records or other information pertaining to the identity of a user of a temporarily assigned network address the service assigns to that user.”

Anybody besides me thinking about the storage/costs required to retain and/or restore these logs?5_Day

That’s Part I of the headache.  Part II is who would be covered under this bill; essentially anyone who serves wireless using DHCP.  That’s right – it includes that little Wi-Fi router you have at home.  Note to those who brought their wireless router home from the store and
just plugged it in; you might want to configure the security feature, lest someone nearby connect through it and start looking at child
pornography.  Starting to sweat, yet?  Maybe you will after I mention Part III; you might go to jail for up to 10 years.

Here’s the really bad news – there’s a Part IV…

Once again, all I think about is Zubulake.  The moment you’re required to retain a record for two years, it may be adjudged ‘accessible’ for Zubulake purposes – and not just the ones covered under this Act, which, as I previously mentioned, specifically targets child pornography.  Any purpose of litigation may be fair game to subpoena these logs!

You think maybe Senator Cornyn knows how to push a bill through Congress by piggybacking it on the hot-button terms that frighten all parents to death?

This Act really could be the legal equivalent to ‘Missile Command’ (or starfish, or octopus…).  The tentacles could reach virtually anywhere.  I’ll be monitoring this closely, as should you.  If it becomes law, it could be…Missile Command - The End

I Have some Good News & some Bad News…

*** My n/w and phones are out, so I’m coming to you live from the Redondo Beach Public Library, courtesy of their free wireless service…THANK YOU!!! ***

j0178660

Why did the goose cross the road?  Let’s take a gander…

I.T. to the Attorneys and Management:  “Great news!  We can leverage our existing ESI backup and/or disaster recovery systems to solve many of our e-discovery challenges and simultaneously cut costs!”

The Attorneys to I.T. and Management:  “Terrible news!  You can leverage your existing ESI backup and/or disaster recovery
systems to solve many of your e-discovery challenges and simultaneously cut costs!”

Why both?  Sauce for the goose is good for the gander – anything that makes it easier for you to access ESI, also makes it easier for your adversary.  But is it that simple?

Key phrases to keep in mind; ‘accessible’, ‘not reasonably accessible’, ‘inaccessible’ and ‘cost-shifting’.  The Federal rule states:

A party need not provide
discovery of electronically stored
information from sources that the party
identifies as not reasonably accessible
because of undue burden or cost. On motion
to compel discovery or for a protective order,
the party from whom discovery is sought
must show that the information is not
reasonably accessible because of undue
burden or cost. If that showing is made, the
court may nonetheless order discovery from
such sources if the requesting party shows
good cause
, considering the limitations of
Rule 26(b)(2)(C).
Fed.R.Civ.P. 26(b)(2)(B), italics added.

j0178039The courts sought to define the parameters in a series of rulings, commonly referred to as ‘Zubulake I, II, III, IV & V‘!  These are not new rulings by any means (2003-2004), and I dealt with a case on this very issue in 1997, but because so many IT groups are ramping up their e-discovery bona fides at this time, this would be a good opportunity to revisit Zubulake and make sure you understand the implications.

In the normal course of business, one might implement a solution, then policy follows.  This is definitely one of those times where you should be thinking about policy – and consulting your legal resources – before you implement the solution or modify your current one.  After all, a lot of IT professionals don’t read cases nor know of their implications.

I can’t count how many times I’ve been asked, “How long do we have to keep this stuff?”  Is it possible 37 days is enough?  Gippetti v. UPS, Inc., 2008 WL 3264483 (N.D. Cal. Aug. 6, 2008)

Think about it; what does “keep” mean, exactly?  What does “stuff” mean, exactly?  Can a single data retention policy apply to “everything” or only certain types of ESI; and should you apply a different retention standard to various forms of ESI, based on their use?

Let’s say you have a policy that you delete ESI after X months.  Do you retain or destroy the backup media?  Do employees thwart you by archiving data to their office PCs – or worse – store it on the internet, a personal PC or a thumb drive?

This should be part of your thinking as you craft policy.  It matters whether you can answer those questions.  If not, be prepared for an unpleasant surprise when your adversary comes looking for this information.

Testing 1-2-3…Are you ‘Really’ Ready for a Litigation Request?

Part II of a two-part series.  Part I appeared 11/24/08.

PART II – ESI COLLECTION

I read a lot of of excellent articles, white papers and documents (as do you)
which present reasonable, astute and prescient approaches to getting a
handle on your company’s ESI (electronically stored information).

However, in virtually all of the materials I see, one important element is missing:

TESTING.

j0433180Buildings run fire drills. Do you run data recovery drills?

Sounds counterintuitive, doesn’t it?  Common sense would tell you that if you’re backing up your data, it should be relatively easy to recover it on demand.  After all, the software “tells” you in your morning report that last night’s run went fine.  But did it?  Is that all that matters?

 Think about it for a moment.  How many spokes are in your hub?  Where are they?  How many people are responsible for protecting the data?  What software do you use?  What hardware?  What media?  Is it easily accessible?  Physically?  Remotely?  Do you handle it in-house or do you depend on outside vendors?  Do you use off-site media storage?  Do you know the time it would require for you to comply with a request to produce data?  Do you have an alternate location to restore it?  It isn’t always restored to the location where it originated, and certainly not when litigation is involved.

Let’s boil it down to one simple question.  What would you do if you received a call with a demand for data – a large quantity of data – that isn’t at your fingertips?

Woman with Headache --- Image by © Royalty-Free/Corbis

It would surprise you how many companies haven’t thought about this.  They do everything right in terms of the front-end of this process, but never anticipate the back-end.  They do a terrific job of thinking about data protection, yet don’t think about more important issues – data integrity and the ability to restore it.

What good is all of this technology if, when the big request comes down, you can’t deliver?  It’s bad enough when this has nothing to do with e-discovery (such as my location in California, where we have to worry about earthquakes), but when it does, there are sanctions on the line – and not just civil sanctions.  Some of the penalties are criminal in nature.

Admittedly, criminal liability would most likely require intentional and/or egregious conduct, but the spectre is out there (I’ll address the facts vs. fictions in a future post).

You don’t want to be the attorney who has to stand in front of the judge and say “I’m sorry, Your Honor.” because you are either experiencing delays in producing the data, produced it very late in the litigation process or are unable to produce it at all.  You might get a response like this one from a Judge in the recent McAfee case – “Heads will have to roll“.

Let’s hope it isn’t your head she’s talking about.

Disaster, Recovery and e-Discovery – What You Don’t Know CAN Hurt You

Part I of a two-part series.  Part II will appear 11/25/08.

PART I – ESI IDENTIFICATION & PRESERVATION

j0439550Perception is reality – or so the saying goes.  With e-discovery, perception cannot be reality.  The divergence of these concepts is illustrated by the following statistics:

When queried, a high percentage of law firms and in-house counsel believe the companies they represent are ready to comply with a litigation request.

Apparently, they didn’t ask the IT department.  A dismal percentage of IT managers believe they are ready to comply.

A lot of this obvious disconnect can be attributed to lack of communication between the parties.  However, another major element is what’s lost in translation.  Do the attorneys understand how IT accomplishes this task – or the difficulty of achieving it?  Does IT understand what the attorneys are asking of them?  Do both groups understand what is encompassed in the term “ESI” (electronically stored information)?

Lawyers are thinking about the litigation hold.  IT is thinking about incremental, differential and full backups.  Never the twain shall meet.

How many times has IT received a call like this?  “I created a document this morning and I accidentally overwrote it this afternoon.  Can you please restore it for me?”  That’s a problem.  Regardless of what day it may be in the rotation, most companies perform a back-up once per evening.  As such, there is no back-up of the caller’s file.  Unless the over-written file can be restored somehow from the disk it was saved to, the caller is out of luck.

Back-ups are not normally a dynamic process; they’re snapshots in time.  Even if you do full backups every night, theoretically, an infinite number of people may “touch” a file between those two periods.  This is something lawyers would easily understand; but many are not aware of it.

What the lawyers need is for the data to not only be located – and restored, if necessary.  The data must also be preserved.  Nobody must touch or modify that snapshot – a line in the sand, so to speak.  Again, this is something IT would easily understand; but many are not aware of it; nor the massive amounts of storage that may be required to accomplish it.

Also, most rotation schemes involve eventually overwriting the media (Grandfather-Father-Son?  Tower of Hanoi?).  What happens if, like in the recent McAfee case, data is requested that is from the year 2000?

Rows of Drawers at Library ca. 2001

Basic definitions also come into play.  Do all of the parties mean the same thing when they use the terminology?  What is a back-up and a restore?  What is disaster-recovery?  Do you have separate processes for each?  Are they considered the same thing at your company?  What is the intent of the process; ready access to the files or worst-case-scenario access?  Is the data stored on-site or off-site?  Both?

IT is thinking about how feasible it is to access the data.  Attorneys are thinking about Zubulake.

Be careful you’re not creating your own homonyms.  Webster’s Dictionary defines them as, “Two words…pronounced or spelled the same way but have different meanings”.

If Legal thinks it’s one thing and IT thinks it’s another, both groups are going to face some very unpleasant realities down the road.  This would be a good time to get those definitions written down.  Then make sure you’re all on the same page.

Back to the Future – Reebok v. Tristar, 1996 (the “Jerry Maguire” case)

*** NOTE – No privileged or proprietary information is contained in this post. ***

Movie Reel

My first foray into the realm of e-discovery occurred in early 1997 – when it was still just called “discovery”.  I was a Consultant to Sony Pictures Entertainment at the time and Manager of Groupware Services Worldwide, which – unfortunately for me – included responsibility for the company email system.  I was not yet an attorney.

(I have a feeling most of you know where this is going…).

In late, 1996, Reebok Int’l filed suit against Tristar Pictures (at the time a subdivision of SPE) for breach of contract due to the handling of a product placement in the movie, “Jerry Maguire“.  Reebok’s attorneys issued a subpoena for relevant email correspondence between Tristar representatives who were parties to the negotiations.

We faced a serious problem, which was not an unusual one given the time elapsed between negotiations to make a motion picture and the actual production and release of that picture.  The emails were several years old and the Company had done away with the archaic tape backup system used at the time.

A consultant’s job is to find a reasonable method to deliver what a client requests.  As such, I tasked one of our best number-crunchers to figure out what it would realistically take to re-create the prior backup system from scratch, then catalog all of the old tapes to
even give us a starting point as to what would be required for review and production.  Keep in mind that this was a much more difficult feat to accomplish in 1997 than it is today.  The results were striking.  The estimated cost to comply with the subpoena was approximately $250,000!Movie Reel and Film

Obviously, management wasn’t too keen on the idea of spending that sum of money, and thus began a motion by Tristar’s representatives to quash the subpoena due to the high cost, or failing that, shift the burden – or at least a large portion of it – onto the Plaintiff.  Being on the tech side of things – and with a stack of responsibilities on my desk – I moved on to the next “crisis” and have no knowledge as to what specifically transpired after that.  Eventually, the word came down from on high; “you don’t have to worry about producing the data”.  Whew!

I wanted to relay this story because it mirrors exactly how an e-discovery request might fall upon an IT department today.  It also raises several of the most important issues:

Are we able to comply with the request?  How much time/resources will this take away from our other pressing issues?  How much will this cost?  Who will bear the cost?

Luckily, I had at my disposal the qualified brainpower to comply – and had we been asked to proceed, we could have done so.  But it would also have meant taking one of my best minds away from what he was doing, leaving me short-handed with the prospect of making do without him or hiring a temporary replacement and bringing him/her “up to speed”.

The question is, what would happen if you received the request?

Attorneys – Get with the ‘Program’

An interesting survey appeared in the September 2008 issue of the American Bar Association Journal.  The subheading states, “Lawyers Slow to Adopt Cutting-Edge Technology“.

I took solace in the statistic that only 2% of lawyers maintain a law blog (assuming readers consider this a blawg then I’m certainly ahead of the pack) and only 8% of law firms follow suit, but as a general trend, the data is somewhat troubling.  It’s a symptom of a larger illness.

The number-one complaint against attorneys is lack of communication.  I’m not just speaking in terms of what their employees or clients say – it’s also the number-one complaint lodged against them with bar associations.

OLYMPUS DIGITAL CAMERA

It’s bad enough when the subject is one in which an attorney feels comfortable discussing.  But add complex technology to the mix and that’s a recipe for disaster.  In the “wild west” days of e-discovery – before it even had a name – one could get away with mistakes.  Now that the rules have been formalized, the path is littered with attorneys – and their clients – who have suffered greatly for their mistakes.

There’s an old saying, attributed to Confucius, which states; “He who does not know, and does not know that he does not know, is a fool”.  In the e-discovery world, the new saying is, “He who does not know, and does not know that he does not know, will be sanctioned”.

 

Core competency in this area is no longer hoped for; it is expected j0439531and presumed, both by clients and the courts.  It’s not enough for attorneys to rely on IT personnel; they must also be able to understand what their IT professionals are telling them so they can communicate this information effectively to their clients, the courts and even their adversaries. Otherwise, it’s the attorneys and their clients who will bear the consequences of mistakes.

Further hampering this process is the fact that very few IT personnel speak “English”.  Many a layman has become glassy-eyed while listening to a “techie” explain a process in “techno-speak” while not understanding a word of what was said.

Like it or not, the onus is on the legal professional to be competent and understand this process.  If something goes wrong, blaming the incident on a lack of technical knowledge and expertise is not going to fly.