When you think of Tom Wolfe’s “The Right Stuff,” test pilots and astronauts immediately come to mind. Aviators who risked their lives testing new technology that went higher and faster than anything else that could fly.
The pilots who flew the X planes of the 1940s to 1960s built the future we have today. Simply put, without the Space Program, we would not have had the Computer Era of the 1970s. Without the innovations of the 1970s, we would not have social networks and smartphones of today.
Lawyers and Judges do not come to mind when you say “test pilot.” However, we do have brave attorneys and judges willing to “fly higher” than others for the greater good. Magistrate Judge Andrew Peck is one of them for his Da Silva Moore v Publicis Groupe & MSL Group opinion.
Judge Peck’s “computer-assisted review” opinion is a watershed because of its recognition of using technology to save money and find responsive electronically stored information. As Michael Arkfeld commented on the Moore opinion being upheld, “Years from now we will look back and refer to this as the Zubulake for Search!”
When discovery is in the terabytes and there are millions of files to review, Federal Rule of Civil Procedure Rule 1 requirements to “secure the just, speedy, and inexpensive determination” of a lawsuit are often the first casualties of litigation. Fed. R. Civ. P. 1.
The Plaintiffs challenged Judge Peck’s Da Silva Moore order on numerous grounds. District Judge Andrew Carter upheld the order, referring to the findings as “well reasoned and they consider the potential advantages and pitfalls of the predictive coding software.” Moore v. Publicis Groupe & Msl Group, 11 Civ. 1279 (ALC) (AJP).
Let’s explore the significance of Magistrate Judge Peck’s order.
What is Computer Assisted Review?
There are many forms of “computer-assisted review” besides “predictive coding.” For example, California Rules of Court Rule 3.750(b)(10) allows a Court in complex litigation to order the parties to use an “electronic document depository.” Such a depository would likely be a “cloud” solution, enabling parties to individually login to access the discovery for search and review.
Magistrate Judge Facciola is also no stranger to cases highlighting “computer-assisted review.”
In El-Amin v. George Wash. Univ., Judge Facciola ordered the parties consider using a hosted review platform.
The order set out as a “primary goal” for the parties to select a review platform with “hyper-linked to fields in a database that will permit the instantaneous retrieval from within the database of the information offered by plaintiffs in support of any factual proposition.” El-Amin v. George Wash. Univ., 2008 U.S. Dist. LEXIS 85009 (D.D.C. Oct. 22, 2008).
Judge Facciola also warned of trying to use every new litigation support technology to trying to find electronically stored information:
[N]ew technologies have the capacity to be outcome determinative but often at significant expense. Thus the courts are required to strike a balance between allowing the requesting party to take full advantage of the technologies available to it and protecting the producing party from having to pay to leave no stone unturned. Resting all of the costs of electronic discovery on the producing party may create a perverse incentive on the part of the requesting party to dispense with reason and restraint and unleash every new technology under the sun to try and find information that supports the requesting party’s claims.
Covad Communs. Co. v. Revonet, Inc., 2009 U.S. Dist. LEXIS 47841, at * 29-30 (D.D.C. May 27, 2009).
Lite This Candle: Data Analytics & Discovery
“This judicial opinion now recognizes that computer-assisted review is an acceptable wayto search for relevant ESI in appropriate cases.”
Judge Andrew Peck
Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *2 (S.D.N.Y. Feb. 24, 2012)
“Computer-Assisted Review” as discussed in the Da Silva Moore opinion focused on “predictive coding.”
This is a significant difference than many of the past cases addressing “computer-assisted review,” because many of those cases either focus on 1) simply using a review application; or 2) search terms and/or the adequacy of a production.
Judge Peck defined “computer-assisted coding” in his article “Search Forward,” as the “use [of] sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer.” Moore, at *5.
“Predictive coding” is probably better defined as data or mechanical analytics. “Data Analytics” is defined on SearchDataManagement.com as follows:
Data analytics (DA) is the science of examining raw data with the purpose of drawing conclusions about that information. Data analytics is used in many industries to allow companies and organization to make better business decisions and in the sciences to verify or disprove existing models or theories. Data analytics is distinguished from data mining by the scope, purpose and focus of the analysis. Data miners sort through huge data sets using sophisticated software to identify undiscovered patterns and establish hidden relationships. Data analytics focuses on inference, the process of deriving a conclusion based solely on what is already known by the researcher.
Taking Flight: Goals of Discovery Review
The practice of law is frequently a casualty when it comes to reviewing large sets of data in discovery.
It is important not to forget the goals of discovery: Document review does not exist for the sake of document review; it exists to find relevant information.
Magistrate Judge Peck outlined the following goals of discovery review in his order, which stand as a strong reminder of the purpose of discovery:
The objective of review in ediscovery is to identify as many relevant documents as possible, while reviewing as few non-relevant documents as possible.
Recall is the fraction of relevant documents identified during a review; precision is the fraction of identified documents that are relevant.
Thus, recall is a measure of completeness, while precision is a measure of accuracy or correctness.
The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the “value” of the case.
Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *27 (S.D.N.Y. Feb. 24, 2012) (Emphasis Added)
The Eyes of the World Are Upon You: Why Court Found Computer Assisted Review Appropriate
Magistrate Judge Peck found “computer-assisted review” to be appropriate for the following reasons:
(1) The parties’ agreement [Something contested by the Plaintiffs]
(2) The vast amount of ESI to be reviewed (over three million documents)
(3) The superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches)
(4) The need for cost effectiveness and proportionality under Rule 26(b)(2)(C)
(5) The transparent process proposed by [Defendant].
Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *35-36 (S.D.N.Y. Feb. 24, 2012)
Fairing: Reading Rule 26(g)
“In large-data cases like this, involving over three million emails, no lawyer using any search method could honestly certify that its production is “complete” — but more importantly, Rule 26(g)(1) does not require that. Plaintiffs simply misread Rule 26(g)(1).”
Judge Andrew Peck
Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, 20-23 (S.D.N.Y. Feb. 24, 2012).
The Plaintiffs challenged the predictive coding protocol under Rule 26(g)(1)(A), arguing that production had to be certified as “complete,” and that the Defendants were given “unlawful cover” with the predictive coding protocol. Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *21 (S.D.N.Y. Feb. 24, 2012).
Judge Peck held the Plaintiffs’ reading of Rule 26(g)(1)(A) was erroneous, because the certification requirements applied to initial disclosures under Rule 26(a)(1); Discovery responses are covered by Rule 26(b)(2)(C)’s proportionality principle. Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, 20-23 (S.D.N.Y. Feb. 24, 2012).
Go Around: Federal Rule of Evidence Rule 702 & Daubert
The Plaintiffs challenged the predictive coding protocol as violating Federal Rule of Evidence Rule 702 and Daubert, in part, because the Defense experts spoke at the hearing, but were not sworn in at the time. Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *23 (S.D.N.Y. Feb. 24, 2012).
The Court held that Rule 702 & Daubert apply to presenting evidence at trial; the Rules are not applicable in how ESI is searched for in discovery. Id. As the Court explained:
If MSL sought to have its expert testify at trial and introduce the results of its ESI protocol into evidence, Daubert and Rule 702 would apply. Here, in contrast, the tens of thousands of emails that will be produced in discovery are not being offered into evidence at trial as the result of a scientific process or otherwise. The admissibility of specific emails at trial will depend upon each email itself (for example, whether it is hearsay, or a business record or party admission), not how it was found during discovery.
Moore v. Publicis Groupe & Msl Group, 2012 U.S. Dist. LEXIS 23350, at *23 (S.D.N.Y. Feb. 24, 2012).
Message from the Control Tower
District Judge Andrew Carter upheld Magistrate Judge Peck’s order after the Plaintiffs challenged Judge Peck’s order on numerous grounds. Moore v. Publicis Groupe & Msl Group, 11 Civ. 1279 (ALC) (AJP).
Judge Carter said the following on the protocol (and the confusion over whether the Plaintiffs agreed to the protocol):
Nevertheless, the confusion is immaterial because the ESI protocol contains standards for measuring the reliability of the process and the protocol builds in levels of participation by Plaintiffs. It provides that the search methods will be carefully crafted and tested for quality assurance, with Plaintiffs participating in their implementation. For example, Plaintiffs’ counsel may provide keywords and review the documents and the issue coding before the production is made. If there is a concern with the relevance of the culled documents, the parties may raise the issue before Judge Peck before the final production. Further, upon the receipt of the production, if Plaintiffs determine that they are missing relevant documents, they may revisit the issue of whether the software is the best method. At this stage, there is insufficient evidence to conclude that the use of the predictive coding software will deny Plaintiffs access to liberal discovery.
Moore v. Publicis Groupe & Msl Group, 11 Civ. 1279, at *3-4 (ALC) (AJP).
Judge Carter found the challenge on the reliability of the “predictive coding” were “premature.” The Court stated, “It is difficult to ascertain that the predictive software is less reliable than the traditional keyword search.” Moore v. Publicis Groupe & Msl Group, 11 Civ. 1279, at *4 (ALC) (AJP).
The Court noted that experts were present at the hearing and that the “lack of a formal evidentiary hearing at the conference is a minor issue because if the method appears unreliable as the litigation continues and the parties continue to dispute its effectiveness, the Magistrate Judge may then conduct an evidentiary hearing.” Id.
Judge Carter stated the following on Plaintiffs’ challenges to the protocol being “speculative”:
Judge Peck is in the best position to determine when and if an evidentiary hearing is required and the exercise of his discretion is not contrary to law. Judge Peck has ruled that if the predictive coding software is flawed or if Plaintiffs are not receiving the types of documents that should be produced, the parties are allowed to reconsider their methods and raise their concerns with the Magistrate Judge. The Court understands that the majority of documentary evidence has to be produced by MSLGroup and that Plaintiffs do not have many documents of their own. If the method provided in the protocol does not work or if the sample size is indeed too small to properly apply the technology, the Court will not preclude Plaintiffs from receiving relevant information, but to call the method unreliable at this stage is speculative.
Moore v. Publicis Groupe & Msl Group, 11 Civ. 1279, at *4 (ALC) (AJP).
Bow Tie Thoughts
Judge Peck and Judge Carter’s opinions will have a lasting impact in eDiscovery. This is not a case of machines replacing humans, but the importance of using technology to identify what is relevant to a lawsuit. Moreover, perfection is not the standard in discovery; proportionality is simply not disappearing ink in the Federal Rules of Civil Procedure.
It is also noteworthy that Judge Carter adopted Judge Peck’s analysis of Rule 26(g) and Federal Rule of Evidence Rule 702 in footnote 3 of his opinion. I have met many attorneys who viewed productions needing to be certified as being “complete and correct” under Rule 26(g)(1)(A). Seeing both a Magistrate Judge and District Judge who held that productions are not held to a standard of perfection, but one of proportionality, should bring more reason to large cases requiring the review of terabytes of ESI.
The goal of discovery is to find relevant information. Data analytics that assist attorneys in determining relevant ESI will enable lawyers to focus on litigating their clients interests, opposed to slogging through protracted document review.
The adoption of “data analytics” in discovery review will be a significant step to focusing on the practice of law instead of review. However, like any technology, it must be used correctly. This requires attorneys with knowledge, who put the time in to learn how the application operates. Moreover, the system must be tested to ensure either the system is working correctly or whether the search terms need to be adjusted to find relevant ESI. There are a host of other technical issues to make sure the process is defensible.
“Data Analytics” will not be limited to helping attorneys with the three million record cases. Solo practitioners to lawyers at mid-sized firms who can leverage this technology (once commercially affordable or offered by service providers) to identify relevant ESI out of 20,000 records in a matter or days, or hours, will provide greater services to their clients than those who are not using such technology.