Both of these clients require confidentiality, but I will be blogging about them anonymously as we move towards final results

“E-discovery sanctions are at an all-time high. We identified 230 sanction awards in 401 cases involving motions for sanctions relating to the discovery of electronically stored information (ESI) in federal courts prior to January 1, 2010. We analyzed these cases for a variety of factors, including sanctioning court, sanctioning authority, sanctioned party, sanction type, and sanctioned misconduct. Our analysis indicates that although the annual number of e-discovery sanction cases is generally increasing, there has been a significant increase in both motions and awards since 2004. Motions for sanctions have been filed in all types of cases and all types of courts. The sanctions imposed against parties in many cases are severe, including dismissals, adverse jury instructions, and significant monetary awards. Sanctions against counsel, although uncommon, are on the rise. All the while, the safe harbor provisions of Rule 37(e) of the Federal Rules of Civil Procedure have provided little protection to parties or counsel.”

This is a crisis that has been a long time in the making, but there have been significant revelations about the depth and breadth of the problem In the past week. Bank of America, GMAC and JP Morgan Chase have all suspended foreclosure actions on the news that legal documents filed in support of foreclosure actions emanating from the leading ‘foreclosure mills’ are fatally flawed and may have been fraudulent. In the wake of these disclosures, Congress and a number of states Attorneys General have launched investigations into foreclosure practices. A few days ago, one of the nations largest title insurance firms stopping writing title insurance for foreclosed properties.

If this were simply a matter of crooked lawyers, paralegals and notaries from a hand-full of foreclosure mills ‘robo-signing’ fraudulent legal filings, a few disbarments or perjury convictions followed by perfecting the pleadings would be the end of the matter. Lawyers cutting corners is nothing new, but the ugly truth is that many of the 64 million mortgages in the Mortgage Electronic Registration System (MERS) many not be capable of have the titles for the underlying properties cleared. See, Peterson, Christopher Lewis, Foreclosure, Subprime Mortgage Lending, and the Mortgage Electronic Registration System (October 5, 2009). University of Cincinnati Law Review, Vol. 78, No. 4, 2010. Available at SSRN: http://ssrn.com/abstract=1469749).

The root of the problem is not that MERS or the banks have failed to maintain the information or records required to secure their interests for reasons of technological or records management incompetence. Rather, the banks and MERS made a conscious decision to avoid the state and country title recording fees because their desire to securitize and trade mortgages with the regularity and speed of the stock market was at odds with the centuries old practice of recording the changes in ownership in the county title recording offices. As Peterson details in his law review article, rectifying this problem will not be easy. First, the trail of ownership from origination through securitization of the mortgage may not be able to be reconstructed due to mortgage originators who no longer exist or poor record keeping on the part of the mortgage servicers or the trusts administering the Mortgage Backed Securities. Moreover, even if the paperwork can be reconstructed,some courts may still not accept that MERS has standing to foreclose in the more than 20 states that require a judicial proceeding to foreclose. This non trivial legal risk seems to be yet another “Black Swan” the Banking Risk Management and legal management community somehow failed to account for while being mesmerized by the prospect of saving a few hundred dollars in filing costs over the life of the mortgage.

I don’t know how this is going to turn out, but speculate that one way or another the courts and the regulators will find a way to prevent the unjust enrichment of deadbeat mortgagors by giving them title to properties they have defaulted on. At the same time, I doubt that many of the state legislatures and courts (who have the most direct control over foreclosure practices will permit the current MERS based system to continue to operate without significant changes. (indeed, just this morning JPMC announced that it stopped using MERS for most purposes back in 2007). The confluence of circumstances will make the banks rethink the risk of a MERS model which does not provide for local title recording and a real repository of the title along with the rest of the mortgage origination/servicing file so that both can be easily obtained and authenticated. I don’t expect these changes to come about because of an altruistic desire to improve the foreclosure process, but because ‘Mortgage-gate”, i.e., the revelation that many mortgages may not be worth the paper they are printed on, will kill the market for Mortgage Backed Securities absent the MBS buyers believing that the underlying assets can be foreclosed on in the event of default. With smart Hedge Funds and lawyers crawling all over the situation and looking for a payoff from someone with deep pockets, suddenly saving a few hundred dollars per mortgage by reducing expenditures for filings and servicing suddenly seems like a recipe for huge losses and the destruction of the Mortgage Backed Security business.

In the past the banking community has created shared archiving solutions that can be extended to provide the Records Management arm of MERS or other bank controlled mortgage servicers. For example, look at the banking community’s response to the Check 21 law. The banks understood that having a better electronic check image archive was not a competitive advantage, so most of the larger banks created Viewpointe to maintain their check archive. The Viewpointe archive now contains about 200 billion check images which are available not only to the banks, but to their checking customers as well. It’s not hard to imagine a similar solution for mortgage records. Most customers think it is a valuable service to have their check images available on line, expect they would feel the same way about their mortgage records. Given the relatively minimal cost of creating a proper archive vs, the risk of not being able to foreclose on properties in default, it is hard to image the banks will let this situation continue.

In response to critics who claimed that the new law left too much of the detail to regulatory rule making, Senator Dodd was quoted in this morning’s Washington Post as saying “What do they expect me to write, a 100,000-page bill? This is far beyond the capacity, the expertise, the knowledge of a Congress”. I think Dodd seriously undercounted the number of pages that will be generated to fill in the blanks. The Harvard Law forum estimates the new bill calls for 243 rule-makings and 67 studies. Perhaps we should rename the bill the Lawyers and Lobbyist’s Full Employment Act of 2010. Let the games begin.

Overview of the proposed Consolidated Audit Trail requirements

The impetus for the new rule is based on the SEC’s belief that,

• “(W)ith today’s fast, electronic and interconnected markets, there is a heightened need for a single uniform electronic cross-market order and execution tracking system that includes more information than is captured by the existing SRO audit trails, and in a uniform format.” (p17)

The scope of the rule:

• “(W)ould require the consolidated audit trail to capture certain information about each order for an NMS security, including the identity of the customer placing the order and the routing, modification, cancellation or execution of the order, in real time. In effect, the proposal would create a time-stamped “electronic audit trail record or report” for every order, and each market participant that touches the order would be required to report information about certain reportable events, such as routing or execution of the order. “(p33)

While the first phase of the rule is limited to National Market Securities, the SEC plainly intends to expand its scope to virtually every variety of regulated equity and debt (including asset backed securities);

• “The Commission ultimately intends for the consolidated audit trail to cover secondary market transactions in other securities, including equity securities that are not NMS securities, corporate bonds, municipal bonds, and asset- backed securities and other debt instruments; credit default swaps, equity swaps, and other security-based swaps; and any other products that may come under the Commission’s jurisdiction in the future. Further, the Commission preliminarily believes that it would be beneficial to provide for the possible expansion of the consolidated audit trail to include information on primary market transactions in NMS stocks and other equity securities that are not NMS stocks, as well as primary market transactions in debt securities.” (p50-51)

While not based solely on the SEC Broker Dealer Books and Records Act(SEC Rule 17a), the proposed rule is intended to work hand in hand with the Books and Records act:

• “(T)he information would allow for better trend analysis and outlier identification. It also would improve pre-examination work and the asset verification process, and focus document requests, making the examination process more efficient for the Commission staff and the registrants subject to the process.” (p53)

The proposed Rule also moves the SEC and the rest of the regulatory community closer to its goal of full life time scrutiny of a debt or equity:

• “The Commission preliminarily believes that the required execution information, in combination with the proposed information pertaining to order receipt or origination, modification, or cancellation, would provide regulators with a comprehensive, near real time view of all stages and all participants in the life of an order.” (p81)

In its proposed Rule, the SEC has left open the questions of who should run the Central Repository, precisely what data it should contain and in what format the data should be stored, but the following quote indicates that the Commission is thinking of the open XBRL format:

• “The Commission has recently required that issuers report certain data in interactive data format such as XBRL.     This proposal does not specify any particular or required data format, but allows the SROs to select a data format. Should the Commission require that the data be transmitted or stored in any particular format? What are the relative merits of flat data files, relational data files, and interactive data files? What other formats should be considered? In what format can the SROs and their members efficiently transmit data? In what format would the data required in the proposal be most easily accessed.” (p93)

Finally, the SEC estimates the initial cost to develop the repository at $4 Billion with annual additional costs at $2.1 Billion. These costs are the estimates for all the exchanges and exchange members.

Implications for Financial Services firms

It’s no secret that financial services firms have information repositories that may charitably be described as ‘highly distributed’. A less charitable description would be a pile of “pick-up” sticks. At first glance, it would appear that implementing the CAT requirements might not be too difficult for them — if all they need to do is map their current trading systems to XBRL and send out a feed. However, a closer reading of the proposed rule reveals that the feed, in the short term, must integrate with Customer Relationship Management systems to provide a unique customer number to all trades and to the firm’s Human Resource systems to provide a unique ID number for all traders. This is going to be harder than it appears. Last year, I worked on a proposal for a Tier 1 FS firm to prepare it to comply with the FDIC’s new “Sweep Account” rules. This effort reminded me how difficult it is for most large FS firms – who are all the product of multiple acquisitions and mergers – to distill a single view of any customers account.

The downstream requirements of the proposed rule are even more daunting for the FS firms. If the rule is extended to all manner of debts and equities across their entire life, it will require FS firms to integrate at the customer, trader and servicing level, all of their systems. This is the holy grail of every FS CIO and one that it not currently within reach of any of them as far as I know. But this is a cloud with a silver lining; there is not a single FS firm that could not reduce its Information Technology complexity and costs if it built an IT framework around this level of integration. Moreover, if XBRL were used as the lingua franca of this framework, it would make merger and separation of business units, assets and systems much easier to accomplish.

Implications for Information Technology firms

This could and should be the next great gold rush for the firms that provide Information Technology to the Financial Services industry. The vendors that profited by the SEC’s extension of the Books and Records (17a 3-4) act to include all email and messaging systems used this event to create a substantial business around message archiving (e.g., Autonomy, Symantec, Iron Mountain and EMC). The proposed CAT rules are the most significant change to the Broker Dealer retention requirements since the 2003 amendment of 17a 3 and 4. The new CAT rules also call to mind the new business model which resulted from the SEC’s move to electronic filing, a move that resulted in the creation of the EDGAR system.

However, the largest implication of the proposed CAT rules for technology vendors is that it shoves the door open to a Cloud based system for managing and storing the complete life cycle history and artifacts associated with every type of regulated equity or debt instrument. This is huge – both symbolically and financially. The SEC estimates a cost of $4 Billion to establish the CAT system and annual upkeep of $2.1 Billion. Moreover, the SEC is clearly looking for an open system that will be integrated to and accessed by a large community of regulators, the regulated and perhaps third parties as well. If ever a set of requirements were custom made for a cloud solution, this is it.

So, who might be well advised to look at participating in this “bonanza”:

• Cloud Providers: CAT would be a natural play for any cloud provider with experience building secure, scalable repositories wrapped in web services. The first player who comes to mind is Google. Google is now building private clouds for government – e.g. consider their recent deal teaming with CSC for the implementation of a cloud for Los Angeles County.

• Enterprise Content Management Providers: IBM, EMC, Autonomy and Oracle all have the combination of:
  • large scale ECM systems
  • Compliance tool frameworks
  • the ability to consume XBRL in their middleware and Business Process Management tools
  • a large installed based among FS firms

• Legal Publishers/Virtual Deal Room Providers/Archive Cloud Providers: Merrill, Donnelley, Bowne and Intralinks or IBM (with its new Lotus-live powered cloud and collaboration space), EDGAR Online (especially now that it has acquired XBRL leader UBMatrix), Iron Mountain and Viewpointe. What all of these vendors have in common is:
  • They manage large, secure clouds used by the FS industry ecosystem (regulators, regulated, lawyers and dealmakers).
  • A need to grow new revenues given the fallout from the financial crisis.
  • The ability, as crowd aggregators to leverage the ecosystem for significant ancillary revenue streams.
• Government system integrators/AKA ‘Beltway Bandits”: These folks tend to focus on Defense/Intelligence deals, but CSC is very much a member of the Beltway Bandit club and, as noted above, they teamed with Google to provide the LA County Cloud.

What all of the technology providers should be doing right now is:

• Develop a business plan around CAT requirements. The smart ones will come up with a plan that uses exploitation of ancillary revenue streams to reduce the costs to FINRA, the SEC and even the regulated entities. The players who are most adept at this are Google and the Legal Publishers.

• Position themselves as a thought leaders in CAT (across the legal, regulatory, operational and technological domains) A toss up here, the large technology players have difficulty with business thought leadership and the large publishers are not the most adept at creating technology agendas that become large scale industry drivers.

• Develop a sand box that becomes CAT headquarters for the major stakeholders in the new ecosystem. Here, if you look to either what Google regularly does via Google Labs, or what firms like UBMatrix did with XBRL, you can see how a sandbox can drive adoption of standards and platforms.

Implications for the Public

I am of the firm belief that one of the root causes of the financial crisis is the lack of transparency in financial transactions. The “Sell Side” used the obscurity to demand premium prices and hide risk. The regulators and their enablers in government used the obscurity to control their turf. An open, cloud based information base which can be integrated with every analytical and social networking tool in the infosphere will benefit the “Buy Side”, regulators and risk managers everywhere. This in turn will greatly benefit the public

These two domains, while largely different, share commonalities, but more important, it is the CIO who is responsible for melding both aspects of governance into a single strategy and executing against it. It is this balancing of what are often opposite forces that requires a CIO to master the Yin and Yang of Information Governance.

The second part of this post deals with how to successfully balance the Conformance and Performance aspects of Information Governance.

How the Conformance and Performance functions differ:

Information Governance Performance exists on a continuum of return on value. It is judged based on its enablement of the firm’s strategy. Does the governance of IT result in a price advantage to the firm or provide a differentiating strategic capability to the firm? There is a clear opportunity cost to IT Performance. Most of the IT budget will be allocated to maintaining shared functions, so only a small portion of the budget is available to be used for enabling differentiating capabilities. Assuming a cost leadership strategy based on IT is also difficult since most IT capabilities are readily available for adoption by your competitors. In the end, IT leadership is as difficult and as rare as business leadership. In devising your IT governance strategy you have complete freedom bounded only by your imagination, budget and capacity to create change in your organization and in your market.

Information Governance Conformance is closer to a binary test, does the IT function comply with the required regulations and laws for IT in your industry and in the jurisdictions in which you do business? In most firms, there are far more compliance needs than there is available budget to deal with compliance. Compliance needs must be filtered through a risk assessment to deal with the most serious risks first. The risks themselves should be filtered through the firm’s ‘risk appetite’. If your business has been found in violation of a requirement to retain records or protect privacy, you will likely be judged more harshly for subsequent violations. In devising strategies to achieve compliance, you are more limited in your choices. In the end, your choices are those that will be accepted by regulators, judges and juries.

How the Performance and Conformance functions are alike

Notwithstanding the differences between Performance and Conformance, they share three common foundational approaches as well as the need for some common tools. Here is where the Ying and Yang balance comes into play.

IT Inventory

As the old adage says, “You can’t manage what you cant measure, you can’t measure what you don’t understand, you can’t understand what you haven’t defined.” You would not try to manage a business’s funds without a complete dynamic knowledge of the firm’s cash flows in and out. You would not try to manage the firm’s human resources without a dynamic knowledge of its employees. You would not try to manage a supply chain without a complete list of your vendors. So how in the world do you think you can manage a firm’s IT without an inventory of what information you have and where it is? I am amazed how often my clients have no clear picture of their Information and IT resources.

An IT Inventory is required to support both the Performance and Conformance domains of Information Governance. You cannot begin to measure the benefit of information or its compliance without a current inventory of Information, Hardware and Software assets.

A series of overlapping inventory measures are needed to assure the conformance of IT, among these are:

• • Data Map: A Data Map is a requirement for eDiscovery. Counsel need to be prepared to identify where potentially relevant ESI resides and who the custodians of that ESI are. For the purposes of both discovery and to comply with cross-border data transfer prohibitions, you need to know the geography/jurisdiction in which the data resides.
  • Records Retention Schedule: Every firm needs a Records Retention Schedule, but that is just the beginning. You can’t manage the retention of electronic records without knowing the applications that contain the records as well as the custodians responsible for managing it.
  • Private Information Inventory: Given the requirements to protect private information, you need to know where the private information is.
  • Essential Information Inventory: The foundation of a functioning disaster recovery plan requires knowing what information and applications are essential and insuring that provisions for backup and recovery of these are sufficient for your purposes.

Tools available for creating and managing information and IT inventories are not as mature or integrated as we would like, but are getting better all the time. Mature application portfolio management suites are available from IBM, CA, HP and others. Most of the functions in these tools are geared more to the Performance than the Conformance domains. To these tools you can add and integrate Conformance Management tools from vendors such as PSS and Exterro. The latter tools provide functions for Information Policy Management, Data Mapping, Litigation Hold and Privacy Mapping (though to implement privacy mapping you may well need a Data Leak Protection application). One of the most difficult tasks in bringing your applications under control is to identify the actual contents of your applications, tools such as IBM’s Automated Content Assessment suite and IBM’s Data Discovery suite can be used for these purposes.

Balanced Scorecard:

As Frederick the Great said, “He who defends everything, defends nothing”. A Balanced Scorecard is the best means I know of to focus on those things that are most important to a business — including the value management of its Information portfolio as well as its information compliance efforts. The Balanced Scorecard was developed by in the early 1990s by Robert Kaplan and David Norton (KPMG Peat Marwick/Nolan Norton & Co., where I worked during the time it was developed.) The Balanced Scorecard is imperfect, but the best tool available to translate strategy into action.

There is so much information about Balanced Scorecard available that I won’t go into detail here. Suffice it to say that the two elements that make a Balanced Scorecard so valuable for both the Performance and Conformance functions is that the Balanced Scorecard requires you to prioritize your objective into a small, actionable list and to devise a set of metrics to achieve these objectives. A Balanced Scorecard forces one to filter their strategic priorities and measures the progress towards success through a combination of quantitative financial,operational and satisfaction measures.  As Kaplan recently noted in an interview, the recent financial crisis underscores the requirement to supplement these measures with a set of ‘Risk” metrics with account for a range of business risks including those risks which though rare can have a material impact on the business — what the author Nick Taleb calls “Black Swans”.

It is no surprise that while Balanced Scorecard is the most widely used strategic framework in the Fortune 1000 with more than 53% of firms using it, according to a recent Foster Research paper on the transformation of the CIO function, the most widely used IT strategy tool is ITIL:

What could say more about the disconnect between business leadership and IT leadership  than their divergence on the tools used to run their respective domains?  Moreover, while a balanced scorecard is highly customized to a business’s particular competitive environment, ITIL is a cookbook for IT.

IT Chargeback:

According to a recent Forrester survey, half of IT shops have no mechanism at all to charge back IT costs to business units and only 25% charge back all of their IT costs to business units. It’s no wonder that IT budgets are out of control and business units prioritize all IT needs as “high” thereby frustrating both Performance and Conformance efforts. I have seen this in practice in my consulting work. You talk to the business unit representative and they tell you that they need to keep every bit of information forever, yet when you talk to their IT support people they tell you that the business never asks for any information more than a year old and relies on the data warehouse for long term views. These same organizations now have warehouses full of tapes, not to mention Records Management and eDiscovery nightmares.

IT Chargeback underscores how Performance and Conformance can be brought into a complementary balance. Reducing information to only that which provides real value to the business reduces the cost to the business which frees up funds to spend on IT capable of providing differentiation. At the same time, reduction of information simplifies most compliance tasks. IBM and AIIM both refer to something they call “Information in the Wild” or uncontrolled information. Information which is worth being retained is worth bringing under control. If the information is not worth bringing under control, then the only question is if you have a compliance need for it.

IT chargeback is where you will find the money to pay for the Performance and Conformance improvements you want to make as well as the lever to accelerate the change to governable information. The movement to Cloud Computing will make it easier to implement chargebacks. Most internal IT shops have neither the time or inclination to develop robust IT provisioning and chargeback infrastructures, most Cloud vendors have spent considerable resources developing the means to support multiple flavors of chargebacks.

Conclusion

The Yin and Yang of Information Governance is all about carefully selecting those handful of initiatives which yield improved information governance performance and then quantitatively measuring your success against your goals. Jazz musicians often say, “It’s not about the notes you play, it’s about the notes you don’t play”. The same is true of information governance strategy, you can only afford to play a few notes so choose them very carefully.

The Two Worlds of Information Governance

Most people have heard the the Indian tale of the Blind Men and the Elephant, where each of them touch various parts of the elephant. The man touching the tail concludes that it is a rope, the one touching the ear a fan, the one touching the leg a pillar. Of course all are right and none are completely right.

So too with Information Governance. Ask Data People what Information Governance is and they will talk about the need to ensure the efficient and efficacious use of the enterprise’s data or the need to improve application development through the use of Master Data Management. Ask the same question of Compliance People and they will talk about Privacy, Security, Retention Management and eDiscovery with their own shopping list of tools to aid in this endeavor. Ask the IT Strategy People about this and they will talk about the need to create competitive advantage from IT focusing on the methods and tools to ensure that competitive advantage and business value drive IT investment. They are all correct, yet none is completely correct.

The Blind Men and The Elephant metaphor also extends to most IT vendors working in the Information Governance space, Compliance Tool vendors see it as a compliance play. Even within this sphere, vendors who work in any particular segment of the market, security and privacy or retention management or data management appropriate the entire information compliance label despite the fact they are not covering all areas of information compliance.

The truth of the matter is that Information Governance is so crucial and so difficult precisely because it requires the management of the entire elephant, not just the individual components. The ISO standard for Corporate Governance of Information Technology sets forth the scope of IT Governance as consisting of two realms – Conformance and Performance.

Conformance concerns itself with abiding by the rules, regulations and laws governing the use of IT. Within the purview of this domain are issues such as:

• Security
• Privacy
• Spam
• Trade Practices
• Intellectual Property
• Record Keeping
• Safety & Health
• Accessibility
• Libel & Slander
• Environmental Standards
• Social Standards
• Political/National Security Standards

In essence, this is a Policing Model of IT Governance

Performance is all about extracting from the IT Investment the most business value for the organization. While there is a long list of examples I could offer it all boils down to supporting, enabling or, if you are really good or lucky, creating the basis of the company’s competitive strategy. Unfortunately, for most companies, this ends up being interpreted as creating and managing IT as a shared service – in other words, making sure the train runs on time at the lowest cost. I call this the Stewardship Model of IT Governance. In great companies, Information Technology Performance means more than just Stewardship, it means that IT is an indispensable catalyst for the company’s strategy, whether that strategy is ‘Low Cost’, ‘Differentiation’ or ‘Unique Niche Provider’. I call this the Innovation Model of IT Governance.

Now, I don’t need to tell you that the kinds of personalities, skills and educations typically associated with the Police who run Conformance, the Stewards who manage shared services and the Innovators who create new and improved basis of competition tend to be very different. It’s not an iron clad rule, but it is a tendency. There are highly innovative lawyers and those who just fill out forms, there are clock punching Stewards and those whose dedication to customer services make their shared services a hard to duplicate competitive advantage. There are those who are ‘Innovators’ in name only, taking credit for innovation when all they have done is mindlessly follow someone else’s strategy in a way that is all too easy for a competitor to duplicate.

The Information Governance model is not unique. Governance Models for corporations as a whole and their subordinate functions including those for Finance, Supply Chain, HR, etc all have the same basic features and the same dichotomy between Performance and Conformance.

Smart leaders know how to meld all of these types into a cohesive, effective team. Effective Information Governance requires mastery of both of its realms and a balancing act between Performance and Conformance. Balance between Yin and Yang is the essence of Tai Chi and the essence of Information Governance.

My career has been spent almost equally divided between helping clients obtain the most strategic business value for their IT investment and helping them address the broad range of compliance requirements for IT. In Part 2 of this post, I’ll lay out the differences and similarities in dealing with Performance vs. Conformance as well as the best methods to enhance and blend the combined functions.

Last year I wrote a blog post on IBM’s Risk Technology site about the application of natural language/semantic computing tools to endeavors as diverse as answering questions on the TV show Jeopardy- which IBM is currently working on – as well as the use of the same type of tools for Records Management and eDiscovery. In that post, I asserted that we have reached a point where computers are at least as good at people at tasks such as categorizing documents as records or non records or for classifying documents for purposes of eDiscovery. A recent study provides further proof for this assertion.

In Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, published in the January 2010 issues of the Journal of the American Society for Information Science and Technology, authors Roitblat, Kershaw and Oot (all members of the Electronic Discovery Institute) set forth the results of a study they conducted comparing human review of documents collected for eDiscovery vs. computer review of the same collection. They concluded “on every measure, the performance of the…computer systems was at least as accurate…as that of a human review”

The Roitblat, et. al., study has implications even beyond those set forth in their paper, so lets look in a bit more detail at their methodology.

Roitblat started with a collection of 1.6 million deduplicated documents that had been collected for the Department of Justice/FTC antitrust review of the Verizon acquisition of MCI. 225 attorneys, representing Verizon at a cost of $13.6 million, conducted the original review of the documents. After the Verizon review, 176,000 items were produced to DOJ.

From the original collection Roitblat selected a random sample of 5,000 documents that they had reviewed by 2 human review teams from two volunteer litigation support firms. The researchers also used two volunteer litigation support firms to do a computer software categorization and selection of responsive documents. As the researchers describe in statistical detail in their paper, the differences between the original human review vs., the two human re-reviews vs. the 2 machine categorizations were not statistically significant. Based on these results, the authors conclude that machine categorization is at least as accurate as human review and thus is “reasonable” and compliant under the Federal Rules of Civil Procedure.

The Roitblat, et al study is a significant empirical advance for advocates of the use of machine classification for eDiscovery, but as I said above, it has implications beyond those set forth in the paper:

Vendors and Software were not critical factors

Interestingly, both litigation support vendors and both machine classification engines achieved similar results. While every vendor will try to convince you of their ‘unique’ capabilities, it appears that any of the leading litigation support vendors and any of the widely used machine classification tools will do an acceptable job. This should not be too surprising; most vendors use variations of the same families of algorithms in their classification engines and most of these algorithms have the same academic roots. (Indeed, a recent article points out that most eDiscovery engines use the same OEM and open source components) Where the engines will vary most (when using the same family of algorithm such as two different implementation of Bayesian logic) is in their efficiency and speed. Where vendors will vary most is in the overall quality and skills they bring to the management of the project.

Two engines might be better than one

In my earlier Jeopardy blog post, I noted that one of the advantages of open standards for text mining tools is they enable one to combine different text mining and classification tools. The Roitblat study points to the potential value of doing this. In the good old days of data entry shops, before Optical Character Recognition, vendors widely used a technique called ‘double keying’. Two operators would key the same information, if both versions agreed, the chance of both making the same keying error were so remote that not further review was necessary. The double keying principle can be applied to text classification as well. Run the same document through two different engines, if they both agree, no further review is necessary. Only on those documents on which they do not agree, do they need higher levels of human review. Vendor engines that conform to standards such as UIMA or GATE simplify the task of creating a voting algorithm among multiple classification engines or between two different types of algorithms available from the same engine.

Even with machine classification, there is no such thing as fully automated eDiscovery project

A key to accuracy in both the Roitblat study and in the recent NIST TREK studies is the use of an expert ‘adjudication’ review team to assess and normalize differences in classification among the various methods. Given the often subjective judgments that determine categorization, adjudication often works to fine tune the classification engines, yielding better results. Perhaps an even more important reason to have a subset of the documents classified by a skilled human review team is to demonstrate the equivalency of the automated results to the human results, thereby supporting the argument that the software is a reasonable substitute for a completely human review for the particular documents and issues in the immediate case. Finally, in keeping with best practices for achieving quality in eDiscovery (see the Sedona standards on this), there is probably no way around conducting a human review on a random, statistically significant sample.

Not only are computers better than humans at document classification, they are less expensive and more efficient

Roitblat noted that the original attorney review for the 1.6 million documents was $13.6 million or $8.50 per document. The cost of automated document classification continues to drop, I estimate that processing and classification from a reputable vendor is now in the range of $700-$1,000 per gigabyte, so in the Verizon case the total cost for the 1.3 Terabyte collection would have been in the range of $4-$6 per document. In addition to a lower cost per document, automated review takes far less time. The original human review of the Verizon documents took the 225 attorneys 4 months to conduct. An automated review, even with time for training the engine, appropriate levels of human adjudication and quality control would likely have taken no more than 2 months.