Array Case Studies

Using aiR for Review as a 2nd-level QC measure 

This case began with 6,000 documents that had been put through a first-pass review process and needed evaluating for privilege and commercial sensitivity in time for a disclosure deadline in 7 days. It was anticipated that of the 6,000 documents, a reasonable number were likely to have been marked as relevant overcautiously. It was therefore important to ascertain the accuracy of the first-level review in order to spare second-level reviewers the time it would take to analyse irrelevant documents for privilege. It was decided that aiR for Review would be used as a QC tool to assess the accuracy of the first-pass review and free up reviewer time to focus on reviewing for privilege. Any overturns produced by the algorithm were to be checked, but in general accepted as they were expected. It was also decided that any documents ranking 2 or below would only be reviewed for privilege if there was time remaining, otherwise they would not be disclosed.

The Process and the Results

Step 1: Prompt Development

We worked with our clients to transform their review protocol into a prompt and tested it on a batch of 50 documents that the project lead (Subject Matter Expert) had already reviewed. We then completed 5 iterations of testing, deciding to include the 1st level issue tags as part of the prompt halfway through the development process.

Step 2: Data Culling

While we were engineering this prompt, we also ran Relativity’s ECA tools over the 6,000 document dataset, looking for any duplicates and textual-near duplicates that we might automatically remove from the data set. This resulted in the culling of 1,439 documents.

Step 3: Complete the 1st -Level QC Review

As all of these documents were going to be checked by a reviewer, there was no need to run a validation-stage test. As such, once the prompt criteria successfully coded a batch of 50 test documents with no concerning conflicts, we ran aiR for Review over the remaining 4,561 documents.

Results

aiR for Review was able to review 4, 561 documents for relevance in just under 8 hours.

Among those documents the algorithm located 205 conflicts, and 1,152 documents with a relevancy score of 2 or below, meaning 1,357 documents were able to be removed from the 2nd-level privilege review queue.

This resulted in 3,204 documents being passed on to the 2nd-level privilege review.

With a privilege review rate of 25 documents an hour per reviewer and 3 reviewers, we were securely on-track to meet our clients disclosure deadline.

Note: an extension of a week was granted and as such, all documents ranked at level 2 relevance were also included in the final privilege review.

Given that the industry standard review rate for relevancy review is 60 documents an hour, the entire deduplication and aiR for Review QC process saved our client 90+ hours of review time. This time was instead spent on complex privilege review, enabling them to keep costs low and confidently meet their deadline.

Interested in how aiR for Review could improve your efficiency? Reach out to one of our aiR experts today!