Computer Beats Lawyers in AI Challenge
When it comes to predicting outcomes in law cases, it seems computers are better than lawyers.
That’s one way to interpret the results of a recent AI vs. Attorney challenge. In that matchup – which you can read about here – the software program CaseCruncher Alpha proved 25 percent more accurate at evaluating insurance claims than 100 flesh-and-blood lawyers.
Looks like a clear victory, at least on the surface. But digging a bit deeper reveals a different story.
For one thing, the competition was staged by CaseCrunch, a legal technology company that also happens to sell CaseCruncher Alpha. Can anyone say conflict of interest?
For another, the subject matter was an area of insurance law (Payment Protection Insurance claims) that was outside the expertise of some of the lawyers who participated. CaseCruncher Alpha, by contrast, had been programmed with a trove of PPI data, which gave it a big advantage.
But a win is a win. And the takeaway is clear: machine-based decision making is a real – and growing – thing that lawyers might as well get used to.
Man versus Machine
CaseCrunch is a tech startup run by four Cambridge law students, three of whom are still undergraduates. The company builds systems that predict legal decisions using AI technology.
To showcase its CaseCruncher Alpha program, CaseCrunch devised a nifty little contest. It recruited more than 100 attorneys from top law firms (DLA Piper, Allen & Overy) to take on the computer in predicting the results of approximately 800 actual, historic PPI claims.
“[T]his competition was covered by BBC and elsewhere as an epic man-versus-machine matchup, like the legal world’s Kasparov versus Deep Blue, a pair of six-game chess matches in the 1990s between the world’s reigning chess champion, Garry Kasparov, and IBM’s supercomputer,” writes Jason Tashea in the ABA Journal.
When the dust settled, the software came out on top, forecasting the actual outcome with 87 percent accuracy. The lawyers were only 62 percent accurate.
But more than one observer noted that the playing field was somewhat less than level. After all, CaseCruncher Alpha had been given all that information in advance. The poor lawyers came in with only their puny brains.
And even CaseCrunch cautioned against reading too much into the event.
“These results do not mean that machines are generally better at predicting outcomes than human lawyers,” a company official said in The American Lawyer. “These results show that if the question is defined precisely, machines are able to compete with and sometimes outperform human lawyers.”
Apples and Oranges
This isn’t the first time computers have vanquished lawyers in staged competitions.
Back in 2004, a group of law professors designed a “classification tree model” that proved better at predicting U.S. Supreme Court results than human experts. More recently, an algorithm correctly “decided” 79 percent of actual cases that had been brought before the European Court of Human Rights, although no lawyers were involved in that test.
But firms might want to wait a bit before firing their lawyers and replacing them with laptops.
That’s because no computer – not even the Demon Speed Silverdraft Supercomputing Workstation – has the experience of trained counsel. Nor does it possess essential qualities like empathy, courage, integrity and self-awareness.
These “challenges” are entertaining. But they compare apples and oranges. The computer is better at doing some things. Lawyers are better at other things. Programs like CaseCruncher Alpha are resources – not replacements – for lawyers.
Meanwhile, we’re left to wonder how a top-flight, highly-paid attorney has the time and inclination to analyze 800 insurance claims just for fun. That, to me, is the real question.
Sources:
- Legal IT Insider https://www.legaltechnology.com/latest-news/machine-beats-man-in-casecrunch-lawyer-challenge/
- ABA Journal http://www.abajournal.com/news/article/artificial_intelligence_software_outperforms_lawyers_without_subject_matter/
- Peer J https://peerj.com/articles/cs-93/
- Silverdraft Supercomputing http://www.silverdraft.com/products/demon/
- The American Lawyer https://www.law.com/americanlawyer/sites/americanlawyer/2017/10/31/artificial-intelligence-beats-big-law-partners-in-legal-matchup/?slreturn=20171112121442
- CaseCrunch http://www.case-crunch.com/
