A SCOTUS Cite Checker Is Mostly a User Interface Problem
What do we want of a cite checker?
- It should use an authoritative source,
- It should detect any difference at the character level
- For quotations that are not character for character it should distinguish typos from paraphrases and fabrications
- The volume, beginning page and pin page should be right
The source
The Supreme Court decisions are open source. All of the bound volumes, preliminary prints and slip opinions can be downloaded by script in a few hours. The collection can be kept up to date automatically. This is ancient technology. They are in pdf format with Government Printing Office optical character recognition (OCR) text layer, so the page images of the bound volumes are authoritative and the text is only as good as the OCR.
The approximate 400,000 pages can stored in a database, one printed page per row in both image and OCR text and the pages can “chunked” into approximately 1.3 million 350-word passages and each chunk embedded in a 384-dimensional vector in a local model, which takes a approximately a day for the entire corpus. This is a one time setup. The database can an Hierarchical Navigable Small World (HNSW) index, which enables semantic search over the entire corpus in addition to traditional text search.
This back end provides the ability to locate cases by citation, exact quotation or concept, which has its own attractions, and doing cite checking on top is just the added bonus.
Given a citation, like 477 U.S. 317, (Celotex Corp. v. Catrett, a famous 1986 summary judgment case) a script caches a copy of the PDF for the case for local viewing of the image.
Consider the following citations:
1 Exact quote, correct pincite
In our view, the plain language of Rule 56(c) mandates the entry of summary judgment, after adequate time for discovery and upon motion, against a party who fails to make a showing sufficient to establish the existence of an element essential to that party’s case, and on which that party will bear the burden of proof at trial. 477 U.S. 317 at 322
2 Exact quote, incorrect pincite
In our view, the plain language of Rule 56(c) mandates the entry of summary judgment, after adequate time for discovery and upon motion, against a party who fails to make a showing sufficient to establish the existence of an element essential to that party’s case, and on which that party will bear the burden of proof at trial. 477 U.S. 317 at 323
3 Inexact quote, correct pincite
In our view, the plain language of Rule 56 c mandates the entry of summary judgment, after adequate time for discovery and upon motion, against a party who fails to make a showing sufficient to establish the existance of an element essential to that party’s case, and on which that party will bear the burden of proof at trial. 477 U.S. 317 at 322
3 Paraphrase, correct pincite
In our view, the clear mandate of Rule 56(c) requires immediate entry of summary judgment without adequate time for discovery against any party who fails to prove their entire case before trial. 477 U.S. 317 at 322
4 Fabricated quote, incorrect pincite
Rule 56(c) was specifically enacted by Congress to eliminate frivolous toxic tort litigation prior to the empaneling of a jury. 477 U.S. 317 at 327
A rough web interface checks these.
- Correct quotation and pincite confirmed
- Correct quotation with bad page reference flagged
- Incorrect quotation flagged and explained: p. 322: page also has “plain”; “56 c” for “56(c)”; “existance” for “existence”
- Paraphrase is identified and scored as to similarity, which allows considering using it as a paraphrase and the page number is confirmed
- The fabrication is given a semantic similarity score
Here is the process in plain terms.
A script does the lookup.
- It finds the case from the cite.
- It looks for the quotation word for word on the claimed pages.
- If that fails, it finds the closest passage in the case. That’s the spot the quotation most likely came from.
Then we ask the Jev classifier model one question. We hand it two things, the quotation and that passage, and ask: How faithful is the quotation to the passage? It picks from four possible answers we give it:
| Answer | Meaning |
| exact | Same words, spelled and ordered the same |
| inexact | Clearly the same sentence, with small slips: a misspelling, a dropped bracket, a word or two changed |
| misquote | About the same point, but the wording is substantially different or the meaning is changed, overstated or reversed |
| fabricated | The passage doesn’t say or support what the quotation says |
Jev doesn’t write an explanation. It gives back a probability for each of the four answers, such as “inexact 80%, misquote 15%”.
Then the local script finishes the job.
- It merges Jev’s “exact” into “inexact”, because the script already knows the quotation wasn’t an exact match.
- It labels the row “uncertain” if the top answer is under 60% or leads the runner-up by less than 15 points.
- It decides whether the pincite is right by checking which page the passage is on. Jev never sees the page number.
In short, Jev judges one thing: how well the quotation’s meaning and wording match the passage. The local script finds the passage and handles everything else.
The token cost to do this is much less than a cent. No other AI is involved.
What more is needed
First, we would want more robust handling of all the varieties of citation, beginning with the Id.s. This so far isn’t a full bluebooker. Second, and the reason I say that it’s a user interface matter is that we’d like something like spell check to first detect, and then check from within an editable document. An initial step would be fairly straightforward, which would be to highlight the quotation and pincite and send the highlighted text to a page like show above.
Extension
The federal circuit courts of appeal and district courts don’t have a comparable open source for authoritative opinions from which to assemble a local store. But for slip opinions, the demo script already can do the checking against a pdf download.