A team of Stanford University researchers has developed an LLM system to cut through bureaucratic red tape.
The LLM—dubbed the System for Statutory Research, or STARA—can help policymakers quickly and cheaply parse voluminous collections of rules to identify laws that are redundant, outdated, or overly onerous. Ultimately, it can make governments more efficient, the researchers say.
San Francisco City Attorney David Chiu’s office recently used STARA to sift through the city’s municipal code and published resolutions—which consist of 27 volumes and nearly 16 million words. Those laws have, over the decades, required different agencies to create around 500 reports for the city to review. The AI analyzed San Francisco’s laws, identifying every city-mandated report and highlighting ones that could be tweaked, combined with similar reports, or zeroed out altogether.
“One of the biggest impediments to regulatory reform is simply having visibility into what existing laws or regulations say,” said Daniel Ho, a Stanford law professor who led the team that created STARA. San Francisco, with its reputation for procedural bloat as well as its proximity to Silicon Valley, seemed like a natural place to test STARA, he said.
Thinking like a lawyer
Historically, combing through federal, state, or city codes using only Boolean searches has been a herculean—and largely inadequate—task. Even the best word or phrase searches can be foiled by opaque legalese. Laws have their own defined terms and often cross reference sections or individual paragraphs nested within very different statutes.
Perhaps most vexingly, laws are dense and infrequently updated.
The U.S. Code—the collection of all federal laws—runs more than 32 million words, and includes 54 distinct titles. There are so many Congressionally mandated reports—official estimates range from 2,500 to 3,359—that Congress lost track of them. In 2020, the Congressional Research Service said it was “unaware of a search method that can obtain an exact accounting of all reports required to be submitted to Congress.”
Similarly, states and cities, including San Francisco, must navigate dense warrens of laws.
The researchers—part of Stanford’s Regulation, Evaluation, Governance Lab (RegLab)—trained the AI to think like a lawyer.
STARA was designed, Ho said, to interpret legal texts holistically, looking out for “statutory trees.” In practice, that means reading various aspects of a law together, not as standalone components. For example, the titles, sections, and statutory definitions of a law—as well as any cross-references to other laws—are understood contextually.
The RegLab team found that this domain-informed interpretation led to dramatic boosts in accuracy and speed.
“Our model system reasons about a code’s provisions in the same way that we teach Statutory Interpretation 101 to law students,” Ho said. “Essentially, students are taught that you have to pay attention to defined terms, to cross references in the code, and to the overall structure of the code.”
As a base model, the researchers used LLaMA-3 70b with NVIDIA A100 Tensor Core GPUs and PyTorch and vLLM. After pre-training, the model was able to accurately identify relevant language within statutes and statutorily mandated reports 94% to 99% of the time. STARA materially outperformed off-the-shelf LLMs that were asked to do the same searches, with its extraction accuracy 2.7x better than a base system, Ho said.
Money and time saver
Perhaps as relevant to cost-conscious lawmakers is STARA’s price-tag and time-saving potential.
A research task that would take two humans eight to 13.5 hours and cost about $3,000 took STARA only 20 minutes to complete and cost about 86 cents.
Ho and his team plan to publish a paper about STARA in the journal Proceedings of the International Conference on AI and Law (ICAIL). In the paper, the RegLab team quotes a San Francisco deputy city attorney saying the model was “outrageously helpful.”
As a result of San Francisco’s work with STARA and Ho’s team, City Attorney Chiu sponsored legislation to modify around one-third of the nearly 500 city-required reports, and eliminate 140 reports entirely.
Ho and his team are currently in discussions with several states and municipalities that are interested in using STARA to help streamline their statutory regimes.
“The city attorney collaborating with us is a powerful illustration of how technology can help cut down on policy sludge,” Ho said.
Read additional coverage of STARA and how the RegLab has deployed similar technology to excise illegal racial covenants in Santa Clara, Calif., deeds.
