Consensus: How pro bono lawyers help us maintain human oversight over our AI tools

23 Jan 2025

At Justice Connect, we leverage technology where it makes sense, to improve the experience of getting and giving legal help.

Artificial Intelligence in the legal sector is still a relatively new frontier. While we’ve seen incredible advances, we know there’s always room for improvement. When implemented and monitored ethically, AI presents a huge opportunity to improve the access to justice ecosystem.

Our award-winning natural language processing model plays a key role in helping people navigate our services and tools. By describing their legal problem in their own words, our AI model interprets and diagnoses areas of law which may be relevant to their situation. In improving the experience of applying for legal help, our AI model contributed to a 48% drop in the number of people who started applying but didn’t make it to the end of the form – connecting more people with the support they need. And, our AI powers critical tools like our Intake Tool, Pro Bono Portal, self-help resources and more. These tools support thousands of people seeking legal help, and the lawyers who assist them.

But we know that trust in AI depends on its performance being carefully monitored and improved over time. Which is why we’re constantly exploring ways to ensure our AI-driven tools are both innovative and reliable.

Our latest measure on monitoring our AI is a new layer of testing called “Consensus” – our second game designed to enhance the quality of our AI, following the success of our “Tagging Game” (TAG).

What is Consensus?

Consensus is more than just a tool or process; it’s an interactive game we’ve designed to test our AI model. In this game, legal professionals review real-world scenarios and compare the AI’s outputs to their expert judgments. The goal? To ensure that our AI provides accurate, helpful, and contextually appropriate responses every time it’s used.

How Consensus works

  1. Scenario Generation: We create realistic scenarios based on actual user queries and inputs.

  2. AI Response Testing: The AI provides its outputs, which are then presented to legal professionals.

  3. Expert Review: Participants rate the AI’s responses, highlighting areas of agreement or concern.

  4. Feedback Integration: The data collected is used to refine and retrain the AI model, ensuring continuous improvement.

By framing this as a “game,” we’ve made the process engaging for participants while still collecting meaningful data. Each session challenges professionals to evaluate the AI’s decision-making and, where necessary, suggest improvements. This ensures that the AI doesn’t just meet baseline standards, but exceeds them by aligning closely with expert opinions.

Consensus helps us keep our AI model reliable by:

  • Raising Quality Standards: By engaging legal professionals, we’re able to fine-tune our AI model to deliver consistently accurate and relevant outputs.

  • Building Ethical AI: Responsible AI development requires transparency and accountability. Consensus incorporates human oversight to ensure the AI operates fairly and equitably.

  • Strengthening Trust: Knowing that skilled experts rigorously test and validate the AI gives users confidence in its recommendations.

  • Enhancing Impact: By addressing gaps and refining outputs, Consensus helps us create tools that better serve individuals and organisations.

Ethical AI in action

Consensus reflects our commitment to ethical AI. By involving legal professionals in the testing process, we ensure that the AI’s decision-making aligns with professional standards and user needs. This process also helps us identify and address any unintended biases, ensuring that our tools promote fairness and accessibility for all users.

Harnessing the power of pro bono

At Justice Connect, pro bono is at the heart of all we do. Our impact would not be possible without the dedication of the over 10,000 pro bono lawyers in our network. We are committed to finding new and innovative ways to effectively harness the capacity of lawyers who want to give back to their community. Our member firms collaborate with us address pervasive justice gaps in our communities, assisting with our pro bono referrals, advising self-represented litigants, reviewing our self-help resources, and training our AI model.

TAG and Consensus provide a unique opportunity for pro bono lawyers to apply their expertise, while contributing to maintaining ethical AI and improving access to justice.

The impact so far

Since introducing Consensus in November 2024, we’ve:

  • Identified key areas where the AI needed additional training, such as handling nuanced legal queries.

  • Improved the accuracy and reliability of our model’s outputs by incorporating direct feedback from experts.

  • Built on the success of our first game, TAG, to create a robust framework for ongoing AI improvement.

What’s next for Consensus and TAG?

We’re excited about the potential for Consensus and TAG to evolve. In the future, we aim to:

  • Expand the pool of legal professionals involved to bring even more diverse perspectives.

  • Incorporate gamified elements to make participation even more engaging.

  • Explore how this approach could be adapted to test other AI-driven tools in our suite.

Consensus is just one example of how Justice Connect is leading the way in ethical, user-focused AI development. By bringing human expertise into the process, we’re ensuring that our tools remain trustworthy and effective, helping us achieve our mission of improving access to legal support for everyone.

If you’d like to learn more about Consensus or how you can get involved, feel free to reach out. Together, we can continue to push the boundaries of what’s possible with AI in the legal sector.

Learn more about our Innovation work.