arxiv:2510.19365

The Massive Legal Embedding Benchmark (MLEB)

Published on Oct 22

· Submitted by

Umar Butler on Oct 24

Isaacus

Upvote

Authors:

Umar Butler ,

Abdur-Rahman Butler ,

Adrian Lucas Malec

Abstract

MLEB is the largest open-source benchmark for legal information retrieval, encompassing multiple jurisdictions, document types, and task types.

AI-generated summary

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.

View arXiv page View PDF Project page GitHub 21 Add to collection

Community

umarbutler

Paper author Paper submitter 6 days ago

Hey all,
This is my first-ever paper! In it, we present the Massive Legal Embedding Benchmark (MLEB), a new open-source benchmark for legal information retrieval. It consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering).

This paper documents our methodology in creating MLEB as well as our findings. We've openly released our datasets here on Hugging Face and have made our code available on GitHub: https://github.com/isaacus-dev/mleb