Skip to content
Retrieval SDK
Esc
↑↓navigate↵open⌘Jpreview
On this page

Evaluation

Score any retriever with NDCG, recall and precision at several cutoffs, or plug in your own metric function.

evaluate runs each query once and scores the results with every metric at every cutoff.

import { evaluate, ndcg, precision, recall } from "retrieval-sdk";

const report = await evaluate({
  retriever,
  queries: [
    {
      id: "q1",
      text: "reset my password",
      qrels: new Map([
        ["doc-7", 2],
        ["doc-3", 1],
      ]),
    },
    { id: "q2", text: "download an invoice", qrels: new Map([["doc-12", 1]]) },
  ],
  ks: [5, 10],
  metrics: { ndcg, recall, precision },
});

report.mean; // { "ndcg@5": …, "ndcg@10": …, "recall@5": …, … }
report.perQuery; // { q1: { "ndcg@5": … }, q2: … }
report.skipped; // queries with no relevant document

qrels maps a document id to a relevance grade. A grade of 0 or a missing id means not relevant.

Definitions

  • precision@k: relevant documents in the top k, divided by k.
  • recall@k: relevant documents in the top k, divided by all relevant documents.
  • ndcg@k: DCG divided by the best possible DCG, with linear gain grade / log2(rank + 1), like trec_eval.

Queries without any relevant document are skipped, like in trec_eval, so they don’t pull the mean toward 0.

Your own metric

A metric is a plain function:

import type { Metric } from "retrieval-sdk";

const mrr: Metric = (ranked, qrels, k) => {
  const i = ranked.slice(0, k).findIndex((id) => (qrels.get(id) ?? 0) > 0);
  return i === -1 ? 0 : 1 / (i + 1);
};

Was this page helpful?