Evaluation
Score any retriever with NDCG, recall and precision at several cutoffs, or plug in your own metric function.
evaluate runs each query once and scores the results with every metric at every cutoff.
import { evaluate, ndcg, precision, recall } from "retrieval-sdk";
const report = await evaluate({
retriever,
queries: [
{
id: "q1",
text: "reset my password",
qrels: new Map([
["doc-7", 2],
["doc-3", 1],
]),
},
{ id: "q2", text: "download an invoice", qrels: new Map([["doc-12", 1]]) },
],
ks: [5, 10],
metrics: { ndcg, recall, precision },
});
report.mean; // { "ndcg@5": …, "ndcg@10": …, "recall@5": …, … }
report.perQuery; // { q1: { "ndcg@5": … }, q2: … }
report.skipped; // queries with no relevant document
qrels maps a document id to a relevance grade. A grade of 0 or a missing id means not relevant.
Definitions
precision@k: relevant documents in the top k, divided by k.recall@k: relevant documents in the top k, divided by all relevant documents.ndcg@k: DCG divided by the best possible DCG, with linear gaingrade / log2(rank + 1), like trec_eval.
Queries without any relevant document are skipped, like in trec_eval, so they don’t pull the mean toward 0.
Your own metric
A metric is a plain function:
import type { Metric } from "retrieval-sdk";
const mrr: Metric = (ranked, qrels, k) => {
const i = ranked.slice(0, k).findIndex((id) => (qrels.get(id) ?? 0) > 0);
return i === -1 ? 0 : 1 / (i + 1);
};