---
title: Evaluation
description: Score any retriever with NDCG, recall and precision at several cutoffs, or plug in your own metric function.
sidebar:
  order: 5
---

`evaluate` runs each query once and scores the results with every metric at every cutoff.

```ts
import { evaluate, ndcg, precision, recall } from "retrieval-sdk";

const report = await evaluate({
  retriever,
  queries: [
    {
      id: "q1",
      text: "reset my password",
      qrels: new Map([
        ["doc-7", 2],
        ["doc-3", 1],
      ]),
    },
    { id: "q2", text: "download an invoice", qrels: new Map([["doc-12", 1]]) },
  ],
  ks: [5, 10],
  metrics: { ndcg, recall, precision },
});

report.mean; // { "ndcg@5": …, "ndcg@10": …, "recall@5": …, … }
report.perQuery; // { q1: { "ndcg@5": … }, q2: … }
report.skipped; // queries with no relevant document
```

`qrels` maps a document id to a relevance grade. A grade of 0 or a missing id means not relevant.

## Definitions

- `precision@k`: relevant documents in the top k, divided by k.
- `recall@k`: relevant documents in the top k, divided by all relevant documents.
- `ndcg@k`: DCG divided by the best possible DCG, with linear gain `grade / log2(rank + 1)`, like trec_eval.

Queries without any relevant document are skipped, like in trec_eval, so they don't pull the mean toward 0.

## Your own metric

A metric is a plain function:

```ts
import type { Metric } from "retrieval-sdk";

const mrr: Metric = (ranked, qrels, k) => {
  const i = ranked.slice(0, k).findIndex((id) => (qrels.get(id) ?? 0) > 0);
  return i === -1 ? 0 : 1 / (i + 1);
};
```
