Skip to content
Retrieval SDK
Esc
↑↓navigate↵open⌘Jpreview
On this page

Retrievers

Build search with bm25, vectorSearch, hybrid and withRerank. Every one of them is a Retriever, so they nest freely.

Every retriever has one method:

retrieve(query: string, options: { topK: number; abortSignal?: AbortSignal }): Promise<{ hits: Hit[] }>

Hits are sorted by score, highest first, with unique ids and at most topK of them.

bm25

In-memory keyword search with k1 = 1.2 and b = 0.75 (Lucene defaults).

const index = bm25();
await index.add([{ id: "a", text: "How to reset your password" }]);
await index.remove(["a"]);

The default tokenizer lowercases and splits on anything that is not a letter or digit. Pass tokenize for stemming or other languages:

const index = bm25({ tokenize: (text) => myStemmer(defaultTokenize(text)) });

vectorSearch

Embeds with an Embedder and stores vectors in a VectorStore. add embeds documents in batches; retrieve embeds the query.

const index = vectorSearch({ embedder, store: memoryVectorStore() });

memoryVectorStore() does exact cosine search. For production, use a store adapter.

hybrid

Runs retrievers in parallel and merges their rankings with reciprocal rank fusion (RRF): each hit scores weight / (60 + rank) per list. Ranks are used instead of scores because scores from different retrievers are not comparable.

const retriever = hybrid({
  retrievers: [lexical, semantic],
  weights: [0.5, 1], // optional, default 1 each
});

Each input is asked for max(topK, 50) hits.

withRerank

Pulls max(topK, 50) candidates, reads their text from a DocumentStore, and reorders them with a Reranker.

const retriever = withRerank({
  retriever: hybrid({ retrievers: [lexical, semantic] }),
  reranker,
  documents: memoryDocumentStore(docs),
});

Candidates missing from the document store are dropped.

Your own retriever

Anything that returns ranked ids is a retriever:

import type { Retriever } from "retrieval-sdk";

const siteSearch: Retriever = {
  async retrieve(query, { topK, abortSignal }) {
    const res = await fetch(
      `https://search.example.com?q=${encodeURIComponent(query)}&n=${topK}`,
      {
        signal: abortSignal,
      }
    );
    return { hits: await res.json() };
  },
};

Was this page helpful?