Retrievers
Build search with bm25, vectorSearch, hybrid and withRerank. Every one of them is a Retriever, so they nest freely.
Every retriever has one method:
retrieve(query: string, options: { topK: number; abortSignal?: AbortSignal }): Promise<{ hits: Hit[] }>
Hits are sorted by score, highest first, with unique ids and at most topK of them.
bm25
In-memory keyword search with k1 = 1.2 and b = 0.75 (Lucene defaults).
const index = bm25();
await index.add([{ id: "a", text: "How to reset your password" }]);
await index.remove(["a"]);
The default tokenizer lowercases and splits on anything that is not a letter or digit. Pass tokenize for stemming or other languages:
const index = bm25({ tokenize: (text) => myStemmer(defaultTokenize(text)) });
vectorSearch
Embeds with an Embedder and stores vectors in a VectorStore. add embeds documents in batches; retrieve embeds the query.
const index = vectorSearch({ embedder, store: memoryVectorStore() });
memoryVectorStore() does exact cosine search. For production, use a store adapter.
hybrid
Runs retrievers in parallel and merges their rankings with reciprocal rank fusion (RRF): each hit scores weight / (60 + rank) per list. Ranks are used instead of scores because scores from different retrievers are not comparable.
const retriever = hybrid({
retrievers: [lexical, semantic],
weights: [0.5, 1], // optional, default 1 each
});
Each input is asked for max(topK, 50) hits.
withRerank
Pulls max(topK, 50) candidates, reads their text from a DocumentStore, and reorders them with a Reranker.
const retriever = withRerank({
retriever: hybrid({ retrievers: [lexical, semantic] }),
reranker,
documents: memoryDocumentStore(docs),
});
Candidates missing from the document store are dropped.
Your own retriever
Anything that returns ranked ids is a retriever:
import type { Retriever } from "retrieval-sdk";
const siteSearch: Retriever = {
async retrieve(query, { topK, abortSignal }) {
const res = await fetch(
`https://search.example.com?q=${encodeURIComponent(query)}&n=${topK}`,
{
signal: abortSignal,
}
);
return { hits: await res.json() };
},
};