tech
Introducing Search Toolkit
Today, we're releasing Search Toolkit in public preview. Search Toolkit is a composable framework for building production search pipelines for AI applications. We built it because teams building search infrastructure still spend too much engineering time on plumbing. Most stitch together separate tools for ingestion, retrieval, and evaluation, each with its own interface and its own assumptions about data. Search Toolkit brings all three into a single framework with a shared interface, so teams spend their time improving search quality instead of maintaining integrations. Search Toolkit is open source and runs wherever your infrastructure does. Cloud, on-premises, edge.
TL;DR
- Search Toolkit is a composable framework for building production search pipelines for AI applications, now in public preview.
- It integrates ingestion, retrieval, and evaluation into a single framework with a shared interface, reducing engineering time spent on plumbing.
- The framework is open-source and can run on cloud, on-premises, or edge infrastructure.
- It addresses challenges in enterprise search, RAG (Retrieval-Augmented Generation) systems, and domain-specific retrieval.
- Key features include configurable ingestion pipelines, multiple retrieval configurations (BM25, dense embedding, hybrid), and built-in evaluation metrics (recall, precision, MRR, NDCG).
- A starter app template is available for quick setup and experimentation.