Search API use cases
From a single product catalog to a web-scale news index, the same Search API — often paired with the Crawler API — covers a wide range of search products.
Below is a tour of common use cases teams build on SeekStorm. Tap a card to jump to its full description.
Crawl only the pages that match your focus, per job.
Aggregate sources into one real-time index.
Build your own web search from crawled sites.
Monitor and index official gazettes in real time.
Track auctions, bids, and registrations as they publish.
Pull names, dates, and places from raw text at scale.
Structured data from webpages, no code required.
Continuous crawling for trend and threat intelligence.
Collect and search public sources at web scale.
Every use case, in depth
The cards above link here — each heading below is a permalink, so feel free to bookmark or share a single use case.
Content & site search
Full-text document search
The Search API offers web-scale, real-time, full-text, instant search for your documents. Even with millions or billions of documents spread across different locations and thousands of concurrent users, the latest changes are reflected instantly. SeekStorm delivers 5–10x the speed of industry-standard solutions like Lucene, and around 30x more queries and documents per dollar spent than comparable SaaS offerings.
A document is any valid JSON object with any number of fields, where each key is a string and each value is a valid JSON type — string, number, array, boolean, or object. For every field, you choose whether it's stored, indexed, and included in the search results. See it in the live demo.
Product search
Rich product results help attract buyers while they're searching a catalog, and real-time indexing keeps product data fresh so customers always find current, relevant items. Spelling correction, auto-suggestion, query completion, and instant search all smooth the search experience, while faceted search lets shoppers narrow results by title, URL, domain, author, or category — without wading through irrelevant matches.
Site search
A site search can be added to your own website, or to any other site you have the rights to make searchable, even one whose owner never built search for it — and be up and running within minutes. If the result is meant to be publicly available, make sure you own or have obtained the necessary rights first.
Wikipedia search
The entire English Wikipedia — more than 6 million pages — can be indexed within minutes by feeding a Wikipedia JSON dump into the Search API; crawling it page by page with the Crawler API instead would be far slower. The public demo also shows implicit phrase search in action: searching for "rolling stones" or "the who" ranks results higher when the terms appear together and in the order given, without requiring quotes — including for phrases built entirely from extremely common words ("stop words") that many search engines exclude from indexing altogether. The same demo doubles as a showcase for automatic query suggestion, query completion, and instant search, displaying results before a user finishes typing or presses enter. Try it in the live demo.
Crawling & aggregation
Focused crawler
A focused crawler collects only the web pages that satisfy a chosen property — a domain, a URL prefix — by carefully prioritizing the crawl frontier and link selection. Each website gets its own crawl job, and additional URL path filters narrow it further; excluding pages outside your area of interest improves result relevance, indexing speed, query speed, and index size.
Pages from different websites and crawl jobs can be indexed together or kept in separate indices, and the aggregated results are searchable in real time even while a crawl job is still running. A configurable re-crawl period keeps the index up to date. SeekStorm crawling is intended for consensual crawling only — you're responsible for owning or obtaining the rights and licenses needed to crawl the content and for obeying every restriction the site owner imposes; SeekStorm doesn't provide tools to get around those restrictions.
Custom news search
Combining the Crawler API with the Search API lets you build a custom news search: define a separate crawl job per news source, index the aggregated pages into a single index, and search them in real time — even while crawling is still underway. A configurable re-crawl period keeps coverage current, and you remain responsible for owning or obtaining the necessary rights to the content you index.
Custom web search
The same pattern — a crawl job per website, one aggregated index, a configurable re-crawl period — lets you stand up a custom web search API of your own, searchable in real time as crawling proceeds. As with any crawl job, you're responsible for owning or obtaining the necessary rights to the content according to your local law.
Scrape public gazettes
Official, government, federal, law, health, and stock-exchange gazettes all publish on public websites. Pairing the Crawler API with the Search API lets you monitor, scrape, and aggregate that information and make it searchable in real time.
Scraping public listings
Public listings for auctions, bids, and company registrations can likewise be monitored, crawled, scraped, aggregated, and made searchable in real time by combining the Crawler API and Search API.
Data & intelligence
Semantic entity extraction
Extracting names, product names, trademarks, dates, and geographical entities from unstructured text at scale starts with fetching the source pages, stripping the HTML markup, and pulling out the raw text. Once converted into structured JSON, stored, and indexed, the documents are ready for downstream processing with deep learning, semantic analysis, or other NLP tools.
Data extraction for deep learning
SeekStorm automatically extracts structured data — as defined by schema.org and the Open Graph protocol — from any number of webpages at scale, without writing code. Unstructured websites become structured data, ready to be consumed by data science, data mining, AI, machine learning, and deep-learning pipelines.
Web monitoring
Trend analysis, competitive intelligence, trademark-infringement monitoring, stock prediction, threat intelligence, and alerting all depend on continuously observing domains of interest through crawling, text extraction, storage, and full-text indexing — the essential groundwork before any further post-processing of the raw data.
Open-Source Intelligence (OSINT)
SeekStorm provides high-performance infrastructure for collecting, indexing, and searching publicly available data for intelligence purposes. Its focused crawler can target and monitor the areas of the internet where relevant information is expected to appear first, and its real-time, web-scale search makes it possible to search an index of billions of documents within milliseconds, from thousands of concurrent users.
See also: Search API features, Architecture, Live demo, API documentation.