Technology & data
Our own index. Our own engine. Every number with a date.
Semantil does not resell other companies' data. We crawl the web ourselves, keep 1.17B websites on record, and search an index of 130.84M websites in under 100 ms on an engine we built for exactly this job. This page explains where the data comes from, how the engine works, what is measured and what is estimated, and what we deliberately do not collect.
1 · Where the data comes from
Crawled by us since 2017, archives back to 2013.
Semantil runs its own crawl of public websites since 2017. For the years before, we use public web archives from 2013, so every website carries the year we first saw it. We read what a website shows to any visitor: text, technologies, platforms, payment and shipping options, countries served. We do not log in or read behind paywalls, and we do not build databases of people.
| Websites observed | 1.17B | Every website (domain) ever checked, live or not. |
|---|---|---|
| Searchable websites | 130.84M | Websites you can search right now. As of 2026-10-01. |
| Verified online stores | 8.49M | Websites confirmed to sell online; 8.48M re-visited in the last 30 days. |
| Countries · languages | 250 · 69 | We know the country for 73% of websites (95.9M). Most: United States, Germany, United Kingdom. We know the language for 92% of websites. About half are English. As of 2026-10-01. |
| First seen | 2013 → | Public archives from 2013, own crawl since 2017. Not a launch date. |
2 · How the engine works
Built where a search index meets an analytical database.
Market questions need two things at once: find every matching website by meaning and text, and count how the result splits by country, segment, platform or provider. Off-the-shelf search engines do the first, analytical databases the second, and gluing them together is slow and approximate. Semantil's engine does both in one query, with exact counts, not samples, over hundreds of millions of websites.
| Typical answer | <100 ms | What we promise for any question across the full index. |
|---|---|---|
| Text search, median | 8 ms | Measured on the live engine, before the browser and network add their time. |
| Public benchmark | 4 engines | Same data, same server: Elasticsearch, OpenSearch, Tantivy, Typesense. Read the benchmark → |
3 · What is measured and what is estimated
We tell you which is which.
Measured
Seen on the website itself, dated.
- Technologies (16,863 recognized), shop platform, payment providers, shipping carriers
- Countries a store appears to sell to
- Company or VAT register match (1.35M stores)
- Year first seen, date last checked
Estimated
Estimated by our models and always marked as estimates.
- Business segment (30K+ segments) and product category
- Store size band, from small to top 1%
- Signs of sales
Seeing something on a page is not proof it is used. Counts are what is in our index, not the whole world.
4 · Evidence and freshness
Every record tells you where it came from and when.
You should not have to trust an average update rate. Every record carries the date we last checked it; contact details carry the page and the sentence they came from; every company in a market answer carries the reason it matched. The whole index gets a new data release about monthly (latest 2026-09-20), technology scans run continuously, and 100% of verified stores are re-visited within 30 days.
| Dates | every field | latest_scan_date, tech_timestamp, contact_*_observed_at, website_year and website_month. |
|---|---|---|
| Source | contact details | contact_email_source_url and contact_email_source_context: the page and the sentence. Same for phone. |
| Match reason | every company in a market | Matched by words on its site, by business segment, or both; the qualification you asked for is confirmed or marked as not observed. |
| Estimates | labelled | Segment, size band and category are marked as estimates, with a confidence where we have one (merchant_size_confidence, ecommerce_ai_probability). |
5 · What we do not collect
Companies, not people.
Semantil indexes companies, not people. A company record can include up to one company email on its own domain, such as info@, sales@ or kontakt@, and one phone number published on the company’s website, when available. Each contact includes its source page and observation date; surrounding page text is included when available. Personal mailboxes and personal names are excluded. Contact data is rebuilt with each data release. For named people and their work contact details, use a provider that specializes in contact data.
6 · How to use it
Browser, API, and your AI agent.
In the browser
Ask Semantil a market question, or open Catalog and filter by hand. Results save as company lists you keep working on.
By API and MCP
REST API and an MCP server with eight tools: discover_market, search, lookalike, breakdown, compare_cohorts, discover_segments, enrich, account. Same counts, same dates, same definitions as on this page. Developers →
7 · Who is behind it
Web and commerce data operations since 2017.
Semantil grew out of ShopRank, an online-store intelligence service run since 2017. The same team, crawl and engine power both. Questions about data, methods or access: contact us.