Tools for building a "data layer for search and AI traffic"
Click a column heading to sort. The Fit filter groups rows by the first word of the Fit column as written in the source.
On this page
- 1. Sourcing and scraping
- 2. Ingestion / ELT
- 3. Storage and query
- 4. Transformation and data testing
- 5. Orchestration
- 6. Publishing
- 7. Schema.org, llms.txt, agent access
- 8. AI-visibility trackers (measurement of citations, not a ranking lever)
- 9. "Data layer for SEO" competitors and analogues
- 10. Reference pipeline for Anthony (assessment, not a recommendation to buy anything)
Collected 2026-10-25 (Asia/Dubai). Pricing was read from each vendor's live pricing page this session; list prices only, plans change. "n/v" means not verified (page blocked, not rendered, or sales-quoted). "Fit" is my judgement against Anthony's stack as described in the brief (Dubai real estate + Benidorm travel; daily-changing pages; Cloudflare/Vercel/Supabase-style tooling in his connected apps). It is a judgement, not a measured result.
1. Sourcing and scraping
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| Firecrawl | Web scrape/crawl/search API returning clean markdown/JSON, credit-based. | Free tier; Standard $83/month billed annually (page lists "$83/month", other tiers not captured) | High for listings/POI pages and competitor monitoring. Check ToS of each source. | https://www.firecrawl.dev/pricing |
| Apify | Actor marketplace and platform for scrapers, scheduling, storage. | Free $5 monthly credit, $0.2 per compute unit; Starter $19 | High: ready-made scrapers for property portals/travel sites; also runs the Crawlee library. | https://apify.com/pricing |
| Crawlee | Open-source scraping library (JS and Python) handling blocking, proxies, browsers. | Free, open source | High if you want to own scrapers in your own repo. | https://crawlee.dev/ |
| Browserbase | Hosted headless browsers for agents/scrapers. | Rate shown: $0.12/browser hour (plan fees not captured) | Medium: needed only for JS-heavy or login-gated sources. | https://www.browserbase.com/pricing |
| Bright Data | Proxies and scraping infrastructure. | "Free tier pricing available"; paid prices n/v | Medium: for blocked sources; legal risk is on the user. | https://brightdata.com/pricing |
| Dubai Land Department / Dubai Pulse open data | Official registered transactions, rents, projects, etc. Distributed via Dubai Pulse; access reportedly needs an API key. | n/v (the Dubai Pulse fetch failed; the description is from propertyindex.ae) | Highest: the primary source for buydubai.io-type data layers. | https://www.propertyindex.ae/dubai-real-estate-data-platforms (secondary) |
2. Ingestion / ELT
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| dlt (dlthub) | Python library that loads data into warehouses/DBs with schema inference. | dlt Free, Apache 2.0; dltHub platform $12,000/month with a 12-month minimum | High as the free library; the paid platform is far beyond a two-site operation. | https://dlthub.com/pricing |
| Airbyte | Connector-based ELT; Standard is volume-based cloud, Pro is capacity-priced. | Dollar amounts not rendered; n/v | Low-medium: connectors fit SaaS sources, not scraped web data. | https://airbyte.com/pricing |
3. Storage and query
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| Supabase (Postgres) | Hosted Postgres plus API/auth. | Free $0; Pro from $25/month (usage-based extras) | High: already in Anthony's connected apps; good for serving page data. | https://supabase.com/pricing |
| DuckDB | In-process analytical SQL engine. 2.0.0 announced 2026-10-21; MIT-licensed. | Free | High for batch transformation/analysis of Parquet/CSV snapshots. | https://duckdb.org/ |
| MotherDuck | Hosted DuckDB. | Free tier $0; Business $250 per org/month plus usage | Medium: only if DuckDB needs to be shared/served. | https://motherduck.com/product/pricing/ |
| BigQuery | Serverless warehouse. | On-demand analysis $6.25 per TiB (page figure) | Low-medium: overkill for DLD-scale data (the dxbdata demo shows 8,480 transactions). | https://cloud.google.com/bigquery/pricing |
4. Transformation and data testing
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| dbt | SQL transformation framework. Open-source repo exists; cloud tier priced per seat. | Starter $100/user/month (cloud) | High as SQL discipline; open source core covers a small team. License of dbt-core not confirmed from the page I read. | https://www.getdbt.com/pricing ; https://github.com/dbt-labs/dbt-core |
| Pandera | Open-source dataframe validation (Python). | Free | High: cheap schema/range checks before publishing (e.g. price per sqm bounds). | https://pandera.readthedocs.io/ |
| Great Expectations | Data quality tests; GX Cloud has a free Developer option. | Free Developer plan; paid tiers n/v | Medium. | https://greatexpectations.io/pricing/ |
| Soda | Data quality checks and monitoring. | Free tier; a paid tier listed at $750/month (plan name not captured) | Medium. | https://www.soda.io/pricing |
| Cube | Semantic layer/metrics API. | Developer plans $40 and $80 per developer/month (plan names not captured) | Low until there are many consumers of the same metrics. | https://cube.dev/pricing |
5. Orchestration
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| Cloudflare Workers / Workflows | Cron-triggered serverless jobs and durable workflows. | Workers Free plan exists; Paid minimum $5/month per account | High if the sites already sit on Cloudflare. | https://developers.cloudflare.com/workflows/reference/pricing/ ; https://developers.cloudflare.com/workers/platform/pricing/ |
| Dagster | Asset-based orchestrator. | Starter-style $10/month and $100/month tiers on page; pay-as-you-go $0.040/credit | Medium-high for a growing pipeline with lineage. | https://dagster.io/pricing |
| Prefect | Python workflow orchestration. | Hobby free forever; Starter $100/month | Medium. | https://www.prefect.io/pricing |
| Apache Airflow | Open-source scheduler. | Free (self-host); Astronomer managed from about $0.35/hour (deployment unit, page figure) | Low: heavy for two sites. | https://airflow.apache.org/ ; https://www.astronomer.io/pricing/ |
| GitHub Actions cron / Vercel cron | Simplest daily trigger. | Not priced here | High for "daily-changing" at this scale. | (no page fetched) |
6. Publishing
| Tool | What it does | Pricing (verified) | Fit | Source |
|---|---|---|---|---|
| Astro | Content-focused static/SSR framework (v7.4 at time of fetch). Free and open source. | Free | High: ships fast, static HTML with embedded JSON-LD. | https://astro.build/ |
| Vercel | Hosting/deploy. | Hobby $0; Pro $20/month plus usage | High; already used for articulate-ai.work microsites. | https://vercel.com/pricing |
| Cloudflare Workers/Pages Functions | Edge hosting and functions. The pricing page covers Workers Free (100,000 requests/day) and Paid ($5/month minimum). Static asset requests are free and unlimited. | See left | High. | https://developers.cloudflare.com/workers/platform/pricing/ |
| Evidence | BI-as-code (markdown plus SQL to pages/charts). Open-source Evidence Core. | Core free; Studio pricing n/v | Medium: good for report pages/dashboards, not as an SEO page engine. | https://evidence.dev/ |
| Search Console Sitemaps | Sitemaps with accurate <lastmod> are the freshness signal. Google ignores <priority> and <changefreq>. |
Free | Mandatory. | https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap |
7. Schema.org, llms.txt, agent access
| Tool/spec | What it does | Pricing | Fit and evidence | Source |
|---|---|---|---|---|
| schema.org | Vocabulary for JSON-LD/microdata. | Free | Useful for eligible rich results; no demonstrated AI-citation lift (see best-practice.md). | https://schema.org/docs/gs.html |
| Google Rich Results Test | Validates structured data eligibility. | Free | Use as a CI gate for templates. | https://search.google.com/test/rich-results |
| llms.txt (spec) | Proposed markdown index file for agents; v2 of the proposal says thousands of sites publish it and docs platforms generate it automatically. | Free | Google says it does not use it; the SE Ranking study found no correlation with citations. Cheap to add for developer-style docs, do not expect traffic. | https://llmstxt.org/ |
| MCP data endpoints (example: dxbdata.io) | Serves DLD transactions over Model Context Protocol; "8,480 DLD transactions, 131 areas, Daily auto-updated" per its own page. | n/v | An example of a data layer exposed to agents directly; first-party claims unverified. A direct competitor/analogue for buydubai.io. | https://dxbdata.io/start |
8. AI-visibility trackers (measurement of citations, not a ranking lever)
| Tool | What it does | Pricing | Fit | Source |
|---|---|---|---|---|
| Ahrefs Brand Radar | Tracks brand/URL mentions in AI answers; used by Glenn Gabe in the National Today case. | n/v (page now promotes a new "Letaido" product) | Medium-high if Ahrefs is already paid for. | https://ahrefs.com/brand-radar |
| Profound | Enterprise AI-visibility platform; page shows a Trial plan and an Agency Growth plan with 400 credits/month per client workspace. | $ amounts not rendered; n/v | Low-medium: enterprise-oriented. | https://www.tryprofound.com/pricing |
| Peec AI | Prompt-based AI visibility tracking, Starter and Pro tiers. | Prices not rendered; n/v (pricing FAQ fetched) | Medium. | https://peec.ai/pricing |
| Otterly.AI | AI search monitoring. | Pricing page blocked; n/v | Unknown. | https://otterly.ai (blocked) |
| Scrunch, Semrush AI Visibility Toolkit, Athena | Named only in an X post. | Not checked | Unverified. | x-posts.md (2045591192717476030) |
| Google Search Console | Google's own clicks and impressions. Treat as ground truth for Google traffic. | Free | Mandatory. | https://developers.google.com/search/docs/appearance/ai-features |
Caveat on trackers: the GEO paper's "up to 40% visibility" figure (arXiv 2311.09735) comes from a benchmark of academic generative engines, not Google AI Overviews or ChatGPT in production. Prompt-based trackers sample a few prompts, and answers vary run to run, so treat their scores as indicators.
9. "Data layer for SEO" competitors and analogues
| Player | What it is | Evidence | Source |
|---|---|---|---|
| Property Finder, Bayut, Property Monitor, REIDIN | Existing Dubai portals/data vendors | Named from my background knowledge of the market, not fetched this session. Unverified here. | n/a |
| propertyindex.ae | Dubai real-estate data platform; has a page distinguishing DLD records, medians, indices, AVMs and listings. | Self-description (secondary, vendor SEO content). | https://www.propertyindex.ae/dubai-real-estate-data-platforms |
| dxbdata.io | DLD data via MCP. | First-party page. | https://dxbdata.io/start |
| Programmatic SEO vendors (e.g. viewengine.ai, seo.to, aioseo.com case studies) | They publish case studies and sell tooling. | Sources used for cases; treat as marketing. | cases.md |
10. Reference pipeline for Anthony (assessment, not a recommendation to buy anything)
Source (DLD/Dubai Pulse, listings, partners) -> Crawlee/Apify/Firecrawl or API -> dlt or plain Python -> Supabase Postgres (or Parquet + DuckDB) -> SQL/dbt transforms -> Pandera/GX tests as publish gates -> Astro/Next pages with JSON-LD and honest <lastmod> -> Cloudflare/Vercel cron -> Search Console + a tracker for measurement. Nothing in this list requires the paid tiers above at two-site scale.