Data Layers

Best practice for building and governing a data layer for search and AI traffic

On this page
  1. 1. What Google says today (grade A)
  2. 2. Consensus (what most credible sources agree on)
  3. 3. Disagreements and open questions
  4. 4. Data-engineering practice for the layer (grade B-C, standard engineering, not SEO evidence)
  5. 5. Governance checklist for scaled pages (derived from Google's published tests plus the cases)
  6. 6. Evidence quality summary
  7. 7. Limits of this research

Compiled 2026-10-25 (Asia/Dubai). Evidence grades: A = official Google documentation (fetched, dated), B = controlled or large-sample third-party study (vendor-run, so read with caution), C = single practitioner case or agency write-up, D = unsourced claim or opinion. Anything graded D is hype until a method is shown.

1. What Google says today (grade A)

All quotes are from pages fetched this session. Dates are the "Last updated" lines on the pages where captured.

Scaled content abuse (spam policies, updated 2026-08-28; https://developers.google.com/search/docs/essentials/spam-policies): - "Scaled content abuse is when many pages are generated for the primary purpose of manipulating search rankings and not helping users. This abusive practice is typically focused on creating large amounts of unoriginal content that provides little to no value to users, no matter how it's created." - Listed examples include generating many pages with generative AI "without adding value", "Scraping feeds, search results, or other content to generate many pages", and "Stitching or combining content from different web pages without adding value". - The test is value and originality, not how pages are produced. A data-driven page is not automatically abusive; a template that only swaps strings is exposed.

Doorway abuse (same page): "Creating substantially similar pages that are closer to search results than a clearly defined, browseable hierarchy", and "Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page". Relevant to city/area template pages (Dubai areas, Benidorm districts).

Using generative AI content (updated 2026-10-01; https://developers.google.com/search/docs/fundamentals/using-gen-ai-content): generative AI is useful "to add structure to original content", but using it "to generate many pages without adding value for users may violate Google's spam policy on scaled content abuse."

Helpful content (updated 2026-10-05; https://developers.google.com/search/docs/fundamentals/creating-helpful-content): asks whether content "provide[s] original information, reporting, research, or analysis" and "substantial value when compared to other pages in search results". Raters assess Effort ("automatically generating pages from feeds or using generative AI to produce large amounts of text without manual oversight or curation represents little to no effort") and Originality. Another self-check: "Are you using extensive automation to produce content on many topics?"

AI features (https://developers.google.com/search/docs/appearance/ai-features, last updated 2025-12-10): to be a supporting link in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet". The page states there are no additional technical requirements beyond Search's.

AI optimization guide (updated 2026-07-10; https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) lists things you do not need: - "LLMS.txt files and other 'special' markup... Google Search itself doesn't use them." - "'Chunking' content: There's no requirement to break your content into tiny pieces for AI". - "Rewriting content just for AI systems". - "Seeking inauthentic 'mentions'". - "Overfocusing on structured data: Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add. However, it's a good idea to continue using it... it helps with being eligible for rich results." - Summary line: "Prioritize effective SEO strategies over 'AEO/GEO hacks'".

Crawl budget (updated 2026-07-22; https://developers.google.com/crawling/docs/crawl-budget): the guidance targets "Large sites (1 million+ unique pages)" with weekly changes and "Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)". Manage the URL inventory so crawling is not wasted.

Sitemaps (updated 2026-07-08): Google uses <lastmod> "if it's consistently and verifiably... accurate" and ignores <priority> and <changefreq>. For a daily-changing layer, lastmod should change only when page content really changes.

2. Consensus (what most credible sources agree on)

  1. Unique, first-party or hard-to-replicate data is the survival criterion. Google's originality test (A), Ramp's data-led pages holding up better than template pages (C, MV3), Zillow and Nomad List (C), and practitioners on X (D) all point the same way.
  2. Indexation and crawl discipline come first. AI Overviews/AI Mode use the same index as Search (A). A page that is not indexed cannot be cited there. National Today's deindexed section also lost most ChatGPT citations in one documented case (C).
  3. Most generated pages earn little. Across 12.3M pages on nine DR~90 sites, 96.7% get <=100 visits/month (C, Ahrefs estimates via seosavages). Plan for a concentrated head and prune the tail.
  4. Do not rely on GEO tactics with no measured effect. llms.txt (SE Ranking, ~300k domains, no correlation, B) and schema (Ahrefs, 1,885 pages vs 4,000 controls, no meaningful lift, B) both show nulls, matching Google (A).
  5. Measure from first-party data. Search Console for Google; own server logs for AI bot hits; referrer analytics for AI visits.

3. Disagreements and open questions

Topic Position A Position B My read
Are AI-citation tactics real? Google and Gary Illyes via Search Central Live reports: good SEO is good GEO (A/D second-hand) Vendors claim large lifts from structure, FAQ schema, freshness (e.g. @aykut_cevik X post: 44%, 40%, 85% stats, no method) Evidence favours A. The GEO paper (arXiv 2311.09735) reports up to 40% visibility gains on its own benchmark engines, not on production systems (B, academic, older).
Why did Wise/Zapier decline from their 2025 peaks? AI Overviews absorbing queries (Ahrefs: AIO correlates with 58% lower CTR for position 1, December 2025 data vs December 2023, B) Spam/quality enforcement on templated pages Unresolved. MV3 itself says it cannot confirm cause (C). Both can be true.
Is Google deindex = AI invisibility? Gabe: "drop in Google and you can drop in ChatGPT" (C, one case) Some National Today citations survived in ChatGPT (same source) Treat as high correlation, not guaranteed.
How big is AI referral traffic? SE Ranking: AI platforms 0.32% of site traffic in 2026 (B); Semrush: AI under 0.15% of visits in 2025 and +66% growth (B) Marketing claims of AI "traffic" bonanzas Small today, growing, and the two vendors measure differently. Do not plan revenue on it yet.
Does freshness help? Anecdotal ("updated within 30 days cited 3.2x", unsourced X post, D) Google: no ideal cadence stated; lastmod must be accurate (A) Update when data changes; do not fake freshness.
"Vary publish dates so it doesn't look scaled" Folklore on X (D) Not in Google docs Skip: it is a manipulation tactic and adds risk.
Penalty taxonomy Practitioners (Lily Ray, Gabe) report "pure spam" manual actions or algorithmic demotions rather than a labelled "scaled content abuse" action (D, 2025) Others report "major spam problems" actions that cite scaled content abuse (anecdotes) Taxonomy is murky. Assume any of these outcomes.

4. Data-engineering practice for the layer (grade B-C, standard engineering, not SEO evidence)

  1. Provenance and licensing per field. Record source, retrieval time, licence. For Dubai: DLD registered transactions vs listing asking prices are different things; label them, as propertyindex.ae's own explainer insists (vendor content, but the distinction is standard).
  2. Methodology pages. Medians, indices and AVMs differ by inclusion rules (off-plan vs ready, building vs master community, late registrations). Publish the rule set; it is both a trust signal and what stops duplicate-looking pages.
  3. Validation gates before publish. Row-count drift, range checks (price per sqm, yield), uniqueness, referential integrity, freshness. Tools: Pandera, Great Expectations, Soda, dbt tests (tools.md).
  4. Idempotent, versioned pipeline. Raw -> staged -> modelled -> page JSON, with snapshots so a bad ingest can be rolled back. Orchestrate with cron-level tools first.
  5. Page-level quality threshold. Do not publish a page unless a minimum number of unique data points exists. Noindex or merge thin combos. This operationalises Google's "substantial value when compared to other pages" test (A).
  6. Operational hygiene. Valid sitemaps with honest lastmod, canonical rules for filter combos, robots for faceted URLs, log monitoring. A sitemap bug is reported by one practitioner to have cost 2,300 pages (D, anecdote).
  7. Experiment design. Use difference-in-differences against matched control pages as Ahrefs did for schema, rather than before/after. Make changes to a subset of templates.

5. Governance checklist for scaled pages (derived from Google's published tests plus the cases)

6. Evidence quality summary

Claim type Quality Notes
Google policies and AI feature guidance A Fetched from developers.google.com; dated. Re-check before each quarter.
llms.txt and schema null results B Vendor studies (SE Ranking, Ahrefs); observational or matched-control. Consistent with Google.
Traffic numbers for named sites C Ahrefs/Similarweb estimates passed through agencies; sources disagree wildly (Zapier quoted 2.6M to 16M per MV3).
National Today case C+ Detailed practitioner analysis; one site; Brand Radar prompts sampled by hand.
X success stories D Self-reported, no site names or method; impressions are not clicks (e.g. 2.15M impressions vs 6,000 clicks is a 0.28% CTR).
GEO tactic lift statistics D Quoted on X without methodology; contradicted by the nulls above.

7. Limits of this research