Data Layers

The Data Layer Manual

On this page
  1. 1. What a data layer is
  2. 2. The pipeline
  3. 3. Choosing sources
  4. 4. Ingestion
  5. 5. Normalisation: identity and definitions
  6. 6. The fact store
  7. 7. Validation gates
  8. 8. Generating pages
  9. 9. Publishing
  10. 10. Freshness
  11. 11. AI search (GEO) without the folklore
  12. 12. Staying on the right side of Google
  13. 13. Monitoring
  14. 14. Runbooks
  15. 15. Checklists
  16. 16. Pitfalls (each one happened to us or to someone we cite)
  17. 17. Who does what
  18. 18. Where to read next

Version 1, 25 Oct 2026, Dubai time (UTC+4). Written for Anthony Booth's team. Everything here is grounded in what we built for buydubai.io and benidorm.co.uk. Where a rule is our own, it says so. Where outside best practice differs from what we do, it says that too.

Evidence labels: [V] checked this run, [R] from a team report, [U] unverified.

1. What a data layer is

A data layer is the part of a website that holds facts nobody else has put in one place, with a source and a date on each one, and turns them into pages and answers that change when the facts change.

Without it, a site is words. Search engines and AI assistants already have words. They do not have, for example, how many apartments sold in JVC last week (buydubai.io), or what a Benidorm Palace dinner show costs this month (benidorm.co.uk). That difference is the whole business case.

Our working definition:

Ingest → clean and structure → validate → generate pages → publish → watch. Every number on a page can be traced to a row, a source and a date.

What it is not: a scraper, a CMS, or a prompt that writes articles. If a model writes the number, it is not a data layer.

2. The pipeline

Stage by stage, with what we do

Stage Rule buydubai.io benidorm.co.uk
Source Official or open first. Record licence and date. No scraped review text. DLD transactions, Ejari, KHDA, OSM, Central Bank rules; Property Finder; Google Places photos behind a spend gate [R] Seven numbered data feeds, each figure with source URL and "checked" date; data/eeat/sources.md register [R/V]
Ingest Raw files are kept. Fetch scripts are not part of the build. Nightly job; on failure restores last good build [R] Fetch scripts write to data/; build never fetches [R]
Normalise Deterministic keys first. Fuzzy matches go to a review queue, never auto-merge. Registers: communities, buildings, aliases [R] Venue and hotel entities from OSM and venue IDs [U on match rate]
Fact store Each fact has an ID, a definition and a revision hash. facts.parquet + cube_meta.json, DuckDB, fact_id, cube_hash, metrics.yaml, factlib.py [R] JSON/CSV feeds plus build_all.py manifest of about 55 steps [R]
Validate A failing check blocks publishing. lint_pages.py, community_eval.py (pass 85), spend gate [R] CI rebuild-and-diff described; I could not locate the workflow file [U]
Generate Templates read facts. Nobody edits generated HTML. Builders must call factlib.py (FL.get, FL.series) [R] "HTML is output, never edit by hand"; frozen clock via freezegun [R]
Publish One route to live. Rollback id recorded. Gated deploy scripts; Vercel overlay from MacPro [R] Only from GitHub main through Vercel [R]
Surfaces Sitemap, canonical, JSON-LD that matches visible text. Optional llms.txt and MCP. Sitemap 8,156 URLs; llms.txt (4,265 B); no llms-full.txt [V] Sitemap 752; llms.txt 221 KB; llms-full.txt 4.9 MB; MCP server with 9 tools [V]
Monitor Freshness, gate failures, search and AI traffic. GSC and GA4 connected; Traks tag check [V/R] GSC and GA4 connected [V]

3. Choosing sources

Ask five questions of every candidate source. If two answers are "no", skip it.

  1. Official or primary? (DLD, Ejari, INE, AENA, a venue's own site.)
  2. Open licence or clear terms? Write the licence into the register.
  3. Changes often enough to matter? Daily or weekly sources create "changes daily" content. Annual ones do not.
  4. Nobody else has joined it? The value is in the join (sales + rents + building identity), not in the single feed.
  5. Can we state its limits? Example: DLD villa records give plot area, so per-sq-ft is not shown for villas (our own page says so [V]).

Never: review text copied from Google Places or TripAdvisor (Fintan's Google Places lesson [R]); a source you cannot name on the page; an LLM's recollection.

Open item: Property Finder and Google Places carry rights questions. Tiers (public_citable, internal_assist_only, citable_with_attribution) were decided on 24 Oct but are not implemented [R].

4. Ingestion

5. Normalisation: identity and definitions

Identity. Give every real-world thing one ID (CM… community, BL… building). Join on the strongest key (DLD project number, licence). Keep an alias table and keep retired IDs with redirects. This is Gully's crosswalk rule and standard master-data practice [V].

Definition. Give every number one definition: name, formula, unit, population, window, minimum sample. Put it in a register (ours: metrics.yaml). Pages read the register. Language models read numbers; they never calculate them.

Suppress small samples. Our cube suppresses cells under its minimum counts (for example lr_* values with n under 10) [R].

Publish match rates. Our own reports: sales to community 87.9%; sales to tower 75.5%; leases to building 30.8% [R]. Decide a floor for each and show the rate on an internal dashboard.

Conflicts. If two sources disagree, keep both with their basis and explain. Do not pick by confidence score. Example: Peninsula Three, 156 on the site at build time, 158 in the cube, 35 in the old Supabase card [R].

6. The fact store

Test for any store: build it twice, diff it. Zero diff or it is not a fact store. The staged cube passed (0 rows differ both ways). The 23 Oct cube failed (13,547 rows differ; 972 NULL fact_id) [R].

7. Validation gates

Gates run before publish. A gate that only warns is not a gate.

Gate What it checks Where
Number lint Every number on a page matches a fact in its manifest; manifest hash equals the live cube; no page prints numbers without a manifest lint_pages.py (written 21 Oct; no-manifest check 24 Oct) [R]
Quality eval 0–100 score on data, content, place, quality, links; pass at 85 community_eval.py (24 Oct; wired into release scripts 25 Oct) [R]
Parity Page, JSON-LD, exports and assistant answer give the same figures Gully section 11; Town Square proof [R]
Spend Paid calls under cap GOOGLE_PAID_OFF, spend log [R]
Rebuild-diff Rebuild from main; fail if output differs Benidorm CI build-check.yml [R, U on file]
Independent audit A second agent that never writes code checks diffs, runs fixtures, signs off a status ladder Grok auditor [R]

Status ladder (Gully): source-present → implemented/wired → locally tested → independently reviewed → integrated → live. "Live" requires a fetch of the live URL, not a green log.

Rule from our own RULES file: every post to the team ends with a SELF-CHECK with real clock times and pasted gate output [R].

Known gaps to close before you rely on gates (from 20-scorecard.md): coverage (5,931 agent pages have no manifest [V]); the eval's pass line was set against one gold page and is uncalibrated [R].

8. Generating pages

  1. A page earns its place. It has a question people ask, a table nobody else has, and a dated line saying when data was checked.
  2. Templates read facts only. No numbers typed in templates. (JVC video section hard-coded 1,296 vs cube 1,273 [R]. That is the failure.)
  3. State limits in the text. Example: "Per sq ft is not shown for villas: many villa records give the plot area" [V].
  4. Prose from a model is allowed; numbers are not. The model gets the fact list and writes sentences. The lint checks the sentences.
  5. Name a human only when the human wrote it. The agent pages carry a templated quote attributed to a named broker; whether he wrote it is unverified [V/U]. Do not ship that.
  6. Size. A gold buydubai community page is 8,000–18,000 words across its pages [V]. A Benidorm page has a median of about 1,900 words [V]. A 5,000-character page repeated 5,900 times is a risk (section 12).

9. Publishing

10. Freshness

Say what you mean by "daily".

Claim Check
"Updated every morning" data_as_of on the page is within 48 hours of today. On 25 Oct buydubai pages say 22 and 23 Oct [V].
Sitemap lastmod Moves only when the page's data changes. 98.6% of buydubai URLs show 23 Oct [V]. Benidorm: 588 of 752 show 24 Oct [V]; confirm that is data, not rebuild date [U].
llms.txt counters Rebuilt with the site. buydubai's says data to 22 Oct, rebuilt 23 Oct [V].

Show dates on the page. Benidorm's asof.py takes "data captured" from git history, never the build date [R]. Copy that.

11. AI search (GEO) without the folklore

What Google says [V]: no extra requirements for AI features; no special schema; no AI text files needed; the page must be indexed and snippet-eligible; structured data must match visible text.

What the X and study evidence says [R, from research/x-posts.md]: schema gave no lift in an Ahrefs test; llms.txt showed no effect in an SE Ranking study; many success claims are self-reported.

So we do these because they are cheap and honest, not because they are proven drivers:

Measure what happens. Our first signal: ChatGPT referrals (78 on buydubai, 101 on Benidorm, 20 Sep–25 Oct) against Google organic (77 and 84) [V]. Tiny numbers; treat as a start.

12. Staying on the right side of Google

Google's spam policy on scaled content abuse: many pages generated mainly to manipulate rankings, however created, may be excluded from Search [V]. Related: doorway pages, thin affiliate pages, scraped content.

A page set passes when:

Our check by page set:

Set Count Test
buydubai communities, towers, weeks about 2,200 Data-rich, manifest, gate. Pass, with placeholders on three community pages.
buydubai agents 5,931 Thin, templated, no manifest. Fail until fixed.
buydubai units about 90,000 (two orphan sitemaps) No manifest; per-unit "worth" estimate; about 915 words per page with navigation. Unproven. Decide before it is crawled at scale. [V]
Benidorm venues 499 of 752 Substantive in sample (median 1,878 words); watch same-template risk.

When in doubt: noindex the weak set, keep the strong set, and remeasure in four weeks. (This is reversible. A site-wide penalty is not quickly so. See the National Today case in research/x-posts.md.)

13. Monitoring

Weekly, one page:

  1. GSC clicks, impressions, pages indexed (per site).
  2. GA4: sessions by source, with internal traffic filtered.
  3. Gate log: failures this week, by gate.
  4. Freshness: oldest data_as_of on live pages.
  5. Link rates and suppressed-cell counts.
  6. Spend.

Alert on: nightly job failed; live hash differs from latest good hash; any 4xx or 5xx on a sitemap URL; GSC impressions down more than 50% week on week.

14. Runbooks

14.1 Daily run (buydubai)

  1. Check the nightly job log. If failed, read why before publishing anything.
  2. Build the cube. Diff the second build. Zero diff.
  3. Run lint_pages.py and community_eval.py on the pages that changed.
  4. Run the gated deploy script. Record the deployment id.
  5. Fetch the live pages. Check the manifest hash, one number, one date.
  6. Post the SELF-CHECK with times.

14.2 New community page (buydubai)

  1. Register the community (CM…) and its page folder.
  2. Build all pages from the cube.
  3. Eval pass at 85 or more. Lint zero problems.
  4. Review the "what we could not confirm" lines.
  5. Deploy through the gated script. Fetch. Post.

14.3 New feed (Benidorm)

  1. Add the source to data/eeat/sources.md (URL, licence, checked date).
  2. Write the fetch script (writes to data/ only).
  3. Add a step to the build_all.py manifest.
  4. Rebuild twice; diff must be zero.
  5. Commit to a branch; merge to main; Vercel publishes.

14.4 A number is wrong

  1. Find the page's manifest and the fact_id.
  2. Trace the row in the cube or feed. Fix at the source or the definition.
  3. Rebuild affected pages. Re-lint.
  4. Deploy. Fetch. Note the correction on the page if it was public for more than a day.

14.5 Rollback

Use the recorded rollback deployment id. Fetch the live URL after.

15. Checklists

Before a page set goes live - [ ] Every sitemap URL has a manifest or a written exclusion - [ ] Lint 0 problems; eval at or above pass line - [ ] One revision hash across the release - [ ] As-of dates visible and recent - [ ] No placeholder text ("not shown yet") - [ ] No named person quoted unless they wrote it - [ ] Sources and licences in the register - [ ] Rollback id recorded - [ ] Live fetch done

Weekly - [ ] Traffic report - [ ] Gate failures reviewed - [ ] Oldest data_as_of - [ ] Spend log

16. Pitfalls (each one happened to us or to someone we cite)

Pitfall Our example
Publishing before the gate exists Five thin community pages live before the eval was wired [R]
Gate that covers a quarter of pages 1,469 of 17,359 on disk had a manifest in the PROJECT report [R]
Hybrid hand-typed numbers JVC 1,296 vs 1,273 [R]
Two data revisions live at once 661f7039 vs 74bc4c52 [V]
Claim says daily, data says three days ago Town Square "to 22 October" on 25 Oct [V]
Fuzzy matching by language model 13 guessed project mappings dropped [R]
Counting AI files as ranking drivers Google and two studies say otherwise [V/R]
Scale before proof National Today deindex [R]
Treating "direct" traffic as real May include our own testing [U]
Quoting a plan as a result The INVRT spec is a plan; 3 of 36 fixtures pass [R]

17. Who does what