Atalaya — A Watchtower over Chile's Open Data
Atalaya harvests Chile's Data Observatory open catalog, profiles every downloadable table, and mines five kinds of cross-dataset relation — same-source, semantic similarity, spatial overlap, joinability and statistical correlation — into an explorable knowledge graph, with client-side semantic search running in the browser. Honest about evidence strength, never causal.
Business Context
Public data is only as useful as your ability to connect it, and the connective tissue — which datasets join, overlap or correlate — is exactly what catalogs never provide. Atalaya turns a flat catalog into a navigable graph of relationships, which is the difference between "here are a thousand tables" and "here are the two that, joined, answer your question." Doing it with statistical discipline (nulls, FDR, partial-correlation guards) is what keeps the connections honest.
Strategic Value
Atalaya demonstrates an end-to-end data-engineering + IR stack — catalog harvesting, profiling, entity-resolution/joinability, statistically-vetted relations and a calibrated fusion score — delivered as a static, backendless, client-inference web app. Its discipline is the point: it reports evidence strength honestly (of ~14,000 edges, the vast majority are cheap priors; the hard evidence is a few hundred joins and a handful of FDR-controlled correlations), it never implies causation, and the modern embedding model beats the classical baseline only modestly. That honest, reproducible framing is what makes it a credible relation explorer rather than a graph that overstates itself.
The Challenge
An open-data catalog is a list of tables, not a map of how they relate. Chile's Data Observatory publishes over a thousand public datasets, but which two could be joined, which cover the same territory, which correlate, which come from the same source — the questions that make open data actually useful — are invisible. Answering them at catalog scale needs real data engineering (harvest, profile, entity-resolve) and statistical care, not a keyword search.
Our Approach
Atalaya is a data-engineering + information-retrieval pipeline — harvest → profile → relate → evaluate → export — run over 1,017 real datasets from the catalog (with a measured multi-GB partial mirror). It profiles every table, then mines five relation types into a knowledge graph: same-source, semantic similarity (a MiniLM sentence model exported to ONNX), spatial overlap, joinability (MinHash containment) and statistical correlation (Spearman with a permutation null, Benjamini-Hochberg FDR control and a partial-correlation guard). A novel calibrated multi-evidence "affinity" score fuses the signals against null-distribution percentiles and reliability weights, and can be re-weighted live. It ships as a static React SPA with the graph baked in and semantic search running client-side (transformers.js / onnxruntime-web) — no backend.
Key Performance Indicators
| KPI | Baseline | Result | Impact |
|---|---|---|---|
| From catalog to map | A flat list of tables | A knowledge graph of 5 relation types over 1,017 datasets | Find the datasets that actually connect |
| Honest evidence | "14,000 relationships" | Strength labelled: cheap priors vs 117 joins + 24 FDR-controlled correlations | You know which edges to trust |
| Where it runs | A served backend | Static SPA; semantic search runs client-side (ONNX) | No server, fully inspectable |
Architecture
atalaya graph
A map of how open data connects
Atalaya is a watchtower over Chile’s open data. It harvests the Data Observatory catalog, profiles every downloadable table, and mines five kinds of cross-dataset relation into an explorable knowledge graph — turning a flat list of a thousand-plus datasets into a map of which ones join, overlap, correlate or share a source. Live at atalaya.fasl-work.com.
Five relations, mined with care
Over 1,017 real datasets: same-source, semantic similarity (a MiniLM model exported to ONNX), spatial overlap, joinability (MinHash containment) and statistical correlation (Spearman with a permutation null, Benjamini-Hochberg FDR control and a partial-correlation guard). A novel calibrated affinity score fuses the signals against null-distribution percentiles and can be re-weighted live. The whole thing ships as a static SPA with the graph baked in and semantic search running client-side — no backend.
Honest about the graph
The number that matters is not “14,000 relationships” — most of those are cheap priors. The hard evidence is a few hundred joinable pairs and a handful of FDR-controlled correlations, and Atalaya labels that strength rather than hiding it. It never implies causation (some small-n correlations are flagged fragile), and the modern embedding model beats the classical TF-IDF baseline only modestly (+1.4 points). It is a relation explorer, reported at the confidence the data supports.
Technology Stack
Visual assets for this project are not publicly available.