Bookmarks

Data Toolkit — October 18, 2026

Public Datasets

NOAA Global Historical Climatology Network – Daily
NOAA's daily station network pools max/min temperature, precipitation, snowfall, and snow depth from more than 100,000 stations across 180 countries and territories, with some records reaching back to the 1830s. The archive is rebuilt every weekend and lags about 45 to 60 days behind the present; pull it from the public AWS S3 bucket, Google BigQuery, or a flat FTP download from NCEI. **Pricing:** free, public domain (CC0-1.0).
Wikidata JSON Dumps
Wikidata's weekly dump compresses the entire structured knowledge graph — every item, property, and lexeme — into a single `latest-all.json.bz2` file running around 103 GB compressed (156 GB as gzip). Each line is one JSON entity, so a reader can stream-process the file instead of loading the whole graph into memory; RDF and XML dumps ship alongside it. **Pricing:** free, public domain (CC0).
Overture Maps Foundation
Overture shipped schema v2.0.0 on September 23, 2026, retiring the old free-text `categories` field on its Places theme for a proper taxonomy. The monthly release now counts 81.5 million places, 474 million addresses, 2.53 billion buildings, and 774 million transportation segments, delivered as partitioned GeoParquet on S3 and Azure and queryable straight from DuckDB or the `overturemaps` CLI. **Pricing:** free; CDLA-Permissive-2.0 for most themes, ODbL for OSM-derived layers.
Common Crawl
Common Crawl's latest archive, CC-MAIN-2026-39, crawled 2.17 billion pages between September 4 and 17, 2026, for 361 TiB of raw HTML plus matching WET text and WAT metadata extracts. It's still the corpus most open pretraining runs lean on for raw web text, pulled from the `commoncrawl` S3 bucket or queried through the Index API. **Pricing:** free, custom Common Crawl Terms of Use (not a CC license).

Free & Open APIs

Open-Meteo
Open-Meteo answers forecast, historical, and climate-model queries — ECMWF, GFS, and others — with no API key at all for non-commercial use, capped at 10,000 calls a day. A single `GET /v1/forecast?latitude=52.52&longitude=13.41&hourly=temperature_2m` call is the whole onboarding process, which is probably why it's become the go-to swap-in for paid weather APIs in side projects. **Pricing:** free, 10,000 calls/day, no key required for non-commercial use.
NASA Open APIs
NASA's API gateway fronts APOD, Near Earth Object tracking, EPIC Earth imagery, and DONKI space-weather alerts behind a shared `DEMO_KEY` good for 30 requests an hour and 50 a day; a free registered key raises that to 1,000 requests an hour. Usage shows up in the `X-RateLimit-Remaining` response header, so there's no guessing when you're near the wall. **Pricing:** free, DEMO_KEY 30/hour or registered key 1,000/hour.
REST Countries
REST Countries spent years as the textbook example of a keyless, limitless API — that ended in 2026, when the old `restcountries.com/v3.1` endpoints were deprecated in favor of a keyed `api.restcountries.com` v5 capped at 1,000 requests a month (20 per 10 seconds, with warning emails at 50/80/100% of the cap before a grace period). It still returns the same country reference data — names, capitals, currencies, borders, flags across 90-plus fields — now bundled with currency conversion and IP geolocation add-ons; a non-counting `rc_live_demo` key lets you try it without an account. **Pricing:** free tier, 1,000 requests/month, key required.
Transitland REST API
Transitland aggregates GTFS static and realtime feeds from thousands of transit agencies into one REST API, so a single `GET /api/v2/rest/stops?apikey=...` call returns stop and schedule data regardless of which agency runs the route. The free Explorer plan allows 10,000 REST queries a month, plus separate caps for the routing API (1,000/month) and vector tiles (100,000/month). **Pricing:** free Explorer tier, 10,000 queries/month, key required.

Data Engineering & Pipelines

Apache Airflow
Airflow still anchors most orchestration stacks, now on the 3.x line, which added a dedicated task-state store and a multi-language Task SDK (Java, Go) so pipelines aren't locked to Python-only workers; 3.3.2 landed September 17, 2026. Install with `pip install 'apache-airflow==3.3.2' --constraint <matching-constraints-url>` — the constraints file matters, since Airflow pins its transitive dependencies hard. **Pricing:** free, open source (Apache 2.0).
dbt
dbt reached general availability on v2.0 on September 16, 2026, after rewriting the compiler core from Python to Rust under the "Fusion" codename — projects that used to take 20 minutes to parse now parse in under one. Install with `brew install dbt` and keep writing the same SQL models and tests; the engine underneath changed, the workflow on top of it didn't. **Pricing:** free, open source (Apache 2.0).
Apache Iceberg
Iceberg lets Spark, Flink, and Trino read and write the same lakehouse tables without stepping on each other's snapshots, and 1.12.0 (September 29, 2026) added server-side scan planning for the REST catalog alongside Spark 4.1 and Flink 2.1 support. Pull it in as a build dependency — `org.apache.iceberg:iceberg-core:1.12.0` via Maven, or the matching Spark runtime package — rather than installing it standalone. **Pricing:** free, open source (Apache 2.0).
dlt (data load tool)
dlt is the code-first alternative to point-and-click ingestion tools: Python functions become pipelines with schema inference and normalization built in, and release 1.31.0 (October 7, 2026) added a stateful `Relation.incremental()` for gap-free incremental SQL loads. `pip install dlt` gets the core; add a destination extra like `dlt[duckdb]` for wherever the data is landing. **Pricing:** free, open source (Apache 2.0).

Analytics & Notebooks

DuckDB
DuckDB embeds a columnar SQL engine directly in the host process, so Python, R, or a CLI can query Parquet, CSV, or Iceberg files in place without a server to run or manage. It's the usual reach for in-process analytics, shipping near-daily commits with 1.5.6 out September 28, 2026; install with `pip install duckdb` or the equivalent for your language. **Pricing:** free, open source (MIT).
Polars
Polars 2.0 shipped October 6, 2026 with out-of-core spill-to-disk support, a new Map dtype, and first-class SQL — and now leads DuckDB and DataFusion on the TPC-H and TPC-DS benchmarks by the project's own numbers. The Rust engine underneath makes lazy, multi-threaded query execution the default rather than an opt-in; install with `pip install polars`. **Pricing:** free, open source (MIT).
marimo
marimo stores notebooks as plain, git-diffable `.py` files and reruns cells by dependency rather than execution order, so a changed variable propagates correctly instead of leaving stale output behind — the class of bug that makes a Jupyter notebook an unreliable record of what actually ran. Release 0.25.1 shipped October 1, 2026, roughly in step with the project's every-two-to-three-week cadence this year; install with `pip install marimo`. **Pricing:** free, open source (Apache 2.0).
Harlequin
Harlequin is a terminal SQL IDE for people who'd rather not leave the keyboard, talking to DuckDB, SQLite, Postgres, MySQL, and chDB from one TUI. Version 2.16.1 (October 1, 2026) followed close on 2.15.0's chDB support and 2.14.0's `hsql --serve` mode, which holds a session open for a coding agent to query against. Install with `pip install harlequin` or `uv tool install harlequin`. **Pricing:** free, open source (MIT).

Scraping & Collection

Scrapy
Scrapy is still the reference Python scraping framework, wrapping spider code in built-in retry, caching, and scheduling middleware. `ROBOTSTXT_OBEY` is on by default in any project scaffolded with `scrapy startproject`, and the bundled `AutoThrottle` extension watches both server load and site response times to throttle crawl speed automatically instead of relying on a fixed delay. `pip install scrapy` is the whole setup; 2.19.0 shipped September 10, 2026. **Pricing:** free, open source (BSD-3-Clause).
Crawlee for Python
Crawlee pairs HTTP parsing with Playwright-driven browser automation under one queue-based crawler, handling proxy rotation and session pools for jobs too large to run as a single script. Set `respect_robots_txt_file=True` and it skips any URL — including ones discovered through link enqueueing — that the target's robots.txt disallows. Version 1.11.0 landed October 15, 2026; install with `pip install 'crawlee[playwright]'`, then `playwright install` for the browser binaries. **Pricing:** free, open source (Apache 2.0).
Firecrawl
Firecrawl turns a crawl job into clean, LLM-ready markdown or structured JSON, which is why it shows up so often behind RAG pipelines that need web content without a separate HTML-parsing step. Robots.txt is respected by default (bypassing it is an Enterprise-only flag), and a `delay` parameter forces single-threaded, rate-limited crawling on sites that need it. Self-host with `docker compose up`, or install the SDK with `pip install firecrawl-py`; the core engine runs on AGPL-3.0, not a permissive license, while the SDKs stay MIT. **Pricing:** free, self-hosted (AGPL-3.0 core); the hosted API has separate paid tiers.

Data Quality & Catalogs

Great Expectations
Great Expectations turns data-quality rules into declarative Python assertions — this column is never null, this value sits between X and Y — run against batches and rendered into human-readable Data Docs afterward. Version 1.24.0 (October 7, 2026) added a schema-level `ExpectColumnTypeToBe` check and Python 3.14 support, on top of years as the default pipeline-validation layer. Install with `pip install great_expectations`. **Pricing:** free, open source (Apache 2.0).
DataHub
DataHub builds a searchable catalog with column-level lineage across 20-plus sources — Snowflake, BigQuery, dbt, Looker, Power BI — merging native connector lineage and OpenLineage events into one graph instead of two disconnected views. Release 1.7.0 (August 4, 2026) extended lineage tracking to the data-product level, not just individual tables. Bring it up with `pip install acryl-datahub` followed by `datahub docker quickstart`. **Pricing:** free, open source (Apache 2.0).
datacontract-cli
datacontract-cli lints and tests data against the Open Data Contract Standard, catching breaking schema changes before they reach a consumer across 15-plus platforms including Snowflake, Databricks, and Kafka. Release 1.2.7 shipped October 16, 2026, two days before this digest, as the project settled into its role as the de facto ODCS reference implementation. Install with `pip install 'datacontract-cli[all]'`. **Pricing:** free, open source (MIT).
Soda Core v4
Soda Core's v4 rewrite swapped its old SodaCL checks language for YAML-based data contracts as the default, bundling more than 50 built-in checks across SQL and Spark sources; 4.27.0 (October 13, 2026) added contract scopes and cloud rate-limit retry handling. The bigger change is the license, not the YAML: v4 moved off the Apache 2.0 terms that covered v3 onto the Elastic License 2.0, which stays free to self-host but blocks reselling it as a hosted service. Install the package matching your warehouse, e.g. `pip install soda-postgres`. **Pricing:** free to self-host (Elastic License 2.0, not Apache 2.0 like the prior major version).

Vector & Search

Qdrant
Qdrant is the Rust vector engine most self-hosted RAG stacks reach for first, handling payload-filtered similarity search plus native hybrid dense-and-sparse retrieval out of the box. Run the server with `docker run -p 6333:6333 -v "$(pwd)/qdrant_storage:/qdrant/storage:z" qdrant/qdrant` and query it with `pip install qdrant-client`. **Pricing:** free, open source (Apache 2.0).
Meilisearch
Meilisearch pairs typo-tolerant full-text search with an embedder API for hybrid keyword-plus-vector queries, and the open core keeps shipping fast — 1.54 landed September 21, 2026, patched to 1.54.3 on October 1. Install the self-hosted binary with `curl -L https://install.meilisearch.com | sh`; Community Edition is MIT, though sharding and replication live behind an Enterprise Edition under the Business Source License. **Pricing:** free, open source (MIT core; BSL-licensed Enterprise features).
LanceDB
LanceDB embeds vector search, full-text search, and SQL over one columnar format (Lance) instead of bolting a vector index onto a separate database, so there's no server process to run. The underlying Lance format hit v2.2 in September 2026 with blob storage, nested schema evolution, and files over 50% smaller than Parquet, while the SDK itself graduated to v1.0 across Python, Java, and Rust. Install with `pip install lancedb`. **Pricing:** free, open source (Apache 2.0).

Visualization & Dashboards

Apache Superset
Superset is the self-hosted BI pick when a team wants a Looker-style SQL Lab and 40-plus chart types without a per-seat license anywhere in the stack. It's been an Apache Software Foundation project since 2021, currently on stable 6.1.0 with 7.0 already in release-candidate testing as of October 16, 2026. Install with `pip install apache-superset` or the official Docker Compose setup. **Pricing:** free, open source (Apache 2.0).
Grafana
Grafana queries and renders dashboards from Prometheus, Loki, Elasticsearch, Postgres, and dozens of other sources through one plugin model, and it's still the standard self-hosted layer for metrics and logs well beyond pure ops teams. Release 13.2.4 (October 13, 2026) patched several CVEs; bring it up with `docker run -d -p 3000:3000 grafana/grafana-oss`. The core relicensed from Apache 2.0 to AGPL-3.0 in 2021 — running it privately is unaffected, but redistributing a modified version or offering it as a hosted service triggers the copyleft. **Pricing:** free, self-hosted (AGPL-3.0).
deck.gl
deck.gl renders millions of points, polygons, or 3D tiles on a map through composable WebGL (now WebGPU) layers, built by Uber's visualization team and now governed under the OpenJS Foundation. Version 9.4.0 (September 5, 2026) added WebGPU support across every official layer and closed out the v9 line with better globe-controller interaction. Install with `npm install deck.gl`. **Pricing:** free, open source (MIT).
Evidence Core
Evidence rebuilt itself around a terser, statically-validatable syntax on August 26, 2026 — chart and table components now manage their own queries instead of a page full of boilerplate SQL, and markup errors catch before the project ever runs, which the team built specifically so coding agents get usable feedback instead of a blank dashboard. Install with `curl -fsSL https://evidence.studio/install.sh | sh` and point it at a warehouse for BI dashboards rendered straight from Markdown and SQL. **Pricing:** free, open source (MIT). --- *30 entries this month. Every dataset and API lists its licence and limits — check them before you build. Found a dead link or have something to add? Reply to the email.*