Developers can now treat the entire web of blog posts like a relational database thanks to a new open-source utility called Select. The project, which appeared on Hacker News with a spike in community interest, provides a SQL-compatible query layer over publicly accessible blog content through a dataset labeled Internet.blogposts.

What You Need to Know

Select enables users to retrieve and filter blog post metadata and content using standard SQL syntax without scraping individual sites. The Internet.blogposts data source is a curated collection of RSS feeds and parsed HTML from millions of blogs. This approach reduces the barrier for researchers, journalists and hobbyists who want to analyze blog trends at scale.

How Select Works

At its core, Select exposes a PostgreSQL-compatible endpoint where developers issue queries against a virtual table called Internet.blogposts. The engine indexes title, author, publication date, body text and tags from more than 10 million blog entries. A typical query may look like:

  • SELECT title, url: Retrieve specific columns instead of full rows.
  • FROM Internet.blogposts: Target the global blog index.
  • WHERE published_date > '2024-01-01': Filter by recency.
  • AND tags @> ARRAY['machine learning']: Use array contains operators for topic filtering.

The results stream back in JSON or CSV format, making downstream analysis straightforward. The project includes a public API key tier for free use with rate limits, plus self-hosted deployment options for heavy users.

Background and Community Response

The initial announcement on Hacker News attracted hundreds of comments within hours. Contributors discussed indexing freshness, query performance and whether the service respects robots.txt directives. The maintainer has indicated that the project remains pre-alpha and expects breaking changes before a stable release.

Select currently supports projection, filtering with WHERE clauses, basic aggregation (COUNT, AVG) and sorting with ORDER BY. It does not yet support JOIN operations across multiple Internet datasets, though the roadmap hints at cross-source queries.

Why This Matters

Aggregating blog content at scale today demands either custom scrapers or reliance on centralized search engines that offer limited programmatic access. If Select matures, it could democratize bulk content analysis for smaller teams that cannot afford enterprise crawling infrastructure. Journalists tracking misinformation across blogs or analysts monitoring product sentiment would gain a SQL-native window into unstructured publishing data. The project also raises important questions about how permissive the web wants to remain with respect to automated indexing beyond traditional search bots. Regulators and site owners will watch closely as adoption grows.