Bluesky indexer

This is a bunch of code that can download all of Bluesky into a giant table in PostgreSQL.

The structure of that table is roughly (repo, collection, rkey) -> JSON, and it is a good idea to partition it by collection.

System requirements

NOTE: all of this is valid as of April 2024, when Bluesky has ~5.5M accounts, ~1.2B records total, and average daily peak of ~100 commits/s.

• One decent SATA SSD is plenty fast to keep up. Preferably a dedicated one (definitely not the same that your system is installed on). There will be a lot of writes happening, so the total durability of the disk will be used up at non-negligible rate.
• 16GB of RAM, but the more the better, obviously.
• ZFS with compression enabled is highly recommended, but not strictly necessary.
  • Compression will cut down on IO bandwidth quite a bit, as well as on used disk space. On a compressed FS the whole database takes up about 270GB, without compression - almost 3 times as much.

Overview of components

Lister

Once a day get a list of all repos from all known PDSs and adds any that are missing to the database.

Consumer

Connects to firehose of each PDS and stores all received records in the database.

If CONSUMER_RELAYS is specified, it will also add any new PDSs to the database that have records sent through a relay.

Record indexer

Goes over all repos that might have missing data, gets a full checkout from the PDS and adds all missing records to the database.

PLC mirror

Syncs PLC operations log into a local table, and allows other components to resolve did:plc: DIDs without putting strain on https://plc.directory and hitting rate limits.

Setup

• Decide where do you want to store the data
• Copy example.env to .env and edit it to your liking.
  • POSTGRES_PASSWORD can be anything, it will be used on the first start of postgres container to initialize the database.
• Optional: copy docker-compose.override.yml.example to docker-compose.override.yml to change some parts of docker-compose.yml without actually editing it (and introducing possibility of merge conflicts later on).
• make start-plc
  • This will start PostgreSQL and PLC mirror
• make wait-for-plc
  • This will wait until PLC mirror has fully replicated the operations log. That's gonna take a few hours.
  • Technically you can start everything before it is caught up: it will return errors and other components will fallback to querying https://plc.directory. But you will be rate-limited quite hard.
• make init-db
  • This will add the initial set of PDS hosts into the database.
  • You can skip this if you're specifying CONSUMER_RELAYS in docker-compose.override.yml
• make up

Additional commands

• make status - will show container status and resource usage
• make psql - starts up SQL shell inside the postgres container
• make logs - streams container logs into your terminal
• make sqltop - will show you currently running queries
• make sqldu - will show disk space usage for each table and index

Tweaking the number of indexer threads at runtime

Record indexer exposes a simple HTTP handler that allows to do this:

curl -s 'http://localhost:11003/pool/resize?size=10'

Advanced topics

Table partitioning

With partitioning by collection you can have separate indexes for each record type. Also, doing any kind of heavy processing on a particular record type will be also faster, because all of these records will be in a separate table and PostgreSQL will just read them sequentially, instead of checking collection column for each row.

You can do the partitioning at any point, but the more data you already have in the database, the longer will it take.

Before doing this you need to run lister at least once in order to create the tables (make init-db does this for you as well).

• Stop all containers except for postgres.
• Run the SQL script in psql.
• Check migrations dir for any additional migrations you might be interested in.
• Once all is done, start the other containers again.