plc-mirror/explore/README.md

Graceful shutdown/restart

docker compose stop lister docker compose stop consumer docker compose stop record-indexer

Take a look at grafana, once all quiet

docker compose stop postgres

Start everything up

docker compose up -d --build

Control number of workers

Full throttle curl 'localhost:11003/pool/resize?size=50'

Half throttle (recommended) curl 'localhost:11003/pool/resize?size=25'

Stop eating all of my Internet curl 'localhost:11003/pool/resize?size=10'

Peak into db

docker compose exec -it postgres psql -U postgres -d bluesky

Seen repos select count(*) from repos;

Fully indexed repos select count(*) from repos where last_indexed_rev <> '' and (last_indexed_rev >= first_rev_since_reset or first_rev_since_reset is null or first_rev_since_reset = '');

Get list blocks

non-partitioned (very slow)

select count(*) from (select distinct repo from records where collection in ('app.bsky.graph.listblock') and deleted=false and content['subject']::text like '"at://did:plc:bmjomljebcsuxolnygfgqtap/%');

partitioned (slow) select count(*) from (select distinct repo from records_listblock where deleted=false and content['subject']::text like '"at://did:plc:bmjomljebcsuxolnygfgqtap/%');

Count all records

analyze records; select relname, reltuples::int from pg_class where relname like 'records';

View errors

select last_error, count(*) from repos where failed_attempts > 0 group by last_error;

Restart errors

update repos set failed_attempts=0, last_error='' where failed_attempts >0;

MONITORING

More verbose logging for queries DEBUG1-DEBUG5 set client_min_messages = 'DEBUG5';

Take a look at slow queries

SELECT pid, age(clock_timestamp(), query_start), state, query
FROM pg_stat_activity
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'
ORDER BY query_start asc;

Monitor index progress select * from pg_stat_progress_create_index;

Explore new collection types

select * from records where collection not in (
    'app.bsky.actor.profile',
    'app.bsky.feed.generator',
    'app.bsky.feed.like',
    'app.bsky.feed.post',
    'app.bsky.feed.repost',
    'app.bsky.feed.threadgate',
    'app.bsky.graph.block',
    'app.bsky.graph.follow',
    'app.bsky.graph.listitem',
    'app.bsky.graph.list',
    'app.bsky.graph.listblock'
    ) limit 20;

count listitems select count(*) from listitems where list='at://did:plc:2yqylcqgxier4l5uplp6w6jh/app.bsky.graph.list/3kkud7l6s4v2m';