astra
/
plc-mirror
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
187 Commits
1 Branch
0 Tags
402 KiB
Commit Graph

187 Commits (56727bbe1105e33fb46b2a2a23b60995cd41a260)

Author SHA1 Message Date
Max Ignatenko d6b5850827 Add plc mirror 
If you want to avoid sending all requests to https://plc.directory while
it's catching up for the first time, do
`docker compose up -d --build plc` and wait for it to catch up before
updating all other containers.
 2024-04-13 16:45:02 +01:00
Max Ignatenko c17730c11f Drop the hack for detecting cursor reset from non-compliant servers 2024-04-10 23:32:21 +01:00
Max Ignatenko 4b40c5919b Don't update LastFirehoseRev more than once a day 2024-04-07 15:06:13 +01:00
Max Ignatenko 581cf8c720 go mod tidy 2024-04-07 14:55:02 +01:00
Max Ignatenko 99c0b08eab Write more words 2024-04-07 14:53:22 +01:00
Max Ignatenko 52dd38f11b Don't attribute context cancellation to firehose record content 2024-04-06 22:14:58 +01:00
Max Ignatenko ecf2fc57d8 Disconnect from firehoses in parallel when shutting down 2024-04-06 21:59:21 +01:00
Max Ignatenko 1358bc3f08 Export a counter of firehose connection errors 2024-04-06 21:55:36 +01:00
Max Ignatenko ff0ea08296 Implement commit signature validation 2024-04-06 21:50:51 +01:00
mathan f9dde4db39 Update dash. 2024-04-03 12:14:49 -07:00
mathan 9cfc23b981 Update dash. 2024-04-03 12:12:28 -07:00
Max Ignatenko 89d3a264c9 Export the number of bad firehose records for each PDS 
Fixes #1
 2024-03-29 12:02:05 +00:00
Max Ignatenko 7c09c37a51 Pass the correct context to Consumer.Start so that it will actually stop 
when singalled
 2024-03-29 10:16:22 +00:00
mathan 567885af60 Fix grafana not restarting. 2024-03-28 14:57:20 -07:00
Max Ignatenko 1abe505ef9 Add support for discovering new PDSs from relays 2024-03-28 20:55:02 +00:00
Max Ignatenko c919050833 Keep the set of running consumers up to date 2024-03-28 20:02:48 +00:00
Max Ignatenko 337f3ef2b8 Delete commented out code 2024-03-28 18:46:39 +00:00
Max Ignatenko 3da7228f35 Wait for postgres to become ready before starting other containers 2024-03-28 16:16:38 +00:00
Max Ignatenko fc5307a971 Add a bit more logging for cursor values 2024-03-28 16:00:28 +00:00
Max Ignatenko 5d3d562ecc Stop the ticker when goroutine exits 2024-03-28 16:00:28 +00:00
mathan 4556d6738a Update makefile. 2024-03-27 19:59:24 -07:00
mathan 3aca467516 Update iexport. 2024-03-27 16:18:18 -07:00
Max Ignatenko ffa2faa420 Properly escape null character in the consumer too 2024-03-24 12:43:52 +00:00
Max Ignatenko 693ae1ba0a Bump the limit on stored bad records to 500 2024-03-24 12:32:13 +00:00
mathan d189912bcf Update dash. Add alerting webhook. 2024-03-20 07:18:15 -07:00
Max Ignatenko bae23a62d0 Don't try to upsert zero records 2024-03-20 13:31:14 +00:00
Max Ignatenko 542be03088 Add all PDSs to whitelist 2024-03-20 13:31:14 +00:00
mathan dc17238a93 Update dash 2024-03-18 18:01:47 -07:00
Max Ignatenko c3bd6d177a Expand some terms used 2024-03-17 22:03:16 +00:00
Max Ignatenko e8c816a3a3 Update FirstCursorSinceReset even if we received zero new records 2024-03-17 21:22:34 +00:00
Max Ignatenko 328a676e2a Fix potential infinite loop for inactive repos after a cursor reset 2024-03-17 19:36:12 +00:00
Max Ignatenko 4c41389e9b Detect CAR files with zero blocks and handle them accordingly 2024-03-17 17:42:09 +00:00
Max Ignatenko 638bdcf515 Add a check for zero bytes fetched from PDS 
And log additional info when we're failing to extract rev from repo
bytes
 2024-03-17 17:15:23 +00:00
mathan bc895b6425 Update dash. 2024-03-17 08:52:28 -07:00
Max Ignatenko 553894dc6a Switch to partial repo fetches 2024-03-17 15:34:26 +00:00
Max Ignatenko 346aa7c539 Document consistency model for repo data 2024-03-17 15:21:54 +00:00
Max Ignatenko 0cc11e75b2 Avoid updating records when then didn't actually change 2024-03-17 14:16:25 +00:00
Max Ignatenko 9c51a4621f Start recording last rev for each repo 2024-03-13 22:28:54 +00:00
mathan 85259baa1c Update dash. 2024-03-13 14:21:05 -07:00
mathan 9d429be424 Update dash. 2024-03-13 11:33:13 -07:00
Max Ignatenko 76a50524a0 Give postgres plenty of time to shutdown cleanly 2024-03-13 18:12:55 +00:00
Max Ignatenko 44d2b25951 Fix nil pointer deref 2024-03-13 16:38:58 +00:00
Max Ignatenko e150f1da90 Populate FirstRevSinceReset if empty 2024-03-13 11:32:23 +00:00
Max Ignatenko 87d510e67a Check repos against PDS cursor resets, instead of waiting for a first new even for them on firehose 2024-03-13 10:39:41 +00:00
Max Ignatenko 57aa4731e5 Properly update repo consistency metadata 
Previously these two queries did nothing at all -_-

Note that this will trigger a full re-index of every repo seen on the
firehose.
 2024-03-13 10:36:41 +00:00
Max Ignatenko 080cbb8b12 Bump WAL size limit to 8GB 2024-03-13 09:51:40 +00:00
mathan 75386c0235 Add export_blocks view. Add iexport + audit table. 2024-03-12 21:15:23 -07:00
mathan e1ec43def7 Add count listitems. 2024-03-06 18:58:08 -08:00
mathan dc80cd95ad Update stats dash. 2024-03-05 14:17:46 -08:00
mathan 209813bd87 Update dash + csv_export. 2024-02-28 10:37:18 -08:00