Max Ignatenko
|
89d3a264c9
|
Export the number of bad firehose records for each PDS
Fixes #1
|
2024-03-29 12:02:05 +00:00 |
|
Max Ignatenko
|
7c09c37a51
|
Pass the correct context to Consumer.Start so that it will actually stop
when singalled
|
2024-03-29 10:16:22 +00:00 |
|
mathan
|
567885af60
|
Fix grafana not restarting.
|
2024-03-28 14:57:20 -07:00 |
|
Max Ignatenko
|
1abe505ef9
|
Add support for discovering new PDSs from relays
|
2024-03-28 20:55:02 +00:00 |
|
Max Ignatenko
|
c919050833
|
Keep the set of running consumers up to date
|
2024-03-28 20:02:48 +00:00 |
|
Max Ignatenko
|
337f3ef2b8
|
Delete commented out code
|
2024-03-28 18:46:39 +00:00 |
|
Max Ignatenko
|
3da7228f35
|
Wait for postgres to become ready before starting other containers
|
2024-03-28 16:16:38 +00:00 |
|
Max Ignatenko
|
fc5307a971
|
Add a bit more logging for cursor values
|
2024-03-28 16:00:28 +00:00 |
|
Max Ignatenko
|
5d3d562ecc
|
Stop the ticker when goroutine exits
|
2024-03-28 16:00:28 +00:00 |
|
mathan
|
4556d6738a
|
Update makefile.
|
2024-03-27 19:59:24 -07:00 |
|
mathan
|
3aca467516
|
Update iexport.
|
2024-03-27 16:18:18 -07:00 |
|
Max Ignatenko
|
ffa2faa420
|
Properly escape null character in the consumer too
|
2024-03-24 12:43:52 +00:00 |
|
Max Ignatenko
|
693ae1ba0a
|
Bump the limit on stored bad records to 500
|
2024-03-24 12:32:13 +00:00 |
|
mathan
|
d189912bcf
|
Update dash. Add alerting webhook.
|
2024-03-20 07:18:15 -07:00 |
|
Max Ignatenko
|
bae23a62d0
|
Don't try to upsert zero records
|
2024-03-20 13:31:14 +00:00 |
|
Max Ignatenko
|
542be03088
|
Add all PDSs to whitelist
|
2024-03-20 13:31:14 +00:00 |
|
mathan
|
dc17238a93
|
Update dash
|
2024-03-18 18:01:47 -07:00 |
|
Max Ignatenko
|
c3bd6d177a
|
Expand some terms used
|
2024-03-17 22:03:16 +00:00 |
|
Max Ignatenko
|
e8c816a3a3
|
Update FirstCursorSinceReset even if we received zero new records
|
2024-03-17 21:22:34 +00:00 |
|
Max Ignatenko
|
328a676e2a
|
Fix potential infinite loop for inactive repos after a cursor reset
|
2024-03-17 19:36:12 +00:00 |
|
Max Ignatenko
|
4c41389e9b
|
Detect CAR files with zero blocks and handle them accordingly
|
2024-03-17 17:42:09 +00:00 |
|
Max Ignatenko
|
638bdcf515
|
Add a check for zero bytes fetched from PDS
And log additional info when we're failing to extract rev from repo
bytes
|
2024-03-17 17:15:23 +00:00 |
|
mathan
|
bc895b6425
|
Update dash.
|
2024-03-17 08:52:28 -07:00 |
|
Max Ignatenko
|
553894dc6a
|
Switch to partial repo fetches
|
2024-03-17 15:34:26 +00:00 |
|
Max Ignatenko
|
346aa7c539
|
Document consistency model for repo data
|
2024-03-17 15:21:54 +00:00 |
|
Max Ignatenko
|
0cc11e75b2
|
Avoid updating records when then didn't actually change
|
2024-03-17 14:16:25 +00:00 |
|
Max Ignatenko
|
9c51a4621f
|
Start recording last rev for each repo
|
2024-03-13 22:28:54 +00:00 |
|
mathan
|
85259baa1c
|
Update dash.
|
2024-03-13 14:21:05 -07:00 |
|
mathan
|
9d429be424
|
Update dash.
|
2024-03-13 11:33:13 -07:00 |
|
Max Ignatenko
|
76a50524a0
|
Give postgres plenty of time to shutdown cleanly
|
2024-03-13 18:12:55 +00:00 |
|
Max Ignatenko
|
44d2b25951
|
Fix nil pointer deref
|
2024-03-13 16:38:58 +00:00 |
|
Max Ignatenko
|
e150f1da90
|
Populate FirstRevSinceReset if empty
|
2024-03-13 11:32:23 +00:00 |
|
Max Ignatenko
|
87d510e67a
|
Check repos against PDS cursor resets, instead of waiting for a first new even for them on firehose
|
2024-03-13 10:39:41 +00:00 |
|
Max Ignatenko
|
57aa4731e5
|
Properly update repo consistency metadata
Previously these two queries did nothing at all -_-
Note that this will trigger a full re-index of every repo seen on the
firehose.
|
2024-03-13 10:36:41 +00:00 |
|
Max Ignatenko
|
080cbb8b12
|
Bump WAL size limit to 8GB
|
2024-03-13 09:51:40 +00:00 |
|
mathan
|
75386c0235
|
Add export_blocks view. Add iexport + audit table.
|
2024-03-12 21:15:23 -07:00 |
|
mathan
|
e1ec43def7
|
Add count listitems.
|
2024-03-06 18:58:08 -08:00 |
|
mathan
|
dc80cd95ad
|
Update stats dash.
|
2024-03-05 14:17:46 -08:00 |
|
mathan
|
209813bd87
|
Update dash + csv_export.
|
2024-02-28 10:37:18 -08:00 |
|
mathan
|
439518fda1
|
Update dash.
|
2024-02-26 12:57:05 -08:00 |
|
Max Ignatenko
|
3a2c6a6f5b
|
Add PDS URL to error message
|
2024-02-26 07:27:15 +00:00 |
|
mathan
|
efc106453f
|
Fixed NPE, fixed failed lister migration.
|
2024-02-25 20:51:10 -08:00 |
|
mathan
|
067be8ec83
|
Update dash. Add smartmon to readme.
|
2024-02-25 18:04:06 -08:00 |
|
Max Ignatenko
|
95ca4f0eef
|
Use materialized views for export to csv
|
2024-02-25 22:05:34 +00:00 |
|
mathan
|
2172651870
|
Merge branch 'main' of github.com:uabluerail/indexer
|
2024-02-25 09:05:50 -08:00 |
|
mathan
|
95abc3691f
|
Update dash, added smartmon
|
2024-02-25 09:05:48 -08:00 |
|
Max Ignatenko
|
b4a0c268d6
|
Bump max_parallel_workers_per_gather to 8
|
2024-02-25 11:35:10 +00:00 |
|
Max Ignatenko
|
fa592c2b34
|
Add a migration to enable compression on already existing tables
|
2024-02-25 11:24:40 +00:00 |
|
Max Ignatenko
|
3432121be2
|
Give postgres a bit more RAM
|
2024-02-25 11:06:47 +00:00 |
|
Max Ignatenko
|
0b6e326762
|
Set an explicit default for Deleted
|
2024-02-25 10:30:49 +00:00 |
|