Max Ignatenko
|
f315059994
|
Avoid querying too few repos from each single PDS
|
2024-05-06 21:14:18 +01:00 |
|
Max Ignatenko
|
ff0ea08296
|
Implement commit signature validation
|
2024-04-06 21:50:51 +01:00 |
|
Max Ignatenko
|
ffa2faa420
|
Properly escape null character in the consumer too
|
2024-03-24 12:43:52 +00:00 |
|
Max Ignatenko
|
e8c816a3a3
|
Update FirstCursorSinceReset even if we received zero new records
|
2024-03-17 21:22:34 +00:00 |
|
Max Ignatenko
|
328a676e2a
|
Fix potential infinite loop for inactive repos after a cursor reset
|
2024-03-17 19:36:12 +00:00 |
|
Max Ignatenko
|
4c41389e9b
|
Detect CAR files with zero blocks and handle them accordingly
|
2024-03-17 17:42:09 +00:00 |
|
Max Ignatenko
|
638bdcf515
|
Add a check for zero bytes fetched from PDS
And log additional info when we're failing to extract rev from repo
bytes
|
2024-03-17 17:15:23 +00:00 |
|
Max Ignatenko
|
553894dc6a
|
Switch to partial repo fetches
|
2024-03-17 15:34:26 +00:00 |
|
Max Ignatenko
|
0cc11e75b2
|
Avoid updating records when then didn't actually change
|
2024-03-17 14:16:25 +00:00 |
|
Max Ignatenko
|
87d510e67a
|
Check repos against PDS cursor resets, instead of waiting for a first new even for them on firehose
|
2024-03-13 10:39:41 +00:00 |
mathan
|
78a17bf238
|
Merge branch 'main' of github.com:uabluerail/indexer
|
2024-02-22 18:56:19 -08:00 |
|
mathan
|
db425b1d5f
|
Fix view in migration. Add by lang metric to consumer.
|
2024-02-22 18:54:29 -08:00 |
|
Max Ignatenko
|
a20ddf0717
|
Fix the fucking regexp
|
2024-02-22 18:16:43 +00:00 |
|
Max Ignatenko
|
8f32c494f7
|
Add whitelist for PDS hosts and update repo PDS pointer on appropriate occasions
|
2024-02-22 12:17:21 +00:00 |
|
mathan
|
600dac7694
|
Added metric for posts by lang (olivčykom).
|
2024-02-21 18:09:12 -08:00 |
|
Max Ignatenko
|
b2003530ba
|
Handle repos with unknown PDS
|
2024-02-21 09:35:25 +00:00 |
|
Max Ignatenko
|
a0934c360e
|
fix division by zero
|
2024-02-19 18:34:31 +00:00 |
|
Max Ignatenko
|
1038ca3bea
|
Add AtRev column to only overwrite records with a newer version
|
2024-02-17 14:29:45 +00:00 |
|
Max Ignatenko
|
1d3c6edf0a
|
Fix panic on closing already closed channel
|
2024-02-17 12:26:57 +00:00 |
|
Max Ignatenko
|
04d521f58c
|
Increase request timeout, default 30 seconds are not enough to download some repos
|
2024-02-16 16:47:36 +00:00 |
|
Max Ignatenko
|
4626b8b9ca
|
Limit the number of retries when failing to index a repo
|
2024-02-16 15:23:34 +00:00 |
|
Max Ignatenko
|
8313c74482
|
Remove deleted_at
|
2024-02-15 20:29:08 +00:00 |
|
Max Ignatenko
|
8561d90caf
|
A bit of query optimization
|
2024-02-15 18:39:29 +00:00 |
|
Max Ignatenko
|
63a767d890
|
Import
|
2024-02-15 16:10:39 +00:00 |