 Astro
|
33c910ff7a
|
hunter: redesign hosts introduction in worker to add scanning reblog mentions
|
2023-04-24 19:27:13 +02:00 |
|
Astro
|
0bef428894
|
hunter: remove unnecessary cloning
|
2023-04-24 19:21:15 +02:00 |
|
Astro
|
fb21cf0335
|
hunter/worker: learn hosts from reblogged accounts
|
2023-03-09 00:58:30 +01:00 |
|
Astro
|
ec368e4557
|
hunter/worker: output more stream stats
|
2023-03-09 00:56:48 +01:00 |
|
Astro
|
f6e607d6ec
|
hunter/worker: eliminate useless RwLock
|
2023-03-04 00:59:09 +01:00 |
|
Astro
|
f32ec17704
|
hunter/worker: let stream post_count increase new_post_ratio
|
2023-03-04 00:50:35 +01:00 |
|
Astro
|
5f1d9862c5
|
hunter/worker: update mean_interval with stream_avg_interval if post_count > 0
|
2023-03-03 21:36:58 +01:00 |
|
Astro
|
b2a5228f98
|
hunter/worker: fix stream_avg_interval calculation
|
2023-03-03 21:24:28 +01:00 |
|
Astro
|
6231884d36
|
hunter: estimate stream_avg_interval
|
2023-03-03 20:49:34 +01:00 |
|
Astro
|
f2e65fd6cd
|
hunter: split into butcher
|
2023-01-22 00:05:10 +01:00 |
|
Astro
|
df50718724
|
Revert "hunter/worker: invert hunter_new_post_ratio metric"
This reverts commit 5260090b19.
|
2022-12-29 03:17:03 +01:00 |
|
Astro
|
5260090b19
|
hunter/worker: invert hunter_new_post_ratio metric
|
2022-12-27 20:34:34 +01:00 |
|
Astro
|
704b883de2
|
hunter/worker: adjust new posts log level
|
2022-12-27 19:05:53 +01:00 |
|
Astro
|
ff9c64ec1a
|
hunter: add more metrics
|
2022-12-27 19:03:09 +01:00 |
|
Astro
|
aeec4045a0
|
hunter/worker: catch runaway metrics
|
2022-12-26 04:14:10 +01:00 |
|
Astro
|
ce3f184d04
|
hunter: add metrics for prometheus
|
2022-12-26 03:46:11 +01:00 |
|
Astro
|
f5fb098bd2
|
switch from log to tracing
|
2022-12-26 03:05:00 +01:00 |
|
Astro
|
cd132a67bc
|
cave/feed: clean up with EncodedPost
|
2022-12-02 23:05:35 +01:00 |
|
Astro
|
f7ebb08274
|
cave/feed: wrap Post in EncodablePost
|
2022-12-02 22:03:38 +01:00 |
|
Astro
|
51a21d3fbc
|
hunter: Arcify host
|
2022-12-02 00:50:01 +01:00 |
|
Astro
|
00ded7dc8e
|
hunter: add posts_cache
|
2022-11-23 23:59:35 +01:00 |
|
Astro
|
91dcf74609
|
hunter/worker: free RobotsTxt earlier
|
2022-11-18 20:13:34 +01:00 |
|
Astro
|
663db47973
|
add working streaming
|
2022-11-11 22:13:33 +01:00 |
|
Astro
|
0e5fab858e
|
hunter/worker: eval robot.delay
|
2022-11-11 19:21:44 +01:00 |
|
Astro
|
53ae6e6ba1
|
hunter/worker: refactor, prepare stream, add robots.txt support
|
2022-11-11 19:00:37 +01:00 |
|
Astro
|
e620c69a32
|
hunter: refactor worker to prepare for more fetch steps
|
2022-11-11 18:08:13 +01:00 |
|
Astro
|
0c23fad76d
|
hunter/trend_setter: implement maintaining the pools
|
2022-11-09 18:11:02 +01:00 |
|
Astro
|
a98a669086
|
hunter: begin trend_setter
|
2022-11-08 00:43:46 +01:00 |
|
Astro
|
a5a5a10d15
|
hunter/redis_store: move into cave::store
|
2022-11-07 22:07:07 +01:00 |
|
Astro
|
1cf7e200ab
|
hunter: rework scheduling algorithm
|
2022-11-07 00:58:28 +01:00 |
|
Astro
|
7913f6fa5f
|
hunter/worker: fetch a maximum of 40 posts
|
2022-11-06 22:30:43 +01:00 |
|
Astro
|
02f047d469
|
hunter: add more logging
|
2022-11-05 03:29:07 +01:00 |
|
Astro
|
2fdd0e079b
|
hunter/worker: don't pass host to save_post()
|
2022-11-05 03:26:34 +01:00 |
|
Astro
|
44a3d68201
|
hunter/worker: simplify error case, hardcode interval_after_error
|
2022-11-05 03:25:30 +01:00 |
|
Astro
|
8d097e2f33
|
don't ignore foreign posts
|
2022-11-04 15:50:25 +01:00 |
|
Astro
|
9c51946284
|
worker: lower DEFAULT_INTERVAL from 3600s to 60s
|
2022-11-03 21:18:15 +01:00 |
|
Astro
|
c7a92a519d
|
store known hosts in redis only after successful fetch
|
2022-11-03 21:17:21 +01:00 |
|
Astro
|
748ccd4f59
|
use env_logger
|
2022-11-03 17:22:21 +01:00 |
|
Astro
|
63ee84b93c
|
move into subdir hunter/
|
2022-11-03 17:17:39 +01:00 |