|
|
c1d2c3786d
|
hunter/main: use jemalloc
|
2022-12-03 01:47:44 +01:00 |
|
|
|
fed37ce56f
|
hunter/main: enable trust-dns
seems to use even more memory but perhaps we're no longer bound by
worker threads?
|
2022-12-03 01:47:07 +01:00 |
|
|
|
cd132a67bc
|
cave/feed: clean up with EncodedPost
|
2022-12-02 23:05:35 +01:00 |
|
|
|
f7ebb08274
|
cave/feed: wrap Post in EncodablePost
|
2022-12-02 22:03:38 +01:00 |
|
|
|
51a21d3fbc
|
hunter: Arcify host
|
2022-12-02 00:50:01 +01:00 |
|
|
|
69ba3ed657
|
posts_cache: use only 1 Arc
|
2022-12-02 00:30:10 +01:00 |
|
|
|
f03a83e9cd
|
hunter/tag_trimmer: act more gently
|
2022-11-24 19:25:03 +01:00 |
|
|
|
77997e9720
|
hunter: redisable reqwst feature rustls
|
2022-11-24 19:24:44 +01:00 |
|
|
|
00ded7dc8e
|
hunter: add posts_cache
|
2022-11-23 23:59:35 +01:00 |
|
|
|
91dcf74609
|
hunter/worker: free RobotsTxt earlier
|
2022-11-18 20:13:34 +01:00 |
|
|
|
3ccc6278d9
|
hunter/main: bump systemd::extend_timeout() for introducing hosts from redis
|
2022-11-18 17:43:35 +01:00 |
|
|
|
73b3641605
|
hunter/main: extend systemd StartTimeout
|
2022-11-15 19:02:14 +01:00 |
|
|
|
e622eace28
|
hunter/main: introduce hosts from redis only on production
|
2022-11-14 22:45:33 +01:00 |
|
|
|
4684274d6d
|
hunter/trend_setter: bump MIN_INTERVAL
|
2022-11-14 21:52:25 +01:00 |
|
|
|
d65abb82c3
|
hunter/main: reenable introducing hosts from redis
|
2022-11-14 19:30:19 +01:00 |
|
|
|
e49ae907e5
|
cave/trend_tag: switch from by_hours to hour_users algorithm
|
2022-11-14 03:14:12 +01:00 |
|
|
|
48156b640c
|
cave/store: bump max_pool_size
|
2022-11-12 02:47:21 +01:00 |
|
|
|
d35004c4d1
|
hunter/tag_timmer: add
|
2022-11-12 01:02:44 +01:00 |
|
|
|
663db47973
|
add working streaming
|
2022-11-11 22:13:33 +01:00 |
|
|
|
0e5fab858e
|
hunter/worker: eval robot.delay
|
2022-11-11 19:21:44 +01:00 |
|
|
|
53ae6e6ba1
|
hunter/worker: refactor, prepare stream, add robots.txt support
|
2022-11-11 19:00:37 +01:00 |
|
|
|
e620c69a32
|
hunter: refactor worker to prepare for more fetch steps
|
2022-11-11 18:08:13 +01:00 |
|
|
|
51a4f7c5ea
|
make languages depend on trend pool size
|
2022-11-11 17:11:13 +01:00 |
|
|
|
adedb40105
|
cave/store: add RedisPool
|
2022-11-10 15:22:45 +01:00 |
|
|
|
aa715aa377
|
hunter/trend_setter: keep score <= 0 out of pools
|
2022-11-10 03:28:51 +01:00 |
|
|
|
d42615b44e
|
hunter: fix debug output
|
2022-11-09 19:25:51 +01:00 |
|
|
|
1092194bba
|
move all trend filters from gatherer to hunter
|
2022-11-09 19:25:07 +01:00 |
|
|
|
e224c14c86
|
gatherer: adopt trend pools
|
2022-11-09 19:03:03 +01:00 |
|
|
|
0c23fad76d
|
hunter/trend_setter: implement maintaining the pools
|
2022-11-09 18:11:02 +01:00 |
|
|
|
a98a669086
|
hunter: begin trend_setter
|
2022-11-08 00:43:46 +01:00 |
|
|
|
a5a5a10d15
|
hunter/redis_store: move into cave::store
|
2022-11-07 22:07:07 +01:00 |
|
|
|
3728fa0d5e
|
hhunter/redis_store: work around bogus language
|
2022-11-07 04:30:14 +01:00 |
|
|
|
e9d3c9aed0
|
hunter/scheduler: add jitter to interval on no new posts
|
2022-11-07 04:08:31 +01:00 |
|
|
|
566aecc3dd
|
hunter/main: add more systemd status
|
2022-11-07 03:30:12 +01:00 |
|
|
|
24a2b94dcd
|
hunter/redis_store: clip language to 2 chars
|
2022-11-07 03:29:53 +01:00 |
|
|
|
88d0abc0a0
|
hunter/redis_store: start writing language stats
|
2022-11-07 03:26:53 +01:00 |
|
|
|
1cf7e200ab
|
hunter: rework scheduling algorithm
|
2022-11-07 00:58:28 +01:00 |
|
|
|
7913f6fa5f
|
hunter/worker: fetch a maximum of 40 posts
|
2022-11-06 22:30:43 +01:00 |
|
|
|
bd7604f639
|
cave: init
|
2022-11-05 20:51:18 +01:00 |
|
|
|
6ac923f270
|
hunter/redis_store: ignore future or ancient posts
|
2022-11-05 19:45:59 +01:00 |
|
|
|
02f047d469
|
hunter: add more logging
|
2022-11-05 03:29:07 +01:00 |
|
|
|
2fdd0e079b
|
hunter/worker: don't pass host to save_post()
|
2022-11-05 03:26:34 +01:00 |
|
|
|
44a3d68201
|
hunter/worker: simplify error case, hardcode interval_after_error
|
2022-11-05 03:25:30 +01:00 |
|
|
|
36adb5f324
|
don't discard foreign posts
|
2022-11-04 15:58:03 +01:00 |
|
|
|
8d097e2f33
|
don't ignore foreign posts
|
2022-11-04 15:50:25 +01:00 |
|
|
|
36b1d688db
|
s/Post.url/Post.uri/
|
2022-11-04 15:50:00 +01:00 |
|
|
|
9c51946284
|
worker: lower DEFAULT_INTERVAL from 3600s to 60s
|
2022-11-03 21:18:15 +01:00 |
|
|
|
c7a92a519d
|
store known hosts in redis only after successful fetch
|
2022-11-03 21:17:21 +01:00 |
|
|
|
4222ce97dc
|
improve debug logging
|
2022-11-03 20:59:36 +01:00 |
|
|
|
98fade50f4
|
limit connections
|
2022-11-03 20:48:36 +01:00 |
|
|
|
9e9e651b10
|
add nixos-module
|
2022-11-03 19:49:00 +01:00 |
|
|
|
ca66146eb3
|
save hosts to/load from redis
|
2022-11-03 18:58:37 +01:00 |
|
|
|
d0a5f11040
|
feed: lower-case hosts
|
2022-11-03 17:37:06 +01:00 |
|
|
|
420d53aa61
|
notify systemd
|
2022-11-03 17:34:29 +01:00 |
|
|
|
748ccd4f59
|
use env_logger
|
2022-11-03 17:22:21 +01:00 |
|
|
|
63ee84b93c
|
move into subdir hunter/
|
2022-11-03 17:17:39 +01:00 |
|
