|
|
3d5c0b78c3
|
Upgrade apalis, add fred pool to state, start publishing in jobs
|
2024-09-22 13:44:55 -04:00 |
|
|
|
6912ef9017
|
Add crawl_entry job
|
2024-08-27 21:54:14 -04:00 |
|
|
|
65eac1975c
|
Move feed fetching to crawl_feed job, DomainRequestLimiter
`DomainRequestLimiter` is a distributed version of `DomainLocks` based
on redis.
|
2024-08-26 01:12:18 -04:00 |
|
|
|
9c75a88c69
|
Start of a crawl_feed job
|
2024-08-25 22:24:02 -04:00 |
|
|
|
a3450e202a
|
Working apalis cron and worker with 0.6.0-rc.5
Also renamed `pool` variables throughout codebase to `db` for clarity.
|
2024-08-21 01:21:45 -04:00 |
|
|
|
764d3f23b8
|
WIP add apalis & split up main process
|
2024-07-27 13:55:08 -04:00 |
|
|
|
4837cbb903
|
Add crawl metadata to feed & improve model interface
|
2023-07-15 00:40:10 -04:00 |
|
|
|
ae8f15f19b
|
Add very basic crawl job
Loops through feeds and adds items from each feed.
|
2023-05-09 23:55:42 -04:00 |
|
