|
|
65eac1975c
|
Move feed fetching to crawl_feed job, DomainRequestLimiter
`DomainRequestLimiter` is a distributed version of `DomainLocks` based
on redis.
|
2024-08-26 01:12:18 -04:00 |
|
|
|
9c75a88c69
|
Start of a crawl_feed job
|
2024-08-25 22:24:02 -04:00 |
|
|
|
a3450e202a
|
Working apalis cron and worker with 0.6.0-rc.5
Also renamed `pool` variables throughout codebase to `db` for clarity.
|
2024-08-21 01:21:45 -04:00 |
|
|
|
764d3f23b8
|
WIP add apalis & split up main process
|
2024-07-27 13:55:08 -04:00 |
|
|
|
4837cbb903
|
Add crawl metadata to feed & improve model interface
|
2023-07-15 00:40:10 -04:00 |
|
|
|
ae8f15f19b
|
Add very basic crawl job
Loops through feeds and adds items from each feed.
|
2023-05-09 23:55:42 -04:00 |
|
