thallada/crawlnicle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
44 Commits 3 Branches 0 Tags
ae959219668523166c4133269657df8f35c97e49
Commit Graph

6 Commits

Author SHA1 Message Date
Tyler Hallada
ae95921966 Update feed last_crawl_error & crawl_interval_minutes 2023-07-15 01:30:45 -04:00
Tyler Hallada
4837cbb903 Add crawl metadata to feed & improve model interface 2023-07-15 00:40:10 -04:00
Tyler Hallada
0fa0cfc508 Only crawl new entries 2023-07-14 21:27:23 -04:00
Tyler Hallada
923776d7a1 Add DomainLocks to serialize requests to each domain 
All async tasks must wait 1 second since the last request to make
another to the same domain.
 2023-07-14 00:55:35 -04:00
Tyler Hallada
b7efc61cfc Add EntryCrawler that uses readability lib 
Actors delegating to actors baybeeee
 2023-07-11 01:52:38 -04:00
Tyler Hallada
f13c7e5e70 Add an async actor FeedCrawler for fetching feed details 
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.

Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
 2023-07-09 21:30:23 -04:00