Commit Graph

8 Commits

Author SHA1 Message Date
276f0e17a8 Remove self from Crawls and Imports in actors at end of task
In case the user never listens to the stream so that I do not create
inifinitely growing hashmaps in the server memory.
2023-08-29 23:30:00 -04:00
eddf39b62e Added importer actor, file upload still not working 2023-07-22 22:54:57 -04:00
e6a37703be Use tokio::sync::mutex for Crawls 2023-07-19 23:03:35 -04:00
d17f909312 Add CrawlScheduler actor, shared client w/ last modified headers 2023-07-15 21:40:31 -04:00
923776d7a1 Add DomainLocks to serialize requests to each domain
All async tasks must wait 1 second since the last request to make
another to the same domain.
2023-07-14 00:55:35 -04:00
f13c7e5e70 Add an async actor FeedCrawler for fetching feed details
Currently, this allows the browser to subscribe to the response of the
asynchronous crawl after they add a new feed.

Eventually I will also use this in the main scheduled crawls. Right now,
it only upserts feed metadata based on the parsed feed.
2023-07-09 21:30:23 -04:00
6713a7a440 Complete log stream implementation
Sets up a watch channel to send tracing lines from tracing-subscriber to
receivers in a axum handler which streams Server Sent Events to any
number of connected /log/stream clients.
2023-06-03 19:03:58 -04:00
f4da3f3ab3 Adding title to config and utilizing it in layout 2023-06-02 00:08:20 -04:00