Linkcrawler
bySchollz
Cross-platform persistent and distributed web crawler :link:
Web CrawlingOpen Source Developer ToolsOpen Source
About
linkcrawler is persistent because the queue is stored in a remote database that is automatically re-initialized if interrupted. linkcrawler is distributed because multiple instances of linkcrawler will work on the remotely stored queue, so you can start as many crawlers as you want on separate machines to speed along the process. linkcrawler is also fast because it is threaded and uses connection pools.
Open Source Health
- Stars
- 113
- Forks
- 9
- License
- MIT
- Last commit
- 9 years ago
Go
Alternatives to Linkcrawler
Deep Researchby David ZhangAn AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine…- select.rsby Utkarsh KukretiA Rust library to extract useful data from HTML documents, suitable for web scraping.
Scrapelyby ScrapyA pure-python HTML screen-scraping library- Faster Than Requestsby Juan CarlosFaster requests on Python 3
MechanicalSoupby MechanicalSoupA Python library for automating interaction with websites.
Resources & Links
Related Categories
Using Linkcrawler?
Track its cost next to the rest of your stack and get a reminder before it renews.
Add to my stackVendor
Schollz
Publisher of Hostyoself, e2ecp and PAKE
