Grab Site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Web CrawlingOpen Source Developer ToolsOpen Source
About
grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses a fork of wpull for crawling.
Open Source Health
- Stars
- 1,611
- Forks
- 157
- License
- Not stated
- Last commit
- 1 years ago
Python
Alternatives to Grab Site
Deep Researchby David ZhangAn AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine…- select.rsby Utkarsh KukretiA Rust library to extract useful data from HTML documents, suitable for web scraping.
Scrapelyby ScrapyA pure-python HTML screen-scraping library- Faster Than Requestsby Juan CarlosFaster requests on Python 3
MechanicalSoupby MechanicalSoupA Python library for automating interaction with websites.
Resources & Links
Related Categories
Using Grab Site?
Track its cost next to the rest of your stack and get a reminder before it renews.
Add to my stackVendor
Archive Team
We Are Going To Rescue Your Shit