You can scrape the site, as long as it doesn't impact the server too much

Automated web scraping is a method of extracting information from a website. The service authorises scraping if it doesn't impact its server(s) too much.

Classification: good

Weight: 50



Service Title Rating Status Author
Fur Affinity "<p>Scrape our services, content, or content belonging to those using our services in a way that negatively impacts site performance.</p> <p>" DECLINED LuneSirius Lv. 9 Curator
Nextcloud "Do not strain the forum with technical requests or requests designed to impose an unreasonable load on the forum infrastructure.</li>" APPROVED AgnesDeLion Lv. 83 Staff
Pixel8Earth "Additionally, you shall not: (i) take any action that imposes or may impose (as determined by Pixel8Earth in its sole discretion) an unreasonable or disproportionately large load on Pixel8Earth's (or its third party providers') infrastructure." APPROVED arlo Lv. 13 Staff
FanFiction "You agree not to use or launch any automated system, including without limitation, "robots," "spiders," or "offline readers," that accesses the Service in a manner that sends more request messages to the FanFiction.Net servers in a given period of time than a human can reasonably produce in the same period by using a conventional on-line web browser." APPROVED Holonium Lv. 8
Parler ".You may not interfere with the Services in any way, such as by accessing the ,Services through automated means in a manner that puts excessive demand on the ,Services" APPROVED AgnesDeLion Lv. 83 Staff
CodeSandbox "interfere with, disrupt, or create an undue burden on servers or networks connected to the Service, or violate the regulations, policies or procedures of such networks." APPROVED private prawn Lv. 25 Curator
LVFS You can scrape the site, as long as it doesn't impact the server too much APPROVED hughsient Lv. 2
Blu-ray.com "Manual content scraping is allowed, as long as it is not done "en masse" and/or systematical in some way.</li> <br> <br>" APPROVED blazyrawr Lv. 8
Reddit "(we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit’s prior written consent is prohibited);" APPROVED Dr_Jeff Lv. 101 Staff
SourceHut "<p>You may use automated tools to access SourceHut data in bulk (i.e. crawlers, robots, spiders, etc) provided that:</p> <ol> <li>You obey the rules set forth in robots.txt</li> <li>Your software uses a User-Agent header which clearly identifies your software and its operators, including your contact information</li> <li>You request data from SourceHut at a rate which does not adversely affect the performance of the services for normal users</li>" APPROVED Dr_Jeff Lv. 101 Staff
CodeAI "NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of CodeAI is expressly prohibited" APPROVED Dr_Jeff Lv. 101 Staff
ToS;DR "Note: We do allow the use of automated tools so long as they do not produce excessive amounts of traffic. For example, running one <code>nmap</code> scan against one host is allowed, but sending 65,000 requests in two minutes using Burp Suite Intruder is excessive." APPROVED Dr_Jeff Lv. 101 Staff


Comments:
No comments found