Abusing Robots.txt<\/strong><\/h3>\n\n\n\n

Authorized ways of using online data can also be regulated by robots.txt. It\u2019s the file containing information on how and which data available on the website can be crawled or scraped. If the document prohibits scraping, the only option to do it legally is to ask the website owner for official permission.<\/p>\n\n\n\n

Abusing Terms of Service<\/strong><\/h3>\n\n\n\n

Similar to robots.txt, every website has its own Terms of Service (ToS). So, each time prior to scraping a particular website you should make sure that it\u2019s allowed. If there\u2019s nothing about web scraping being prohibited, you\u2019re safe. Otherwise, you should also receive permission in writing.<\/p>\n\n\n\n

Exceeding crawl rate<\/strong><\/h3>\n\n\n\n

Websites are made for humans. This is why most of the bots try to imitate human behavior in order to prevent exceeding the crawl rate with too intensive scraping. However, on the other hand – advanced, human-like bots tend to process much more data and can significantly slow down the page. So, use bots that will help you to keep the right balance between maintaining crawl rate on a reasonable level and preventing server overload.<\/p>\n\n\n\n

Examples of legal and illegal web scaping<\/h2>\n\n\n\n

To provide you with a better understanding of the legal issues of web scraping, let\u2019s catch a glimpse of a few cases in which web scraping can be used in either right or wrong manner.<\/p>\n\n\n\n

For example, bots can be used to search for YouTube video titles or descriptions. It can be legally scraped, downloaded, and saved into a file. However, the videos cannot be re-posted on our own site, as this would be copyright abuse.<\/p>\n\n\n\n

To give you another example, a web crawler is able to scrape the names of users available publicly on social media. However, you can\u2019t log in to their Facebook or LinkedIn accounts to retain protected data as it\u2019s not permitted by the rules of the service. <\/p>\n\n\n\n

What are the other tools that companies can use to protect their data from being scraped? For example, they can implement CAPTCHA verification technology. They can also use \u201crate-throttling\u201d to protect their websites from downloading too many web pages at once. This helps to avoid malicious bots which can overload the site.<\/p>\n\n\n\n

Legal case: LinkedIn vs HiQ<\/h2>\n\n\n\n

Although cases in which web scraping is legal are pretty defined, a few companies still decide to protect their data by legal means. An example is LinkedIn who took HiQ to court.<\/p>\n\n\n\n

HiQ is the company scraping public information from LinkedIn profiles to provide businesses with data and insights on employees. As LinkedIn decided to launch a similar tool themselves, in 2017 they sued HiQ for unauthorized data collection. Long story short, in late 2019 the US Court of Appeals claimed that there was no CFAA abuse, and LinkedIn was asked to stop applying blocking measures against HiQ.<\/p>\n\n\n\n

This case proves that any data which is publicly available and not copyrighted can be scraped freely. However, it still cannot be used for unlimited commercial purposes.<\/p>\n\n\n\n

Summary<\/h2>\n\n\n\n

To sum up, web scraping is perfectly legal unless you use it unethically. It\u2019s just a tool that is not harmful itself but can become illegal when used with the wrong intention. However, if you have any doubts, you can always ask the lawyer for professional advice.<\/p>\n\n\n\n

If you need to scrape data for business and receive it in a processed, ready-to-use form, feel free to contact us. We will analyze your case and let you know what kind of data you can scrape and what is recommended in your case. <\/p>\n","protected":false},"excerpt":{"rendered":"

Web scraping was a breakthrough for a number of businesses around the […]<\/p>\n","protected":false},"author":1,"featured_media":82,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[15,5],"class_list":["post-81","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-webscraping","tag-legal-web-scraping","tag-web-scraping"],"yoast_head":"\n