PHNo liabilities
Elite
You do not have permission to view the full content of this post. Log in or register now.
What are good ways to protect website content from crawlers?
1) Have your server reject a large series of requests from the same IP address in a given time period. Google is very effective at this -- if you attempt to crawl any of their search results they'll block your ip after about 30 seconds.
2) Block requests with HTTP headers from certain user agents. You can do this by creating a robots.txt[1] file.
3) Hide content behind something that forces human involvement like a CAPTCHA[2].
4) Pull important content to the page after it loads with AJAX.
5) Watermark your content so it will be easy to tell its yours on other pages.
6) Add detailed clauses in your terms of use that discourage crawling, and threaten legal action against those that do (like Craigslist has done recently with Padmapper[3]).
Realistically however, there is no way to stop web crawlers 100% of the time. If you're content is on the web, it's generally quite easy for it to be stolen.
From Quora Digest
I know this is a dumb solution but for those who are not into computers this is helpful.. If youre sharing sensitive links on facebook all you have to do is write your links on your notes app then screenshot.. then send it to others via messenger, they need to retype the link to avoid facebook bot to open the link..