Collaborate Disseminate
I’m looking for a way to check for emails on the pages of my domain. Is there a crawler that can do that?
Continue reading Tool to automatically collect email from crawled pages [closed]
I’m writing a simple crawler with node.js, which searches for web pages and conditionally executes any JavaScript present.
The problem is that in doing so, I execute code form untrusted sources in my node.js environment. Can running untrus… Continue reading Threats that JavaScript poses to a web crawler
I’m writing a simple crawler with node.js, which searches for web pages and conditionally executes any JavaScript present.
The problem is that in doing so, I execute code form untrusted sources in my node.js environment. Can running untrus… Continue reading Threats that JavaScript poses to a web crawler
I am a website developer (mainly using MVC.NET). Recently, we have been contacted by a hacker. He claimed that he knows our admin URL. The problem is we do not publish or put the admin URL anywhere on our webpage. The only place where the … Continue reading How does msnbot keep finding my unpublished admin url?
I would assume that if someone is daily crawling a website in a way which is heavily laborious on the server it may overkill that server.
Are malicious crawling bots a form of application layer DoS?
Continue reading Are malicious crawling bots a form of application layer DoS?
So I have some data that isn’t overly sensitive, but I’m still on the fence on whether or not we should invest the additional time into managing it as a private resource, vs just publicly available.
The data (images & pdfs) are to be h… Continue reading Risks of web crawlers on public buckets
So I have some data that isn’t overly sensitive, but I’m still on the fence on whether or not we should invest the additional time into managing it as a private resource, vs just publicly available.
The data (images & pdfs) are to be h… Continue reading Risks of web crawlers on public buckets
In the current day and age we have the problem of malicious/spam crawlers and similar concerns.
My suggestion would be implementing cookie support for crawling and by that I mean giving specific cookies with crawler ID (at best refreshed u… Continue reading Why can’t you give special security cookies to a specific crawler so that they could securely crawl the website?
I want to scrape our university’s learning platform website, to let myself know via notifications when a new entry added to any lesson.
But, I’m scared that they’ll put robots.txt afterwards and sue me or something, I don’t know. I just do… Continue reading How do I prove that robots.txt was not provded
Say you have the following files/folder in your webserver:
public_html/index.html
public_html/test/index.html
public_html/test/foo-randomString.jpg
For an average user, foo-randomString.jpg can’t be seen unless they know the exact filena… Continue reading Can a web crawler still see files in a directory even with an index file? [closed]