![]() ![]() Use generators to lower memory footprint and gain a bit more speed The script sticks to the url provided and does not dive into subdomains of the given domainĮven if encounters internal redirect like -> Possible enhancements It’s single-threaded script that walks every page it gets and it’s The whole site and generating simple sitemap. The NightCrawler is site crawling/spider tool to gather links at the given domain by walking through ![]()
0 Comments
Leave a Reply. |