No internet connection
  1. Home
  2. Ideas / Feature Requests

Use robots.txt for initial crawl list (option)

By Leon Stafford @leonstafford2018-12-12 15:51:10.165Z

As per title, if a robots.txt file exists within the WordPress site, this is probably a good list to use for the initial crawl list and may be faster to read than the database.

Expanding the "URL Detection" mechanism to allow for robots.txt and other items, togglable, would be nice to have:

  • existing methods (WP posts, pages, archives, media URLs, vendor URLs, like custom permalinks plugin)
  • robots.txt
  • sitemap(s)
  • 0 replies