Use robots.txt for initial crawl list (option)
As per title, if a robots.txt file exists within the WordPress site, this is probably a good list to use for the initial crawl list and may be faster to read than the database.
Expanding the "URL Detection" mechanism to allow for robots.txt and other items, togglable, would be nice to have:
- existing methods (WP posts, pages, archives, media URLs, vendor URLs, like custom permalinks plugin)
- 0 replies