Checks whether search engines are allowed to crawl a site, validates its XML sitemaps, and tests whether a specific URL is blocked.