.Google.com's John Mueller answered a concern about why Google marks web pages that are disallowed from crawling by robots.txt as well as why the it's safe to neglect the related Browse Console reports about those crawls.Bot Visitor Traffic To Query Parameter URLs.The individual asking the question recorded that crawlers were creating hyperlinks to non-existent concern specification Links (? q= xyz) to web pages with noindex meta tags that are likewise blocked out in robots.txt. What urged the concern is actually that Google.com is creeping the hyperlinks to those pages, obtaining shut out by robots.txt (without envisioning a noindex robots meta tag) then acquiring reported in Google Explore Console as "Indexed, though blocked by robots.txt.".The person asked the following inquiry:." Yet listed below's the large question: why will Google mark web pages when they can't also view the web content? What's the benefit in that?".Google's John Mueller validated that if they can't creep the web page they can't view the noindex meta tag. He likewise produces an appealing mention of the site: hunt driver, encouraging to ignore the results considering that the "normal" users will not find those end results.He wrote:." Yes, you're appropriate: if our company can not crawl the webpage, we can not view the noindex. That pointed out, if we can't creep the webpages, at that point there's not a great deal for our company to index. So while you could find a few of those pages with a targeted site:- question, the average consumer will not find them, so I would not bother it. Noindex is also great (without robots.txt disallow), it just implies the URLs will wind up being actually crawled (and find yourself in the Search Console file for crawled/not indexed-- neither of these conditions cause concerns to the rest of the internet site). The integral part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in using the Web site: hunt progressed search driver for analysis causes. Among those factors is actually considering that it is actually not hooked up to the normal hunt index, it is actually a distinct factor completely.Google's John Mueller discussed the internet site search driver in 2021:." The brief response is actually that a web site: question is actually certainly not meant to be complete, neither used for diagnostics reasons.An internet site concern is actually a certain sort of hunt that restricts the results to a particular website. It's essentially simply the word website, a colon, and then the internet site's domain.This inquiry limits the outcomes to a specific site. It's certainly not indicated to become a complete collection of all the pages coming from that web site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of scenarios where a robot is actually connecting to non-existent pages that are getting uncovered by Googlebot.3. URLs along with the noindex tag will create a "crawled/not indexed" item in Explore Console which those will not possess an adverse effect on the rest of the web site.Read through the concern and respond to on LinkedIn:.Why will Google.com mark web pages when they can't even find the material?Featured Image through Shutterstock/Krakenimages. com.