.Google.com's John Mueller responded to an inquiry concerning why Google.com marks webpages that are actually refused coming from creeping by robots.txt and also why the it is actually safe to dismiss the associated Look Console reports regarding those crawls.Crawler Traffic To Concern Criterion URLs.The person talking to the inquiry chronicled that robots were making web links to non-existent query criterion URLs (? q= xyz) to webpages with noindex meta tags that are actually also shut out in robots.txt. What cued the question is actually that Google is actually creeping the web links to those web pages, acquiring blocked out through robots.txt (without envisioning a noindex robots meta tag) after that obtaining reported in Google.com Explore Console as "Indexed, though obstructed through robots.txt.".The individual inquired the observing concern:." However right here's the big concern: why will Google.com mark web pages when they can not even view the content? What's the benefit in that?".Google.com's John Mueller verified that if they can't crawl the page they can not see the noindex meta tag. He likewise helps make an exciting acknowledgment of the web site: hunt operator, suggesting to dismiss the results since the "ordinary" individuals won't view those end results.He composed:." Yes, you are actually proper: if our experts can't crawl the web page, our company can't see the noindex. That said, if our experts can't crawl the web pages, at that point there's certainly not a whole lot for our company to index. Thus while you could observe a few of those web pages along with a targeted site:- question, the ordinary consumer will not view them, so I would not fuss over it. Noindex is also alright (without robots.txt disallow), it only means the Links will definitely wind up being crawled (as well as wind up in the Browse Console report for crawled/not listed-- neither of these statuses cause issues to the rest of the website). The integral part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the restrictions in operation the Site: hunt progressed hunt driver for diagnostic causes. Some of those explanations is since it is actually certainly not hooked up to the frequent hunt index, it's a separate point entirely.Google's John Mueller talked about the web site hunt operator in 2021:." The short solution is actually that an internet site: question is certainly not meant to become comprehensive, neither made use of for diagnostics purposes.An internet site question is actually a particular type of hunt that limits the outcomes to a particular web site. It's generally simply words web site, a colon, and afterwards the internet site's domain.This question limits the end results to a details internet site. It is actually certainly not indicated to become a detailed assortment of all the webpages coming from that website.".2. Noindex tag without using a robots.txt is great for these type of conditions where a bot is linking to non-existent web pages that are receiving found through Googlebot.3. Links with the noindex tag are going to create a "crawled/not listed" item in Browse Console and also those will not have an adverse result on the rest of the web site.Go through the question and also address on LinkedIn:.Why would certainly Google index pages when they can't even observe the web content?Featured Graphic by Shutterstock/Krakenimages. com.