Stop Google from Crawling a WordPress Website

World Wide Web(WWW) is a public place where most of the content is visible to all its users for accessing the data associated in it.Search engines like Google,Bing,Yahoo etc do us a favor by crawling our websites and allows us to keep in touch with our subscribers,audience,customers,clients etc.But we may face a criteria to maintain personal data which is only visible to its publisher or owner ,Or sometimes a project has to be done which is only accessible to it members only .so this data should not be crawled by google or any other search engines to display in their search results.So in this article,we will show you how to stop google from crawling a WordPress website

Block/Stop Google from Crawling a WordPress Website

Method 1

By Default,WordPress comes with a built-in option that allows administrator to request search engines like google,yahoo,bing etc, not to index or crawl your WordPress website. In order to achieve this,the user has to visit Settings » Reading and check the input box next to Search Engine Visibility label.

Stop Google from Crawling a WordPress Website

Method 2

Adding a meta tag manually  to header.php located inside all WordPress themes can stop Google from crawling a WordPress Website,but it is little bit harder than Method 1

<meta name=’robots’ content=’noindex,follow’ />

Method 3

Setting an User-agent to all search engines by entering * and Disallowing them to index all directory inside robots.txt file which is located inside root directory of Wordpress Installation. If it is not located,then create a file named robots.txt and enter the following code in it and then save it

User-agent: *
Disallow: /

Method 4

If you are using cPanel to manage your Wordpress hosting,then by protecting the complete WordPress website using password protected web directories

setting-setting-password-protected-web-directory

 

 

 

 

then select the document root for which the protection is needed

choose-directory-to-password-protected-web-directory

After selecting the root, check the box next to ‘Password protect this directory’ option. Next, provide a name to the protected directory and hit the save button,Create an authorized user to access this directory with a password

setting-username-and-password-to-password-protected-web-directory

Method 5

By writing a password protected WordPress post,its content is not visible to public or any search engine crawler to index a WordPress site without entering a correct password associated with it. if you are not sure how to create a password protected post in WordPress,follow this topic on How to create a password protected Wordpress post .

Note: By following the Method 1,the Wordpress automatically sets the robots.txt to disallow the entire directory to all search engines and sets the meta tag to noindex

crawling,google,indexing,wordpress