-
How to Create a robots.txt File?
As with anything in life, you need rules to control the main issues. For your website, you must create a robots.txt file with certain rules for search engines to follow. These robots tell the
Create a robots.txt FileTo create a robots.txt file for your website, follow these steps:1- Checking the existing robots.txt fileUsually, each installed WordPress comes with a default WordPress robots.txt. This default robots.txt file is found to prevent search engines from crawling into the admin folder.
- You can see the robots.txt by typing “yourdomainname.com/robots.txt” in Google Chrome.
- The page will display like this:
Yeah, it looks stupid, but this robots.txt file is as simple as this.
2- Create a robots.txt file with your own rules
To set your own rules, we will learn how to create a robots.text file.
You can simply do it by following these steps:
- Create a new .txt document using notepad or notepad++.
- Start your robots.txt file by typing “User-agent*” (actually, all robots.txt start with this line):
3- Put your rules
As we all know, the role of creating a robot.txt file is to set your own rules. The possible rules are:
- Block a specific path
Type the words “Disallow: /your-path”.
Write the path which you want to block instead of “your-path”.
With this line, you will be blocking this path and every other path comes through it.
- Block a file type:
Type the words “Disallow: /*.filetype$.”
Write the name of the filetype which you want to block instead of “filetype”.
If you write “Disallow: /*jpg$, you will prevent the crawlers from accessing all jpg files on your website.
Additionally, if you want to prevent the crawlers from accessing a certain file, write its name instead of “*”, for example: “Imagename.jpg”.
- Allow specific sub-paths to be accessed by crawlers
Type the words “Allow: /yourpath/sub-path”..
Write the path which you want to allow the crawlers to access instead of “yourpath/sub-path”.
Let’s say you wanna block the access to pictures but allow it for pictures/public, then you should put 2 lines like this:
“User-agent: *
Disallow: /yourdomainname.com/pictures
Allow: /yourdomainname.com/pictures/public”
- Allow specific file types to be accessed by crawlers
Type the words “allow: /*.filetype$.”
Write the name of the filetype which you want to allow the crawlers to access instead of “filetype”.
- Block specific crawlers from crawling your website
To block certain crawlers, you just need to add 2 new lines under the existing lines in your robots.txt file.
- “User-agent: Unwanted Crawler”
- “Disallow: /”
Look at the image below:
Note:
After creating or editing any robots.txt file, you must save it.
4- Adding the robots.txt file
You can add the robots.txt file using several ways, including:
- Through FTP or SFTP
If you are using any platform, you just need to upload robots.txt file to your website through FTP or SFTP.
Usually, this requires some developing knowledge and access to your server.
Therefore, you can ask your developer to help you with it to avoid any stability problems.- Creating and adding robots.txt file using Yoast Plugin
If you are using WordPress, you can do it easily with the help of the Yoast plugin. Follow these steps:
- Log in to your WordPress dashboard.
- On the left side menu, select “SEO” then tools.
- At the tools section, select “File editor”, then “Create robots.txt file”.
Notice that Yoast SEO doesn’t consider the default robots.txt file. Therefore, if you don’t have a robots.txt file, you will see a message saying that you don’t have one as in the image below:
- A TXT file editor will open where you have to edit the rules according to yours.
- Write your own rules as we mentioned above and make sure to add a line with your website sitemap to facilitate the crawling to all your subpages.
- Congratulations! You’ve finished. Just click on “Save changes to robots.txt”.
-
0 Replies
Sorry, there were no replies found.