Skip to main content

What is Robots txt in SEO? Example with code.


A robots.txt file is a text file that is used to instruct search engines, such as Google and Bing, which pages or sections of a website should not be crawled or indexed. The file is typically located in the root directory of a website, and is accessed by search engine bots when they visit the site.

The robots.txt file uses a specific syntax, known as the Robots Exclusion Protocol (REP), to specify which parts of a website should be excluded from crawling. For example, a website owner may use the robots.txt file to exclude pages that are under development, or to exclude pages that contain sensitive information.

It's worth noting that the use of a robots.txt file does not guarantee that a page will not be indexed or that it will not appear in search results. Some crawlers might ignore the robots.txt file and some might not, and also some hackers may use the file to find vulnerabilities.

However, having a robots.txt file and using it correctly is a best practice in SEO. It helps to prevent search engines from indexing duplicate or irrelevant content, and it can also help to improve website performance by reducing the number of pages that need to be crawled.

Sure, here is an example of a basic robots.txt file that disallows all search engines from crawling the entire website:

This code tells all search engine crawlers ("User-agent: *") not to crawl any pages ("Disallow: /") on the website.

Here is another example that allows all search engines to crawl the entire website:

This code tells all search engine crawlers ("User-agent: *") that there is no page that should not be crawled ("Disallow: ") on the website.


Popular posts from this blog

100 Backlinks. Search Analyst Sasikumar Talks. Tamil

  #backlinks #seo #digitalmarketing #searchanalystsasikumartalks 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.

Introduction about me. Search Analyst Sasikumar Talks. (English)

Introduction about me:   I am Sasikumar. Up to 12th standard, I studied at Walter Scudder higher secondary school in Tindivanam, Vizhuupuram district. I completed my bachelor's degree In computer science engineering at St Joseph's College of Engineering, Chennai and I completed my master's degree In computer science engineering at  Arunai Engineering College, Thiruvannamalai.    I have more than 10 years of experience In digital marketing. I start working on digital marketing platforms since 2008.  I have worked in so many companies and to list out a few, I worked in Securenext software private limited, Kapp software private limited, Radiant Dental Care, B&A Digitactical, 8k Miles software, and so on.   My skills include digital marketing, social media optimization, Content writing, blog, AdSense, and so on.  The YouTube channel that covers digital marketing and SEO In  Tamil. Thanks for your support.

Your video has an unsupported aspect ratio. Search Analyst Sasikumar Talks. (Tamil)

  #youtubeseo #youtubeoptimization #seo #searchanalystsasikumartalks I have uploaded more than 300 videos to my YouTube channel but I have done a small mistake and the mistake is I did not put the right aspect ratio when I use the camera. yes, the aspect ratio is very important when you upload videos on YouTube channels. if you upload a video with the wrong aspect ratio you cannot use the end screen element in your YouTube setting. The recommended aspect ratio is 16 9 and sum of the people discussed is 4:3. it is very simple to fix this issue I just open the camera open to the video and then I changed the aspect ratio from full to 6:9. now I uploaded the video and it works perfectly well and I could add the end screen element. After uploading 300 videos I noticed that I am not recommended by YouTube by showing up my video in the suggested videos and I wonder and question whether the end screen element error may be the problem for the videos that I upload. Thank god at least by n