How to Create robots.txt file & sitemap.xml file for seo

By | August 9, 2019


Hello and welcome,
Today I’m going to show you how to create robots.txt file and also sitemap.xml file for seo purposes. Best way for me to demonstrate how to create these two particular files is to give you an example.
I’ve got www.rankya.com.au open now if I was to simply type robots.txt file, you will see that it is simply a text file with some user agent directives. Now as far as we are concerned, we need to put this particular information into this file, because Google will honor this particular directive, now if we didn’t have this directive, then Googlebot will not honor it.
What this is saying is to Googlebot, saying “ok when you come visit my website, I will disallow you to index these particular folders” now you can have files, you can disallow some file, as in some html files, php files and so on, or you can also, disallow some folders. Now why would you want to do that? Is because, let’s say that you have a privacy policy file, or you got some terms and conditions file on your website, which most websites do have, now I don’t really want to rank for these two particular files, you know they are important for my customers, but for ranking purposes, for seo purposes, they are not that important for Google, they are important for my potential customers, but once again we are in the business of Google ranking, therefore, we need to be intelligent, as to what we want to rank for, and what we want to disallow Google to rank us for and so on. Therefore we need to have this directive targeting Googlebot specifically, if not, Googlebot will simply index all your pages and so on. Another important element of robots.txt file is the URL for your sitemap, that’s why we will talk a little about sitemap file, how it is important and also we’ll look at how to create proper sitemap files. There are many online generators for creating sitemaps, but I will give you as usually, I will give you important knowledge for creating sitemaps. Therefore you need to have a sitemap; it is intelligent for your Google rankings to have a sitemap, if you haven’t got one, simply go online, do bit of research and generate your xml file, and then simply upload it on to your root folder of your website, ok?
But this is saying, let’s go step by step, when Googlebot comes and visits your url, what it will do is, it will first look for this file, and when it does, it’s going to say “has it got any information for me? hmm, it has” therefore, it will honor this, right? But also, since it is still reading it, it’s going to say “hmm there is a sitemap here” see? and then it is aware of that url therefore it will then visit that particular url. So let me now quickly open that particular file, to show you, what this is all about right? Now, this particular file contains some information in the form of xml, and what I’m saying here is “I have a url which is my main url, it has been modified at this date, it gets changed weekly, and it has a priority setting set as 1, now that’s the highest that I can have, surely this is my home page, therefore, it is intelligent to have my home page as the most important page, therefore, I say it in that priority setting. Ok? now that’s kind of important, and also, a lot of people still make an honest mistake, as to, they think that if they were to have that information as daily because which you can, they think it is intelligent for them, now, the only time you should have that changefreq set to daily, is if you change you content daily, and if you don’t change it daily, and if you change it weekly, then you should have it weekly, and if you change it monthly, then you should have it set to monthly. It’s that simple. Also, as you can see there is another page, surely rankya.com.au at this stage is a small website; you may have hundreds of pages, nonetheless, sitemaps are important for you to create now, once again, when we look at this url set here, we need to have a look, we need to be smart here, when we look at this, we need to think “hmm ok, my home page is the most important, but the way that I structured rankya.com.au, as far as I’m concerned, this particular page is the second most important” therefore, I set that priority, as to point 9, that’s my most important page most important resource, and this is my second most important resource ok? And this is how we do it we say it here ok?
And also, this particular url, I’m saying well then that one comes as far as priority settings are concerned. And if scroll down you will see that the way I structured it, is I’m going from 1 point 0 to lower ok? so that kind of important, as you can see I do not have my contact page here, as url set, now there is nothing wrong having that, but we are in the game of ranking, therefore we need to optimize these particular files in such way, that it serves their purpose, ok? And this is a, perhaps you know, quick example as to what these pages are all about, we need to be intelligent as far as setting these, a lot of sitemap generators you will come across, they will simply have information as to, it will have 0 point 5, and that’s like neutral it’s like baseline it’s like middle line, ok? Therefore it’s up to you to then go ahead and start thinking a little saying “ok I have this website” surely all your pages are important to you, but, you know once again, you are in the ranking game, right? Therefore, you then got to sit down and think “hmm, ok, I know that all my pages are important, but, which ones are more important than the others?” because when you start thinking like that, you’ll end up coming with some that you will set higher priority than the others, ok? So that’s very important. And once again, you can simply Google “robots.txt Google webmasters” to find out little bit more information about this particular file, and also, you can visit, sitemaps.org website and read more about it, because these are important elements for Google ranking, surely they are not the only part of seo, nonetheless, having these simple procedures set properly, will only benefit your Google rankings, ok? Also you may come across information as to, telling you these particular files are not important. I promise you and I assure you, do not listen to misinformation like that, because they are important.
I hope you have enjoyed watching this particular video, and if you would like to find out more about search engine optimisation methods, simply visit us at rankya.com.au thank you very much

24 thoughts on “How to Create robots.txt file & sitemap.xml file for seo

  1. tradesmanstips Post author

    Hi, I am still learning but Google removes underscores so the urls are ineffective as part of Google's 200 seo criteria. I had to start redirecting to new urls with dashes? Pointless missing out an easy bonus point.

    Reply
  2. RankYa Post author

    G'day mate, I am not familiar with web.com and how their CMS work. I have briefly looked at their site to see any documentation but couldn't find it. I encourage you to get in touch with their support team to see if they allow you to add one. If you have access to hosting back-end, then I can assist you to create one and also to upload it. Let me know how you go, thanks for stopping by.

    Reply
  3. RankYa Post author

    Yes, you save the file as plain text, so what is important is the file extension you save it as. Meaning, simply create your sitemap using plain text editor of your choice, and once you place all the details of your URL's within the file, simply save it as:

    sitemap.xml

    making sure that the extension is .xml
    then simply upload this file inside public_html folder of your hosting account using any FTP program.
    Don't forget to add your new sitemap to Google Webmaster Tools account, I have many videos which explain how to add and also verify sitemaps. Here's one video I think will be of interest for you.
    http://youtu.be/DZHYQ0otfBk
    Hope this helps, happy rankings

    Reply
  4. Miranda Alegre Post author

    the information is good however you never told us how to do it…

    Reply
  5. Dave Munday Post author

    FEEDBACK..
    Please check your sound next time you make a video.  Your voice is understandable, but the volume of your recording is very poor. Perhaps check your mic volume? Speak closer to the mic?  

    Reply
  6. Valentin Hîncu Post author

    Very well explained. thank you very much

    Reply
  7. Arturo Garza Post author

    How is this even done? You need access to the back office of the website? You save the robot file and then transfer it to the website? Do you have any videos where you do this?

    Reply
  8. TOPMusic Post author

    awesome tutorials! I learned tons of great info. Btw, I notice that you're using SEOyoast to generate your sitemap, is there a way to set priorities using yoast plugin? Thanks

    Reply
  9. Binod Jha Post author

    when i try to create they show message like this
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    why xml.sitemap generator create erorrs when I do submit my website url.

    Reply
  10. Market Man Post author

    Very nice video Rankya!

    Is there any disadvantage of having different pages in the xml-sitemap with the same priority? and what kind of advantage brings a higher priority to my page exactly?

    Reply
  11. Nandan M Post author

    its showing robots.txt fetch failed. i have successfully uploaded sitemap

    Reply
  12. Sneha Sastry Post author

    Aweomse content and explanation. Voice quality is quite bad

    Reply
  13. Pikaflix Productions Post author

    hey hey. Just a quick question. I used your robots.txt guide. When i test my sitemap on google webmaster. i get 7 warning messages with the following : Sitemap contains urls which are blocked by robots.txt. I have looked everywhere on the net for someone with the same problem but cant seem to find a solution. would you have any ideas?

    Reply
  14. Glow0110 Post author

    Is there a difference between having http://yoursite.com and http://www.yoursite.com in the xml sitemap? Also, thanks for all the awesome information that you put out!

    Reply
  15. Pankaj kumar Post author

    THANKS A LOT FOR .. PLEASE DO FAVOUR CAN you elaborate what is local and assets
    User-Agent: Googlebot
    Disallow: /Local/
    Disallow: /assets/
    User-Agent: slurp
    Disallow: /Local/
    Disallow: /assets/
    User-Agent: Bingbot
    Disallow: /Local/
    Disallow: /assets/

    Reply
  16. Jack Xscript Post author

    Guy is obviously not an audio engineer that's for sure….

    Reply
  17. rahul gogoi Post author

    hei . how to redirect https in custom domain.plz reply

    Reply
  18. rahul gogoi Post author

    hei . how to redirect https in custom domain.plz reply

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *