Lesson 14: Crawlability and the Yoast SEO plugin – SEO for beginners training

By | January 26, 2020


In the previous video,
we discussed crawlability. We looked at all the ways
you can make sure that certain parts of your site
are not accessible to search engines. All of this can be a bit tedious
if you don’t have a technical background. That’s why we take care of a lot of aspects
of crawlability for you in our Yoast SEO plugin. In this video we’ll explore
what the plugin does. You’ll learn what options you have
to make sure that Google can index everything you want to show
in the search results. And, of course, that search engines stay away from everything you don’t want
to show in the results. Let’s start with the basic settings. In the Search Appearance settings,
you can determine your default settings for every type of content
available in WordPress. For every type of content, we ask you whether you want search engines to put it
into the search results. If you pick “no”,
we add a “no-index” robots meta tag, making sure that nothing will show up
on the results page. You can do this for posts, pages, tags
and lots of other kinds of content. Just choose “yes” or “no” to either show
or not show them in the search results. Don’t worry if you’re not sure
about what you should choose. If you don’t do anything, the settings reflect
what is best in most cases. If your theme or developer has created
other content types, you can see them here as well. Now, say you have told the search engines
to actually put all posts in the search results with the Search Appearance settings. But say that now you have a post
which you don’t want to appear in Google, for example because it’s an old article
that you’re not very proud of. Can you also no-index specific posts? Well yes, of course you can. To do this, go to the advanced tab
of the Yoast SEO meta box beneath your post. Here, you have the same option
we discussed earlier, only now it applies
only to that specific post. The only difference is that here
you have a third option, which is to always follow the default setting
you set earlier. Besides determining whether the post
should be shown in the results, you can also choose whether to allow
search engines to follow links on the page. If your SEO skills
are a little more advanced, you can also tell Google not to put images
from the post into the index, to not archive the post (which means Google cannot show
a previously saved copy of the page), or to not show a snippet
in the search results. We advise you to stay away from this
unless you really know what you’re doing. But it’s good to know it’s there for you
to use if you need it. You can not only change your settings
for posts, but also for pages and tags,
and so on and so forth. If you want to move
beyond the simple crawlability settings, the Yoast SEO plugin also allows you
to easily edit your own robots.txt file. We already discussed robots.txt
in the previous video. It’s a file in which you tell the search engine
which URLs on that site it’s allowed to visit. How this works is advanced stuff
which is beyond the scope of this course. You can learn more about how to edit
your robots.txt in our Technical SEO course. Okay, so now you know how you can
determine your crawlability settings for a content category
and for individual posts (or pages, etc.), and where to edit your robots.txt. But is it possible to quickly check
your indexability? Yes, it is!
There are actually several ways. Let’s start with the easiest one. If you go to your WordPress dashboard,
you’ll see the Yoast SEO posts overview. There, below the published SEO scores, you’ll see a tool called
the indexability check by Ryte. This tool gives you feedback
on whether your homepage is indexable. If it’s not,
you need to fix this immediately. If it is, you know for sure that the search engines are able to index
at least part of your site. The last plugin feature that is important
when it comes to crawlability is the Search Console setting. Here, you can connect
with Google Search Console and check all the crawl errors that Google
encountered when crawling your site. This is a great way to check what individual
pages have crawlability issues. There are two basic errors
Google can encounter. First of all, there may be
site-wide problems, like connectivity issues
or problems fetching your robots.txt file. More often, though, there are problems
with individual URLs. The page may have been deleted,
for example. Solving these errors
(generally by making redirects) makes it much easier for Google
to crawl your site, which can have a positive effect
on your rankings. Good luck!

One thought on “Lesson 14: Crawlability and the Yoast SEO plugin – SEO for beginners training

Leave a Reply

Your email address will not be published. Required fields are marked *