Hello again,
Today I am talking about the robots.txt file and why it is necessary to include in your site files. The robots.txt file is a directionary file in which it directs or tells search engines what to include in their site crawls and what to exclude. For instance, you really do not need the search engines to crawl or include the names of your image files in their results, in this case you would use the robots.txt file to specifically exclude the directory from being crawled by the search engines.
Hopefully this helps add a bit more information regarding how the search engines work.
Have a great day!
