As stated in the last article, writing a robots.txt file is important for SEO. It helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.
Not all search spiders will follow your instructions left in the robots.txt file.
A poorly done robots.txt file can stop the search spiders [...]