Yahoo!-specific REP directives that are not supported by Microsoft and Google include:
* Crawl-Delay:
Allows a site to delay the frequency with which a crawler checks for new content
* NOYDIR META Tag:
This is similar to the NOODP META Tag above but applies to the Yahoo! Directory, instead of the Open Directory Project

* Robots-nocontent Tag:
Allows you to identify [...]

As stated in the last article, writing a robots.txt file is important for SEO. It helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.
Not all search spiders will follow your instructions left in the robots.txt file.
A poorly done robots.txt file can stop the search spiders [...]