SMX Advanced: Yahoo Issues Bot-Herding Standard

Presented by Priyank Garg from Yahoo

Standard Across All Engines

Robots.txt

• Disallow
• Allow
• Wildcard tags
• Sitemap location

Meta Tags:

• NoIndex
• NoFollow
• NoSnippet
• NoArchive
• NOODP

Optional Robots.Txt/MetaTags by Engine
 Yahoo

• Crawl delay
• NOYDIR
• Robots-nocontent

Google

• UNAVAILABLE AFTER
• NOTRANSLATE
  NOIMAGEINDEX

MSN

•     Crawl Delay