有时候明明网站访问人数不是很多，但就是CPU和内存占用多，这时候可能就是一些垃圾蜘蛛爬虫导致的，像百度谷歌这些的搜索引擎的蜘蛛我们肯定是欢迎的，但那些爬网站搞营销甚至采集资料的蜘蛛，还是屏蔽了比较好。
服务器压力大？禁止这些蜘蛛访问可以省资源
比如那些SemrushBot，MJ12bot，AhrefsBot的，除了增加服务器的负担，没别的好处，我们就直接屏蔽就好。
所以大家直接把下面代码加入网站的robots.txt文件里面就行了：
User-agent: DotBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: CheckMarkNetwork
Disallow: /
User-agent: DigiCert DCV Bot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Barkrowler
Disallow: /
User-agent: Uptimebot
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Researchscan
Disallow: /
User-agent: DnyzBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: YandexBot
User-agent: MSNBot
Disallow: /User-agent: YoudaoBot
Disallow: /User-agent: JikeSpider
Disallow: /User-agent: EasouSpider
Disallow: /User-agent: AhrefsBot
Disallow: /User-agent: MJ12bot
Disallow: /User-agent: YandexBot
Disallow: /User-agent: YisouSpider
Disallow: /User-agent: YYSpider
Disallow: /User-agent: Teoma
Disallow: /User-agent: rogerbot
Disallow: /User-agent: exabot
Disallow: /User-agent: dotbot
Disallow: /User-agent: gigabot
Disallow: /User-agent: DOCOMO Sprider
Disallow: /User-agent: ToutiaoSpider
Disallow: /Disallow: /
