User-agent: XoviBot Disallow: / User-agent: Riddler Disallow: / User-agent: Baiduspider Disallow: / User-agent: Qwantify Disallow: / User-agent: spbot Disallow: / User-agent: linkdexbot Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: HTTrack Disallow: / User-agent: SMTBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: Zealbot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: GigablastOpenSource Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: Nutch Disallow: / User-agent: Nutch-1.10 Disallow: / User-agent: tbot-nutch Disallow: / User-agent: spock Disallow: / User-agent: DomainAppender Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: BecomeBot Disallow: / User-agent: genieBot Disallow: / User-agent: dotbot Disallow: / User-agent: MLBot Disallow: / User-agent: 80bot Disallow: / User-agent: Linguee Bot Disallow: / User-agent: aiHitBot Disallow: / User-agent: Exabot Disallow: / User-agent: SBIder/Nutch Disallow: / User-agent: Jyxobot Disallow: / User-agent: mAgent Disallow: / User-agent: MJ12bot Disallow: / User-agent: Speedy Spider Disallow: / User-agent: Huasai Disallow: / User-agent: DataCha0s Disallow: / User-agent: Atomic_Email_Hunter Disallow: / User-agent: Mp3Bot Disallow: / User-agent: WinHttp Disallow: / User-agent: betaBot Disallow: / User-agent: core-project Disallow: / User-agent: panscient.com Disallow: / User-agent: Java Disallow: / User-agent: libwww-perl Disallow: / User-agent: wget Disallow: / User-agent: curl Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: meanpathbot Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mail.Ru Disallow: / User-agent: BLEXBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Yandex Crawl-delay: 10 User-agent: * Disallow: /report-error Disallow: /db_backup.1744711664/# Old DB crash data