| 蜘蛛 | 404 路径 | Nginx 建议配置 (自动编号) | User-Agent |
|---|---|---|---|
| Douyin | /robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n002.html last; } |
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/) |
| Douyin | /robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n006.html last; } |
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/) |
/robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n001.html last; } |
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
/robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n005.html last; } |
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Other | /robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n003.html last; } |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| Other | /robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n004.html last; } |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| Other | /robots.txt |
location = /robots.txt { rewrite ^/robots\.txt$ /new_path/n007.html last; } |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |