google蜘蛛一天访问近十万次正常吗
Regoogle蜘蛛一天访问近十万次正常吗
这是可能的,我的系统刚上线前几天谷歌一天就是来几十万次. 66开头的这个ip特别多,后来我用nginx设置了下链接数限制.
[root@www nginx]# grep '^66' access.log-20160326 | head -10
66.249.73.219 - - [25/Mar/2016:03:16:03 +0800] 'GET ########################## HTTP/1.1' 200 3230 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:04 +0800] 'GET ########################## HTTP/1.1' 200 3609 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:04 +0800] 'GET ########################## HTTP/1.1' 200 2906 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:04 +0800] 'GET ########################## HTTP/1.1' 200 3068 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:05 +0800] 'GET ########################## HTTP/1.1' 200 8260 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:05 +0800] 'GET ########################## HTTP/1.1' 200 4452 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:05 +0800] 'GET ########################## HTTP/1.1' 200 6774 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:06 +0800] 'GET ########################## HTTP/1.1' 200 5279 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:06 +0800] 'GET ########################## HTTP/1.1' 200 3628 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
66.249.73.219 - - [25/Mar/2016:03:16:07 +0800] 'GET ########################## HTTP/1.1' 200 4504 '-' 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' '-'
[root@www nginx]# grep '^66' access.log-20160326 | wc -l
209549
赞0
踩0