tag:blogger.com,1999:blog-6569681.post7243610144219173585..comments2024-01-15T13:17:33.771-08:00Comments on Geeking with Greg: Crawling is harder than it looksGreg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-6569681.post-53594260166881455372008-06-04T06:17:00.000-07:002008-06-04T06:17:00.000-07:00AdSense...AdSense...Unknownhttps://www.blogger.com/profile/17747030397404428124noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-29453052423018891612008-05-24T01:12:00.000-07:002008-05-24T01:12:00.000-07:00"...G even using GA and AS data"What does 'AS' ref..."...G even using GA and AS data"<BR/><BR/>What does 'AS' refer to ?Singapore Oppositionhttps://www.blogger.com/profile/09585522235830596671noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-65439966859484606912008-05-20T00:27:00.000-07:002008-05-20T00:27:00.000-07:00Vicaya should probably keep in mind that Google/Ya...Vicaya should probably keep in mind that Google/Yahoo use clusters of thousands of servers to crawl the web. Also the paper says the crawler was limited by the university bandwidth. <BR/><BR/>I would imagine that with a mere $20M invested into the crawler, this project could do 6B pages in 1 day.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-72938146567036970122008-05-09T19:22:00.000-07:002008-05-09T19:22:00.000-07:00@anon: Agreed. It's an important contribution. I j...@anon: Agreed. It's an important contribution. I just lamented the fact that neither G nor Y published anything on crawling since the altavista papers. <BR/><BR/>AFAIK, both G & Y's crawls are now driven by click data, with G even using GA and AS data. So, yes, nobody is catching up to G anytime soon, unless they commit seppuku themselves.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-11414929188177022202008-05-08T02:24:00.000-07:002008-05-08T02:24:00.000-07:00@vicaya:Even if the paper brings nothing new when ...@vicaya:<BR/><BR/>Even if the paper brings nothing new when compared to the current state of the art in the industry, this contribution is very important.<BR/><BR/>What is not published is not available for future generations, or even developing nations. Science needs to look only to publicly available knowledge. Knowledge inside Google or Yahoo! is restricted to a minority.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-79201601747004808742008-05-07T21:38:00.000-07:002008-05-07T21:38:00.000-07:00Crawling is hard for sure.Kudos to the authors for...Crawling is hard for sure.<BR/><BR/>Kudos to the authors for publishing the paper, but the techniques described in the paper just rediscovered (with minor interesting variations) a small fraction of tricks used by G and Y for years (it's sad that with all those PhDs they have, they don't publish anything new in this area, in the name of "competitive advantages"). <BR/><BR/>6 billion+ valid responses recorded in 40+ days is not that impressive, esp. when they were not storing all the content. The numbers are pretty good (average 319mbps on a 1gb uplink. A state of the art web crawler these days should be able to saturate any pipe in existence) but really didn't blow me away given the hardware spec. The comment in the paper about scaling to multiple nodes is naive at best.Anonymousnoreply@blogger.com