tag:blogger.com,1999:blog-6569681.post116941841525302056..comments2024-03-24T10:38:16.997-07:00Comments on Geeking with Greg: Yahoo Research on distributed web searchGreg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-6569681.post-60216087022322695352007-08-02T07:54:00.000-07:002007-08-02T07:54:00.000-07:00Thanks, Fabrizio!Thanks, Fabrizio!Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-74999023885917936982007-08-02T07:06:00.000-07:002007-08-02T07:06:00.000-07:00Hi Greg,nice post! (I know I'm a little bit biased...Hi Greg,<BR/><BR/>nice post! (I know I'm a little bit biased :P)...<BR/><BR/>About the idea mentioned a page 5... Actually we have published a couple of papers on that topic. You can find them on my home page... They contains some preliminary ideas and results but... if you want to see some outcomings of what happens playing around with shards... well... there you'll find some...<BR/><BR/><A HREF="http://soave.isti.cnr.it/%7Esilvestr/wp-content/uploads/2007/07/infoscale2007cache.pdf" REL="nofollow">Incremental Caching for Collection Selection Architectures</A><BR/><A HREF="http://soave.isti.cnr.it/%7Esilvestr/wp-content/uploads/2007/07/infoscale2007termpartitioning.pdf" REL="nofollow">Mining Query Logs to Optimize Index Partitioning in Parallel Web Search Engines</A><BR/><BR/>Enjoy!Fabrizio Silvestrihttps://www.blogger.com/profile/06148151554175640595noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1169488487995459322007-01-22T09:54:00.000-08:002007-01-22T09:54:00.000-08:00All great points and totally correct. However, the...All great points and totally correct. However, these are all problems found in any peer to peer search as well.<BR/><BR/>Ultimately for either to get much traction, the system would need to embrace the strengths of a narrow but deep and distributed search architecture. The system may never be able to answer a document keyword search in a few milliseconds, but may be able to answer a question requiring a more exhaustive analysis of text with an answer quality worth waiting for.Deephttps://www.blogger.com/profile/06005395387091047024noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1169481907862515312007-01-22T08:05:00.000-08:002007-01-22T08:05:00.000-08:00Good point, Deep. Meta or federated search is a p...Good point, Deep. Meta or federated search is a possibility. As you said, Metacrawler, Dogpile, A9, and many others have built systems that query many search engines and databases, then merge the results.<BR/><BR/>Several problems come up when you try to do this though. First, relevance rank can be poor because of the limited information returned from each database or search engine (usually top N by widely differing relevance ranking functions).<BR/><BR/>Second, speed can be an issue. You can only as fast as the slowest database you query, and probably much slower due to the additional computation to merge and rerank the results.<BR/><BR/>Third, at very large scale, you can no longer query all the databases on every request, so you have the hard problem of determining which databases are most likely to return a useful answer to a query, a problem that may be almost as hard as building your own search index.<BR/><BR/>Even so, I agree that it is an interesting direction to pursue, especially for smaller players who cannot hope to duplicate the massive crawl and indexing clusters of the search giants.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1169441588192952162007-01-21T20:53:00.000-08:002007-01-21T20:53:00.000-08:00The Bender paper is especially interesting. Thinki...The Bender paper is especially interesting. Thinking about the limitations of true peer to peer, made me think of a related idea. What if some of the myriad "vertical search" providers proliferating on the web could follow a standard for communicating the contents of their verticals along the lines of thinking in the Bender paper. Its not peer to peer, but to some extent it addresses the monopolistic concerns of the big 4, and it does address the dependability of a search node problem. In other words, create a meta search infrastructure, with built in economic accommodations of some sort, that vertical search providers could plug into; unlike the meta crawlers of yore that focus on meta crawling general search engines (e.g. metacrawler, dogpile, ...), they metacrawl niche vertical engines constrained by topic. If done right, this might lower the barrier of entry for small players to make an impact in broader search.Deephttps://www.blogger.com/profile/06005395387091047024noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1169431566532719822007-01-21T18:06:00.000-08:002007-01-21T18:06:00.000-08:00Greg,Thanks for passing along these great referenc...Greg,<BR/>Thanks for passing along these great references and thoughts.<BR/><BR/>Regarding client-side personalization of search, one has to wonder if Yahoo and Microsoft are pursuing this line more due to competitive weaknesses versus Google rather than the security red herring. <BR/><BR/>Similar to our recent discussion on recommendations, personalization is a much more fundamental concept than search. One can envision not only personalization of searches, but also personalization of recommendations, personalization of the user interface itself, etc. To try to force-fit the locus of personalization on the client-side when it will have to serve so many purposes strikes me old-think.Steve Flinnhttps://www.blogger.com/profile/03055018730006376234noreply@blogger.com