tag:blogger.com,1999:blog-6569681.post113111957629107712..comments2024-01-15T13:17:33.771-08:00Comments on Geeking with Greg: Amazon Mechanical Turk?Greg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-6569681.post-1131482106472909682005-11-08T12:35:00.000-08:002005-11-08T12:35:00.000-08:00Seems like another case of amazon thinking that be...Seems like another case of amazon thinking that because a feature is really useful to <B>amazon</B> that it's equally of interest/use for others. A thread on slashdot pointed out that unless the prices go up, it's a big waste of time for anyone but the chronically unemployed in America to do these tasks ($3/hour, max, woohoo!).<BR/><BR/>Personally, I think Internet usage is nearing a saturation point, much like other mediums (people only watch a certain amount of TV, go to movies #times/week, etc.). People simply aren't going to spend 24 hours on the Internet. They use it for work, they use it for email, shopping, etc. But they're not looking for ways to spend more hours to make $3/hour doing things that benefit others. And now that we're at that point, companies will have to compete very hard for eyeballs as competition among sites and activities increase. <BR/><BR/>I think Amazon (like Microsoft) thinks they can win that war of eyeballs simply by doing "new stuff" or "more stuff". That by sheer dint of their size, people will stay loyal to them and try new things that they produce.<BR/><BR/>I wonder if this an example of Amazon "jumping the shark"?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131386376637094482005-11-07T09:59:00.000-08:002005-11-07T09:59:00.000-08:00Well, I think the model here is (or should be!) le...Well, I think the model here is (or should be!) less like Google Answers or eBay and more like an actual market: i.e. responders should get paid according to their trust metric, not just the number of HITs being done. <BR/><BR/>This can be done after a big enough pool of humans is built up by offering some secondary reward scheme that differentiates the price enough without alienating new drones. But initially, yes, Greg's right, the price would be higher than optimal as the pool would need to be initialized: at this point it's just doing outsourcing work for Amazon itself, so really it's near-free for Amazon. But with enough high-trust humans in the pool, repetition (and thus end-price) will come down.<BR/><BR/>Now if only could Amazon could build a better interface to the mTurks... hmm, maybe a plug to the back of the scull would work better ;-)Costashttps://www.blogger.com/profile/05157234618935426983noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131316213851781122005-11-06T14:30:00.000-08:002005-11-06T14:30:00.000-08:00While it is true that reputation-based selection o...While it is true that reputation-based selection of workers is expensive, a quick read of the FAQ indicates that a requester can make a reputation score a requirement for working on his/her tasks.<BR/><BR/>It also looks like requesters can create additional qualifications that have to be met in order to do some work, although i couldn't find any work that had any requirements.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131290033539478452005-11-06T07:13:00.000-08:002005-11-06T07:13:00.000-08:00Hi, Costas and Elroy. Good points on ways to deal...Hi, Costas and Elroy. Good points on ways to deal with incorrect answers.<BR/><BR/>It's true that there are ways to increase quality, but they are expensive and imperfect.<BR/><BR/>If I duplicate effort or pay people to check over other people's work, I now take a single piece of work and send it out to N people instead of just one, multiplying the costs.<BR/><BR/>If I have an eBay-like reputation system, I increase the costs for the buyer, who now has to spend time checking reputation, trying to verify the quality of the answer (which may be difficult for the buyer), and leaving feedback. This works fine on Google Answers and eBay because they don't sell a lot of items under $5. It might not work as well on cheaper items, since the cost of the time spent on verification could approach or exceed the cost of the answer.<BR/><BR/>Finally, there's the issue of what quality you achieve after all this. 90% correct answers perhaps with all the noise, ambiguity, and cheating? Is that good enough? Considering speed, accuracy, and cost, is that competitive with using an automated solution?<BR/> <BR/>I'm not saying Amazon Mechanical Turk could never work for any problem; Google Answers does this in some form for some problems. I'm also not saying there aren't ways to improve the quality of the answers provided.<BR/><BR/>I do think it is complicated and expensive to scale this up and keep the quality high. I also think, for many problems, it might be nearly as hard to dig the correct answer out of all the answers provided as it would have been to try to solve the original problem.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131285044928546392005-11-06T05:50:00.000-08:002005-11-06T05:50:00.000-08:00I think it's an interesting concept. Imagine for ...I think it's an interesting concept. Imagine for example that Findory (or Memigo ;-) have trouble classifying an article automatically. If the article exceeds some popularity threshold (that makes it worthwhile to do something about it) it may make sense to ask a human to classify it manually.<BR/><BR/>As for incompetent or malicious responses, treating them isn't much different than treating malicious ratings on Amazon, Netflix, or wherever: have multiple humans do the same task many times, and gradually build trust metric for each human ...drone.Costashttps://www.blogger.com/profile/05157234618935426983noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131252974697586002005-11-05T20:56:00.000-08:002005-11-05T20:56:00.000-08:00One question, why is the web service part so inter...One question, why is the web service part so interesting? Its actually pretty hard to think of a useful programmatic system that could use an web services API call that takes a few hours to return the data. Users tend to like faster performance than that.<BR/><BR/>Maybe for offline processing tasks...but its unclear to me why those types of things would be a proper web service - seems like systems such as ScriptLance that provide a skills marketplace are more reasonable to fulfill this kind of demand.<BR/><BR/>Maybe this could be a cheap BPO marketplace but overall I'm a little unclear why this will be really useful beyond that.chadhttps://www.blogger.com/profile/12272480610329333973noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131135529878453192005-11-04T12:18:00.000-08:002005-11-04T12:18:00.000-08:00As far as people getting it right. The SETI proje...As far as people getting it right. The SETI project also solved this problem. Send the same work packet out multiple times.<BR/><BR/>Run some trends on this and eliminate the bad results keeping the only the good.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131135234239258942005-11-04T12:13:00.000-08:002005-11-04T12:13:00.000-08:00I think you have this all wrong. It's brilliant. ...I think you have this all wrong. It's brilliant. I can have people, for less than minimum wage, perform mind numbing tasks such as the one listed to make automotive titles more human friendly.<BR/><BR/>It takes the brilliance of the SETI projects distributed computing and applies it to a low wage resource pool. Fantastic idea.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131133502948538422005-11-04T11:45:00.000-08:002005-11-04T11:45:00.000-08:00To say nothing of the ethics of computers outsourc...To say nothing of the ethics of computers outsourcing tasks to humans. Did we learn nothing from Terminator 2?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131123271300195542005-11-04T08:54:00.000-08:002005-11-04T08:54:00.000-08:00Thanks, Guido. Great explanation. It is compelli...Thanks, Guido. Great explanation. It is compelling for some applications.<BR/><BR/>However, you've subtly made the assumption that the anonymous humans are always honest and correct. <BR/><BR/>They won't be. Given that money is involved, I think you're going to find these pesky humans are pretty clever at doing things that might be in their self-interest, but not in the interest of people trying to get correct answers.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131123173449923962005-11-04T08:52:00.000-08:002005-11-04T08:52:00.000-08:00That last comment wasn't very clear (I guess I'm s...That last comment wasn't very clear (I guess I'm still asleep). What I meant to say was that one of the problems with AI right now is that it can't ask questions like a child or interact with the real world. But with this software it can.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1131122290478373362005-11-04T08:38:00.000-08:002005-11-04T08:38:00.000-08:00I think that it takes a while for all of us to und...I think that it takes a while for all of us to understand the full impact of this technology. They've provided a web service interface to human labor. It's not necessarily humans asking humans other questions. It can be computers asking humans questions. <BR/><BR/>Imagine you are writing software that recognizes faces. It works pretty well but sometimes it gets confused. When that happens it asks for help. Instead of popping up a window and waiting for a local operator to help out it just sends out a web service request and waits a few minutes for a reply.<BR/><BR/>I think software like that is just the first step towards AI. You use anonymous humans to do stuff that the computer isn't very good at.Anonymousnoreply@blogger.com