tag:blogger.com,1999:blog-6569681.post113889792080799686..comments2024-03-29T05:14:10.903-07:00Comments on Geeking with Greg: Early Amazon: Splitting the websiteGreg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-6569681.post-60422905878437296892009-07-19T15:22:31.099-07:002009-07-19T15:22:31.099-07:00I think Akamai used Netapp to solve these kind of ...I think Akamai used Netapp to solve these kind of problems, but even that is unneccessary as free open source solutions are available: ZFS on the free and open source OpenSolaris operating system now integrates with Lustre 3.0, so you can use Lustre as your distributed file system and have ZFS as the back end of it (for block level checksumming and more reliability) so you don't need the insane amount of duplication of data that Google has in their Google File System or the insane Netapp bill that Akamai pays. Looks like Sun has solved everybody's problems and open sourced the code, but perhaps they are too late?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-35921104983472104252007-09-16T05:47:00.000-07:002007-09-16T05:47:00.000-07:00What about ZFS?What about ZFS?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1166941516795295952006-12-23T22:25:00.000-08:002006-12-23T22:25:00.000-08:00We spent quite a bit of time looking for a Distrib...We spent quite a bit of time looking for a Distributed File System for our large web application. I liked MogileFS, but ran into problems with maintaining backwards compatibility (since its not a mountable file system) and scaling that happens when you try to put more than 500 million objects in it. <BR/><BR/>We just signed a contract with Isilon systems for a 150tb usable distributed filesystem. We were able to haggle them down to a reasonable price (about 20% more of what I estimated Mogile hardware would cost). <BR/><BR/>I'm excited about Isilon. They've got a great product. Their support contract is incredible expensive, and thats the part we grilled on them to reduce cost.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1146779477100417572006-05-04T14:51:00.000-07:002006-05-04T14:51:00.000-07:00Lustre works nothing like CODA, but to really take...Lustre works nothing like CODA, but to really take advantage of it you need really big clusters.<BR/><BR/>GFS runs on shared media, i.e. SANs.<BR/><BR/>Another distributed file system in the Linux world that looks interesting is OCFS. I haven't had time to try it out, but the design is very close to ext3 and much simpler than GFS.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1146777935578425702006-05-04T14:25:00.000-07:002006-05-04T14:25:00.000-07:00Thanks, Jeff, Lustre looks interesting. It seems ...Thanks, Jeff, Lustre looks interesting. It seems to be an offshoot of Coda.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1146765063775105772006-05-04T10:51:00.000-07:002006-05-04T10:51:00.000-07:00I just came in off the DB War stories on O'Reilly ...I just came in off the DB War stories on O'Reilly Radar. <BR/><BR/>Has anyone mentioned <A HREF="http://www.lustre.org/" REL="nofollow">Lustre</A>?<BR/><BR/>It is a high performance distributed POSIX compliant file system (as opposed to a mere cluster FS with Redhat GFS). Lustre is open source, with support and premium packages from Cluster FS. In short it provides a distributed RAID 0.<BR/><BR/>It supports some levels of redundancy, at the server level. In early 2007 their roadmap is support a distributed RAID 5. Check out the <A HREF="http://www.clusterfs.com/roadmap.html" REL="nofollow">RoadMap</A> on the Cluster FS site. It's the closest thing I've found to the Google File System and its more general purpose. Has anyone else had experience with it?jeff.daltonhttps://www.blogger.com/profile/12887721174386884522noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1146735878043193042006-05-04T02:44:00.000-07:002006-05-04T02:44:00.000-07:00This is also very true of "the world's largest fin...This is also very true of "the world's largest financial exchange" (I'll let you google to find out who). They are at the limits of Oracle running on the biggest Sun box you can buy. And since it's financial you absolutely require minimal caching and ACID properties. The fact it's an exchange also means it's extremely difficult to scale out the db - the user account balance table is checked at least once for every single write operation (of which there are 300/second).<BR/><BR/>Where do you go if Oracle isn't big enough? You go your own way.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1139438654596453422006-02-08T14:44:00.000-08:002006-02-08T14:44:00.000-08:00As the personwho had to maintain and update the we...As the personwho had to maintain and update the website management code after Greg moved on to more fun tasks I can personally attest to the limits of Greg's sysadmin knowledge, <BR/><BR/>Seriously, amazon's web site operations were so "high-end" that there wasn't anything they could buy (no COTS solutions) that would handle the situations amazon was encountering. From the CMS to the web site management tools, to inventory and customer databases, it was all (at least originally) written in-house out of necessity. Sure, it ran on top of hardware, OSes and database systems built by others, but that was about it.<BR/><BR/>I'll bet this is also true of places like eBay or any other large ecommerce stores.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1139019713804653802006-02-03T18:21:00.000-08:002006-02-03T18:21:00.000-08:00GFS does look very cool. POSIX compliant too. Ex...<A HREF="http://www.redhat.com/en_us/USA/home/solutions/gfs/" REL="nofollow">GFS</A> does look very cool. POSIX compliant too. Excellent.<BR/><BR/>Do you know of any information on performance? On exactly what happens when a node goes down? On how it tries to handle really nasty stuff like a network failure that partitions the network?<BR/><BR/>I'm having a hard time finding anything other than marketing goo white papers on it. Can't seem to get the answers to those questions.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1138934892545247752006-02-02T18:48:00.000-08:002006-02-02T18:48:00.000-08:00Sistina's Global File System, now part of Red Hat ...Sistina's Global File System, now part of Red Hat and open source.<BR/><BR/>I have heard good things about it from people using it in real world situations.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1138924260456456062006-02-02T15:51:00.000-08:002006-02-02T15:51:00.000-08:00Thanks, Nick. I have heard some people talking ab...Thanks, Nick. I have heard some people talking about MogileFS.<BR/><BR/>As you said, it's not a file system. From <A HREF="http://www.danga.com/mogilefs/" REL="nofollow">their site</A>, "you don't run regular Unix applications or databases against MogileFS. It's meant for archiving write-once files and doing only sequential reads."<BR/><BR/>But, if those constraints are fine for your needs, it might be a good option.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1138905670055240112006-02-02T10:41:00.000-08:002006-02-02T10:41:00.000-08:00Findory uses an architecture roughly similar to wh...Findory uses an architecture roughly similar to what I described for Amazon in 1997 but quite a bit simpler.<BR/><BR/>As for other options, I think AFS might work if you don't need RW replication. The new MySQL cluster product looks really good. It might work if you really need a database, not a file system, and you have the budget for a larger number of machines (because all data is kept in memory). I think CODA might be an option if you're willing to use it on a production system (which they recommend against).<BR/><BR/>What would work best probably depends on your specific needs.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1138905273895606882006-02-02T10:34:00.000-08:002006-02-02T10:34:00.000-08:00Gregg,I was wondering how you've solved the clust...Gregg,<BR/>I was wondering how you've solved the clustered file system and grid computing issues at Findory from a 10,000 foot view. I have a number of apps I'd like to write that need such features. Unless you work at Google we all share the same pain =). Nutch on the Java side of things looks interesting but it needs more support.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-1138904590356590142006-02-02T10:23:00.000-08:002006-02-02T10:23:00.000-08:00Hey Greg,Just wanted to say I really love this Ear...Hey Greg,<BR/>Just wanted to say I really love this Early Amazon series. It's a really interesting look at what it was like back then (I was there this summer interning).<BR/><BR/>Thanks for the great writing,<BR/>IanIan Seffermanhttps://www.blogger.com/profile/10314628938760728819noreply@blogger.com