Sarah Davies

I’m at a sxsw panel on scaling websites. Here are the speakers:

	Serkan Piantino Facebook Inc
	Alan Schaaf Imgur LLC
	Kevin Weil Twitter
	Christopher Slowe Reddit
	Jason Kincaid TechCrunch

Imgur was released a year ago on Reddit. It was on a shared hosting plan. It lasted two days before the site was terminated for generating too much traffic. The site went down. Imgur moved to Mediatemple. That lasted three weeks, so they moved again, and again. Imgur moved four or five times in four months, scaling up to a better server with more bandwidth. Imgur went to foxhole.net, a content delivery network, because they have servers all over the world. That allowed the devs to concentrate on making the site faster rather than keeping the servers up.

Reddit is running on EC2 using about 50 machines. They have 20 app servers. They got a big speed boost by going single-threaded. They use Postgress and memcache.

Twitter started as a rails application tied to a single MySQL database. They have an open source queuing system, so they can do asynchronous processing.

A lot of the core architecture behind Facebook is still LAMP. They run newsfeed, ads, and search all on separate servers. On Facebook, you’re typically friends with 0-5000 people, whereas on Twitter you can follow millions of people, so they can render everything on the fly where Twitter can’t.

What is Reddit using for indexing?
They roll their own indeces using memcachedb. They are getting a .02% failure rate with that.

At what point are LAMP stacks not enough?
Knowing when a machine needs to be replaced is tough. Facebook has a monitoring system set up with alerts and teams dedicated to figuring out where they will have scaling problems before they have them. Monitor. Monitor. Monitor.

How do you scale search?
Search is really hard. The metric you are measured against is Google, which is a ridiculous standard. Reddit does about two queries per second. Getting quality results is really hard to tweak. It’s very qualitative in terms of what is “good” search.

What was the first thing that blew up?
Imgur had apache blow up first. “It was like trying to hammer a nail with a sledgehammer.” Twitter originally put the whole social graph in a MySQL database, but it was getting into the billions of rows. They had to build their own social graph store. They are in the process of open sourcing it.

What modules is Facebook using to convert PHP to C++?
They built a project called hiphop which compiles all their php down to binary C++. There are whitepapers about it, and it’s open source.

How do you deal with deployment?
Facebook and Twitter use BitTorrent to deploy builds to all their servers, cutting deployment from 12 minutes down to 30 seconds. Reddit cobbled something together in perl.

Why haven’t any of you used proprietary databases?
We prefer to work with open source. As you deal with scaling problems, you have to peak under the hood and see what you can tweak. Calling a vendor is a pain. Oracle is expensive. We like to be nimble and play well with the community.

Posted by Sarah Davies, filed under Facebook, FOSS, SxSW, technology, the intarwebs, twitter. Date: March 14, 2010, 8:28 am | View Comments