think we didn’t...
think we didn’t even go far enough. Whatever it is, question every single aspect
of conventional wisdom. “Is that the right way to do it or can we break that and
make it better?”
That’s also dangerous, because, if you are doing a lot of paradigm innovation,
call it—which is not a good word—but if you are breaking boundaries
elsewhere, maybe you need to be very within boundaries on other fronts.
Livingston: Do you remember a time when you were worried about something?
Schachter: Site’s down. Site’s slow. Table crash—MySQL corrupted a table.
That happened all the time. A great deal of what we did was putting out fires.
We didn’t have a lot of process management in place, which probably hurt us a
great deal.
A week after the acquisition, the power of the data center dropped and corrupted
every single machine. We were down for like 48 hours. That was horrific.
The power bounced in the network; the machines didn’t come back up
because they weren’t configured quite properly. We weren’t careful about that.
In general, assume that whatever you are doing is going to go wrong. How
can you make it so that it will go faster when it does go wrong? Because it will.
For example, the rebuild script takes 24 hours, but that’s not a big deal because
this part of the system isn’t live yet. But when it is live and it takes 24 hours to
redo, that’s a big deal. So fix it. Make it work in 2 or whatever. There’s a lot of
stuff that you can’t get around due to SQL, like you can’t change the database
without bringing the site down.
Livingston: What kind of technology inspired you?
Schachter: Inspired? We built in Perl, MySQL, Apache. Very standard LAMP
stack kind of stuff. That was the standard mechanism for everything.
Livingston: Would you do anything differently if you could?
Schachter: Knowing what I know now, I would have designed the back-end
architecture differently, and that would have saved a lot of work now. Scaling
past one machine, one database, is very challenging, even with replication. The
tools that are there are not quite right.
For example, when you add things to a table and it numbers them, that
means you can’t have a second machine also adding to them because the numbers
will collide. So what do you do? You have to come up with some
Joshua Schachter 227
completely different way to do it. Do you have a central server that hands out
number sets, or do you come up with something that’s not numbers? Do you
use random numbers and hope they never collide? Whatever it is, autoassigned
| ← capital. | IDs just don’t → |