October 1st, 2003

Status update

The switch to Speedera last night went well, but now Speedera's world-wide caches are filling up (by requesting userpics from us, their origin) and is in turn killing db caches on our servers, hindering overall performance.

This'll die down over time, but in the meantime we're racing to finish the project we mentioned here previously to remove the database query that's blocking the most (and blocking now even more because of the smaller effective caches).

Once we get the NetApp, userpics and other media are all moving to it, so the fast disks and caches on our db servers won't be used to do simple stuff like images.

I know you're all getting sick of the waves of suck and fast (and I am too), but we're really getting to a nice place architecture-wise... I think you'll all be happy in a bit here.

Maintenance -- Friends pages

Due to a temporary surge in traffic caused by Speedera's caches filling up with everybody's userpics, we have to temporarily restrict access to friends pages. We're rushing to finish the new friends view code which uses memcached. It should only be for the afternoon (next few hours). As traffic drops tonight we'll turn friends pages back on.

Paid users can still use their friends page, but we're closing off our payment area, lest anybody think this is a marketing ploy to get more sign-ups. We care more about the site being fast than your money.

The rest of the site will still work for all users: comments, recent entry pages, etc. And hopefully it will work a lot better than currently, as friends pages are the biggest burden on the databases. We feel it's more important for the site to mostly work than to not work at all. Of course, our goal is for everything to work fast, and hopefully that'll happen tonight.

This is something we don't plan to do again, and regrettably can't be avoided today. Thanks for your understanding. We're all working as fast as we can....

Update: Friends views are re-enabled, now that traffic has gone down. We're still working on the new code... should be up tonight. It's looking promising.