My Hidden Room

So one of the issues that I deal with a lot is tuning web applications to not suck.  This is done by a few things; by monitoring, by profiling, caching, caching (CACHING!), and by tuning.  The process for making a web application more awesome basically boils down to this list of steps:

1. Monitor your application performance (http threads, cpu, memory, thread response time, etc)
2. Profile your code
3. Fix slow requests/implement caching
4. Tune your web-server
5. Goto 2.

Monitoring

Monitoring the response time of your application is useful and awesome for making positive changes to your environment .  This means paying attention to your application response time, cpu, memory, network traffic, disk IO, disk capacity, etc.  All those metrics that say whether your application is healthy or not.  There are a few different tools available for this that all work pretty well, here’s an incomplete list:

• Cacti – http://www.cacti.net
• Munin – http://www.munin-monitoring.org
• Cricket – http://cricket.sourceforge.net

They all work well and solve slightly different problems.  Pick the one you like most.  I’m a fan of Cacti.

Profiling

Profiling means being able to see how long each call that an application makes takes to execute.  It’s invaluable for getting a feel for what parts, what components, of your application perform badly or perform well.

Caching

Whenever an application fetches data from a resource, that’s an opportunity to improve performance.  Every time something is fetched, there’s the ability to take that result set, and keep it.  Caching the results of database calls, of external API lookups, of intermediate steps, all these things leave lots of room for improving performance. Memcached is the de facto standard for a caching engine.

Cache early, cache often!

(Apache) Tuning

A well configured web-server is crucial to a happy environment.  This means not running into swap, not wasting time with connections that are dead, and other such things that waste time.  In short, don’t look up what you don’t need, don’t save what you don’t need, and be efficient.  Here are some basic things that apply to Apache:

   KeepAlive          Off (Or On, see below, it depends on workload)
   Timeout            5
   HostnameLookups    Off
   MaxClients         (RAMinMB * 0.8) / (AverageThreadSizeInMB)
   MinSpareServers    (0.25 * MaxClients)
   MaxSpareServers    (0.75 * MaxClients)

About these parameters:

• KeepAlive – this controls whether when one request from a client to the web-server is completed whether that thread will remain connected to the clients for subsequent requests.  In high-scale applications, this can lead to contention for available resources.  Some workloads, however, benefit from keeping this on.  If you are serving lots of different content types on a page to a client, leaving this on can be a good thing.  Test it out, YMMV.
• Timeout — how long before we assume that the client has gone away and won’t be requesting further data.  The default is 300.  It is in seconds.  This value is aggressive.
• HostnameLookups — this is for logging, and if it is on, each client will cause a DNS request to be made.  This slows down the request.
• MaxClients — the total number of threads that the server will allow to run.  Each thread consumes memory.  This model assumes that 20% overhead for other system tasks is appropriate and will keep us out of swap.  On machines above 16GB of ram, use 0.9 instead of 0.8.
• MinSpareServers — the fewest threads that Apache will leave running.  Setting this too low will result in load spikes when traffic increases.
• MaxSpareServers — the most spare, unutilized threads that Apache will leave running.  Setting this too low will result in lots of process thrashing and threads are used and then terminated.  The tradeoff is utilized Ram.

There are a lot of other things that can be done as well, so don’t take this as a complete set…

These are my handout style tips on performance tuning.  There are whole volumes of books dedicated to this topic.  Some great resources include:

• http://www.mysqlperformanceblog.com/
• The Art of Capacity Planning (Allspaw)
• http://www.standalone-sysadmin.com/blog/

-Gabriel

So at work, I had a blog.  Not my blog, of course.  One I support.  So this blog was thrown together quickly to facilitate business goals.  Like you do.  And that’s great.  We met the deadline.  We got the product functional.  But performance kinda sucked.  A little backstory.  Here’s how it got set up to start.

We love virtualization here.  Everyone should.  It’s a fantastic way to take adventage of hardware that you’ve got laying around that’s being idle.  Idle hardware is lame. So we run KVM.  This means we get to manage machines more effectively, and can provision things faster.  That’s awesome.

So this blog.  It’s a cluster of boxes, made up of a pair of webservers, and a pair of DB backends.  The DB backends are a master, and a backup.  This is a small project, so these got set up with just 4GB of ram.  So that’s fine.  The web servers are each a VM with 1 vcpu, 4GB of ram, and some disk space.

So we set that all up, and it did fine.  But not great.  here’s what it did.

Incidentally, if you’ve got data you’ve collected for performance with Pylot, and need to CSV the averages to make a pretty graph, here’s my way of extracting the averages from the results HTML to feed into gnuplot.

for file in load-dir/*/results.html ; do cat $file  | egrep 'agents|avg' | head -3 | perl -e 'while(<>){s/<\/?t[dr]>//g; s/(\d+\.?\d*)/\t$1/g; chomp; @p=split /\t/,$_; push @r,$p[1]; }; printf "%s\n", join ",",@r; '; done


So what we can see then from these graphs is that a pair of single core boxes, tuned with the default cherokee+php installation don’t do all that great.  They can handle 1-2 simultaneous requests, but past that, the response time gets pretty bad. That’s where the project got left for a while, until the other day I got the request to make it handle 500 requests per second.  “Wow, shit.” I thought.  So I dove in to see what I could do to improve performance on the blog, and it turned out that there was a lot I could do.

1. Single-core boxes don’t have a lot of performance available to them.
2. Cherokee uses PHP in FastCGI mode, and does a terrible job of tuning it out of the box.  It defaults to a single PHP thread.
3. WordPress is very hungry for CPU time.  It chews up a lot of resources doing its DB queries and wrangling them.

To address these points, I did the following.

The VMs themselves were restarted with more CPU cores — four cores per box.   This allowed me to dive into discovering that Cherokee wasn’t tuned well.  Under a single core, I saw 80% cpu utilization on PHP, with high system wait time.  That sucked.   But after I bumped up the core count, it still looked bad.  Still only one CPU @ 80%. WTF.  So then I turned to Cherokee, and I tuned PHP so that it would invoke 16 children, enough to handle the request volume.  This helped a lot, but there was still room to do better.  So I added APC, the Alternative PHP Cache, to the configuration.  This helped out a lot.  I then looked at wordpress specific caching engines, and settled on using W3-Total-Cache to help out.  This brought performance into the range of fulfilling the customer’s request.  I felt great about it.

I used pylot to graph performance at various points through this project so I could figure out how I was doing a better or worse job of tuning the boxes.

Here’s the performance data showing the improvements that caching at various layers added to the party:

So it turns out that these are some great ways to make sure your blog performs:

• Make sure that there’s enough hardware supporting it
• Cache in depth and breadth, early and often
• Tune your PHP/WebServer to handle the project
• Employ performance testing to measure your real improvements

With some hard work and performance tuning, you can turn an abysmal 7 requests/sec peak performing blog into one that can sustain 350 requests per second, and do it in a tiny fraction of the time.

-Gabriel

Texting me early in the morning without identifying yourself is a fantastic way to piss me off. If it was your goal to antagonize me against your position and company, you did a great job. Also: When I text you back, “do not txt me,” it does NOT mean that you should then call me five minutes later!

Advice to recruiters who haven’t yet pissed me off: Email. It’s great. If I’m interested in the position you have, I’ll let you know! Don’t cold-call me. Don’t cold txt me.

So it’s time for the annual pilgrimage to Burning Man.  Like so many other folks, I’m going.  Unlike a lot of people, I haven’t had the ticket scramble that the selling out process led to — I bought mine back in January.  Anyway, this post isn’t so much about tickets, as it is about the experience, and how I feel about the upcoming trip.

I’m super excited about it.

This year, I’m bringing my own big art project to the Burn.  The Air Kraken has been a ton of work this year, and I’m so happy that I’ve gotten it done, and that it’ll be on playa.  The project has gone really well due to help from my friends by way of Kickstarter contributions, INW’s Art Grant, and construction assistance.   I’m super appreciative to all their wonderful help!

Check out more about the Air Kraken on my project page.

So what else is going on for the Burn?  I’m planning on taking a ton of photographs this year, and that means bringing a lot of storage… I’m bringing 28GB of CF cards, and 6GB of SD cards for two cameras.  I’m bringing a cool selection of lenses, too:  Nikkor 50mm f/1.8 D, Sigma 21-35mm Alpha f/3.5-5.6, Nikkor 70-210mm f/4-5.6 D, and on my D80 the 18-200mm alphabet soup lens as backup.  There should be lots of cool stuff to see.

If you see me on playa, I demand you ask for a portrait.    See you in the desert…

So this past Friday, I went up with some friends to Rattlesnake Lake off exit 32 on I-90.  It was a lot of fun, and we laughed and talked and drank good wine.  It was a lovely time.  I shot a lot of photos, but didn’t catch any meteoroids.  The moon was too full and bright, and there just weren’t that many.  I saw about 10 trails in the sky, which was quite pleasant.   Here are some photos I did shoot.  Funny thing about the full moon… It’s a lot like the full sun.

Specifically, 30 sec @ f/5.6, ISO 200 looks like a daytime exposure.  It’s pretty awesome to have images that look like daytime shots with star trails in the backgrounds!

One thing that I realized about this trip was that I was lacking a good release for my D700.  Fortunately, a quick trip over to Amazon and my Prime membership means that next time, that won’t be a problem for me.

Until next time…

So today, like a few tens of thousands of other burners, I was after a ticket to go burn in the desert. I got in line promptly at 10am, and got the initial queue position of 2008. SWEET, I thought.

But no, there were server problems on their ticketing vendor’s side of things, and that resulted in the queuing system misbehaving and generally punting people way back in line.

So what I did, after waiting for a number of hours was I figured out how to retrieve my old queue position in line, and get my ticket.

Here’s what I did, and what you can, too. This only works on Firefox.

1) Pull up a blank page, and go to the URL about:cache. That’ll pull up the cache entries page where you can look through your browser cache for old URLs you’ve visited. Search the memory cache and then the disk cache for the string “key_queue=” — this will be found in URLs for the Burning Man ticket page links. You’ll want to find all of them, and collect them.

2) Then take the URLs you’ve collected, and load them into your browser. I was able to get immediately to the ticket purchase page doing this, and bought my second-tier ticket.

This worked for me. It might work for you, it might not. But I thought I’d share.

So I had a problem; when I left my previous employer, I had been using my iPhone to interface with work email.  That worked great.  But one problem that I face after I left was that all my contacts were in the Active Directory/Exchange address book system.  I asked for, and received, a CSV of my contact info when I left.

Read more…

Tomorrow is my first day with the new gig.  I resigned from my old job about ten days, and had a nice break.  But now it’s time to get back to things.  I’m a little apprehensive about this whole thing, as this is a new way of working for me.  Changing gears in so many ways after my long time at PopCap (4.5 years – Wow) is filling me with all kinds of feelings of volatility.  What happens now?

I’m eager to get into the swing of things, and past the anxious new beginning with this.   Until tomorrow.

Thanks to flickr user redjar for the image.

I hate airports.  Not flying, flying I love.  But the process of dealing with airlines, and luggage, and most especially the TSA/Security Theater aspect of the process drives me crazy.  I flew down to SJC this week for a technical conference, MySQLConf.  It was a great conference: I met neat people, I learned new things, I got excited about tech again.  I loved that part.  The flight down was fine.  Easy, no problem through the airport.  Only the usual anxiety about the process.

Read more…

CC licensed image via alancleaver on flickr.

Today I went to breakfast at Voula’s Offshore Cafe in the University District.  I had the steak and eggs with rye toast for breakfast.  It was quite tasty.  But this isn’t about my breakfast so much as it’s about the interesting example of information leakage.

I use foursquare because it’s fun, and to find where my friends are going for food, activities, and other things.  I don’t link my foursquare to any other accounts, but other people do.   This morning at Voula’s, I checked in, and saw one other person there.  I’m curious, and so I look at the foursquare profile. Turns out there’s a facebook link on the foursquare login, so I follow it.  Turns out she was sitting at the table next to mine.   I said hello and made a comment about foursquare.  Then I returned to my meal.

Her facebook profile said a lot of things: that we have quite a few friends in common, who she’s seeking, and her full name.   Now, I didn’t say anything to her about any of this, nor attempt to engage in a conversation at this time.  It would have been weird and creepy.  I didn’t want to be That Guy, or a weirdo.  But it was very much an interesting lesson in how we chose to interact online can affect us in the offline world.

When we all got out to the car, I had a conversation with the teenagers about information that you share online, and used this very real example from just now to illustraite that what you do online can have real results in the rest of your life.   Pay attention to what you post, to what you link, to all those things.  It’s very easy to draw connections between people via the social graphs.

There are many wonderful things that I get out of connecting my social graphs.   I get to point people whom I want to connect with me to where they can find me, and how they can do so.  But it bears consideration before you blithely do so.  Everything connected means there aren’t any secrets.