Category Archives: Ramblings

DNS30 – Spammers

So you may have seen my previous blog post about wrangling Route53.  I got a comment that got marked as spam in my blog configuration, and here’s the text from it.  I’ve seen it all over the place on route53 blog posts:

Hello, Thanks for sharing your article with us.

Route 53 is designed to automatically scale to handle very large query volumes without any intervention from user.We have developed a UI tool for this interface – DNS30 Professional Edition.We also have online interface for this application.

Now, at first, this would seem like a moderately useful kind of post to get.  But it turns out that this is all over the place, exactly the same.  It doesn’t add anything to the conversation, and is just a solicitation for services.  Hell with that!  These clowns are spammers, engaging in comment spam, and that’s really lame.  They’ve pretty much guaranteed that I’ll badmouth them and avoid them in all contexts.  Good job, DNS30!

Building the Air Kraken Trike

Line art concept sketch (by Molly Friedrich)*** More pictures are here ***

Last year, around this time, I started building the Air Kraken Trike.  It’s a project I was inspired to create from the scenes of great big things like it at Burning Man.  This article is an overview of my process, challenges, costs, and thoughts.   I’ll describe how I came to the design, and various elements of building it.  I’ll provide a broad-strokes cost overview of what it took to make it, to take it to the Burn, and to store it.


So what’s an Air Kraken?   The Air Kraken is a fun idea from the steampunk community that has to do with an event in march, called Air Kraken Day, where folks take refuge in bars, and carry umbrellas to keep the sky kraken from descending to feast on folks.  It’s pretty hilarious, and I liked the name, so I went with it.    What else is an Air Kraken?  It’s also a ten foot long, six foot wide metal tricycle made out of new and used materials that has been to several events.

It’s made of steel, of drainage culvert, of bus parts, of old bicycles.  I’ve put more than a little blood, sweat, and tears into the construction and application of it.  It’s been a lot of fun, and I’ve learned a lot. There have been frustrations and mis-steps.  But it’s been overall pretty awesome about it.

It has been a very time intensive project.  I put in a lot of nights and weekends into it.  For about four months, it was effectively my second job.  I’d get home from work around 6ish, eat food, and go out to the shop to work on it.  I’d throw 8-10 hours a day on the weekends into it.  I probably put 600-700 hours into it.  Definitely a lot more than that with the time I wasn’t actively working on it, and “merely” thinking about it.

Continue reading

Web Operations Performance – The Handout

So one of the issues that I deal with a lot is tuning web applications to not suck.  This is done by a few things; by monitoring, by profiling, caching, caching (CACHING!), and by tuning.  The process for making a web application more awesome basically boils down to this list of steps:

  1. Monitor your application performance (http threads, cpu, memory, thread response time, etc)
  2. Profile your code
  3. Fix slow requests/implement caching
  4. Tune your web-server
  5. Goto 2.


Monitoring the response time of your application is useful and awesome for making positive changes to your environment .  This means paying attention to your application response time, cpu, memory, network traffic, disk IO, disk capacity, etc.  All those metrics that say whether your application is healthy or not.  There are a few different tools available for this that all work pretty well, here’s an incomplete list:

  • Cacti –
  • Munin –
  • Cricket –

They all work well and solve slightly different problems.  Pick the one you like most.  I’m a fan of Cacti.


Profiling means being able to see how long each call that an application makes takes to execute.  It’s invaluable for getting a feel for what parts, what components, of your application perform badly or perform well.


Whenever an application fetches data from a resource, that’s an opportunity to improve performance.  Every time something is fetched, there’s the ability to take that result set, and keep it.  Caching the results of database calls, of external API lookups, of intermediate steps, all these things leave lots of room for improving performance. Memcached is the de facto standard for a caching engine.

Cache early, cache often!

(Apache) Tuning

A well configured web-server is crucial to a happy environment.  This means not running into swap, not wasting time with connections that are dead, and other such things that waste time.  In short, don’t look up what you don’t need, don’t save what you don’t need, and be efficient.  Here are some basic things that apply to Apache:

   KeepAlive          Off (Or On, see below, it depends on workload)
   Timeout            5
   HostnameLookups    Off
   MaxClients         (RAMinMB * 0.8) / (AverageThreadSizeInMB)
   MinSpareServers    (0.25 * MaxClients)
   MaxSpareServers    (0.75 * MaxClients)

About these parameters:

  • KeepAlive – this controls whether when one request from a client to the web-server is completed whether that thread will remain connected to the clients for subsequent requests.  In high-scale applications, this can lead to contention for available resources.  Some workloads, however, benefit from keeping this on.  If you are serving lots of different content types on a page to a client, leaving this on can be a good thing.  Test it out, YMMV.
  • Timeout — how long before we assume that the client has gone away and won’t be requesting further data.  The default is 300.  It is in seconds.  This value is aggressive.
  • HostnameLookups — this is for logging, and if it is on, each client will cause a DNS request to be made.  This slows down the request.
  • MaxClients — the total number of threads that the server will allow to run site.  Each thread consumes memory.  This model assumes that 20% overhead for other system tasks is appropriate and will keep us out of swap.  On machines above 16GB of ram, use 0.9 instead of 0.8.
  • MinSpareServers — the fewest threads that Apache will leave running.  Setting this too low will result in load spikes when traffic increases.
  • MaxSpareServers — the most spare, unutilized threads that Apache will leave running.  Setting this too low will result in lots of process thrashing and threads are used and then terminated.  The tradeoff is utilized Ram.

There are a lot of other things that can be done as well, so don’t take this as a complete set…

These are my handout style tips on performance tuning.  There are whole volumes of books dedicated to this topic.  Some great resources include:


Of Tuning WordPress on Cherokee

So at work, I had a blog.  Not my blog, of course.  One I support.  So this blog was thrown together quickly to facilitate business goals.  Like you do.  And that’s great.  We met the deadline.  We got the product functional.  But performance kinda sucked.  A little backstory.  Here’s how it got set up to start.

We love virtualization here.  Everyone should.  It’s a fantastic way to take adventage of hardware that you’ve got laying around that’s being idle.  Idle hardware is lame. So we run KVM.  This means we get to manage machines more effectively, and can provision things faster.  That’s awesome.

So this blog.  It’s a cluster of boxes, made up of a pair of webservers, and a pair of DB backends.  The DB backends are a master, and a backup.  This is a small project, so these got set up with just 4GB of ram.  So that’s fine.  The web servers are each a VM with 1 vcpu, 4GB of ram, and some disk space.

So we set that all up, and it did fine.  But not great.  here’s what it did.

Incidentally, if you’ve got data you’ve collected for performance with Pylot, and need to CSV the averages to make a pretty graph, here’s my way of extracting the averages from the results HTML to feed into gnuplot.

for file in load-dir/*/results.html ; do cat $file  | egrep 'agents|avg' | head -3 | perl -e 'while(<>){s/<\/?t[dr]>//g; s/(\d+\.?\d*)/\t$1/g; chomp; @p=split /\t/,$_; push @r,$p[1]; }; printf "%s\n", join ",",@r; '; done

So what we can see then from these graphs is that a pair of single core boxes, tuned with the default cherokee+php installation don’t do all that great.  They can handle 1-2 simultaneous requests, but past that, the response time gets pretty bad. That’s where the project got left for a while, until the other day I got the request to make it handle 500 requests per second.  “Wow, shit.” I thought.  So I dove in to see what I could do to improve performance on the blog, and it turned out that there was a lot I could do.

  1. Single-core boxes don’t have a lot of performance available to them.
  2. Cherokee uses PHP in FastCGI mode, and does a terrible job of tuning it out of the box.  It defaults to a single PHP thread.
  3. WordPress is very hungry for CPU time.  It chews up a lot of resources doing its DB queries and wrangling them.

To address these points, I did the following.

The VMs themselves were restarted with more CPU cores — four cores per box.   This allowed me to dive into discovering that Cherokee wasn’t tuned well.  Under a single core, I saw 80% cpu utilization on PHP, with high system wait time.  That sucked.   But after I bumped up the core count, it still looked bad.  Still only one CPU @ 80%. WTF.  So then I turned to Cherokee, and I tuned PHP so that it would invoke 16 children, enough to handle the request volume.  This helped a lot, but there was still room to do better.  So I added APC, the Alternative PHP Cache, to the configuration.  This helped out a lot.  I then looked at wordpress specific caching engines, and settled on using W3-Total-Cache to help out.  This brought performance into the range of fulfilling the customer’s request.  I felt great about it.

I used pylot to graph performance at various points through this project so I could figure out how I was doing a better or worse job of tuning the boxes.

Here’s the performance data showing the improvements that caching at various layers added to the party:

So it turns out that these are some great ways to make sure your blog performs:

  • Make sure that there’s enough hardware supporting it
  • Cache in depth and breadth, early and often
  • Tune your PHP/WebServer to handle the project
  • Employ performance testing to measure your real improvements

With some hard work and performance tuning, you can turn an abysmal 7 requests/sec peak performing blog into one that can sustain 350 requests per second, and do it in a tiny fraction of the time.



Pro Tip: For recruiters…

Texting me early in the morning without identifying yourself is a fantastic way to piss me off. If it was your goal to antagonize me against your position and company, you did a great job. Also: When I text you back, “do not txt me,” it does NOT mean that you should then call me five minutes later!

Advice to recruiters who haven’t yet pissed me off: Email It’s great. If I’m interested in the position you have, I’ll let you know! Don’t cold-call me. Don’t cold txt me.

It’s that time of year…

Air Kraken Trike; shooting fireSo it’s time for the annual pilgrimage to Burning Man.  Like so many other folks, I’m going.  Unlike a lot of people, I haven’t had the ticket scramble that the selling out process led to — I bought mine back in January.  Anyway, this post isn’t so much about tickets, as it is about the experience, and how I feel about the upcoming trip.

I’m super excited about it.

This year, I’m bringing my own big art project to the Burn.  The Air Kraken has been a ton of work this year, and I’m so happy that I’ve gotten it done, and that it’ll be on playa.  The project has gone really well due to help from my friends by way of Kickstarter contributions, INW’s Art Grant, and construction assistance.   I’m super appreciative to all their wonderful help!

Check out more about the Air Kraken on my project page.

So what else is going on for the Burn?  I’m planning on taking a ton of photographs this year, and that means bringing a lot of storage… I’m bringing 28GB of CF cards, and 6GB of SD cards for two cameras.  I’m bringing a cool selection of lenses, too:  Nikkor 50mm f/1.8 D, Sigma 21-35mm Alpha f/3.5-5.6, Nikkor 70-210mm f/4-5.6 D, and on my D80 the 18-200mm alphabet soup lens as backup.  There should be lots of cool stuff to see.

If you see me on playa, I demand you ask for a portrait. ;-)   See you in the desert…

(Not) Shooting the Perseids

So this past Friday, I went up with some friends to Rattlesnake Lake off exit 32 on I-90.  It was a lot of fun, and we laughed and talked and drank good wine.  It was a lovely time.  I shot a lot of photos, but didn’t catch any meteoroids.  The moon was too full and bright, and there just weren’t that many.  I saw about 10 trails in the sky, which was quite pleasant.   Here are some photos I did shoot.  Funny thing about the full moon… It’s a lot like the full sun. ;)

Specifically, 30 sec @ f/5.6, ISO 200 looks like a daytime exposure.  It’s pretty awesome to have images that look like daytime shots with star trails in the backgrounds!

One thing that I realized about this trip was that I was lacking a good release for my D700.  Fortunately, a quick trip over to Amazon and my Prime membership means that next time, that won’t be a problem for me.

Until next time…

Burning Man Tickets – How to reclaim your queue position

So today, like a few tens of thousands of other burners, I was after a ticket to go burn in the desert. I got in line promptly at 10am, and got the initial queue position of 2008. SWEET, I thought.

But no, there were server problems on their ticketing vendor’s side of things, and that resulted in the queuing system misbehaving and generally punting people way back in line.

So what I did, after waiting for a number of hours was I figured out how to retrieve my old queue position in line, and get my ticket.

Here’s what I did, and what you can, too. This only works on Firefox.

1) Pull up a blank page, and go to the URL about:cache. That’ll pull up the cache entries page where you can look through your browser cache for old URLs you’ve visited. Search the memory cache and then the disk cache for the string “key_queue=” — this will be found in URLs for the Burning Man ticket page links. You’ll want to find all of them, and collect them.

2) Then take the URLs you’ve collected, and load them into your browser. I was able to get immediately to the ticket purchase page doing this, and bought my second-tier ticket.

This worked for me. It might work for you, it might not. But I thought I’d share.

Remote Working – Day Zero

Tomorrow is my first day with the new gig.  I resigned from my old job about ten days, and had a nice break.  But now it’s time to get back to things.  I’m a little apprehensive about this whole thing, as this is a new way of working for me.  Changing gears in so many ways after my long time at PopCap (4.5 years – Wow) is filling me with all kinds of feelings of volatility.  What happens now?

I’m eager to get into the swing of things, and past the anxious new beginning with this.   Until tomorrow.