We met on Aug. 12th at the Black Sheep in Cambridge and had 10 very interesting CTOs from Boston. It was just our first, but I learned a great deal already. It felt like the conversation hovered around hiring, devops, QA but there was a lot to be gleaned from the experiences of many around the table. I know there are other similar groups, but nothing beats face-to-face and great food with good folks (@justinsheehy, @graysky, @shearic, @agoeldi, @YoavShapira, @rseanlinday and a few others more).

Our next lunch will be on Sept. 9th at the Black Sheep as well and we’ll be rotating a small number of the folks to let some of the other interested join us. Please forgive me for the backlog, but we want folks to get to know each other and rotating the entire table every time might just not work.

Please don’t worry about the backlog, if you think you can benefit from meeting other CTOs and having them for a whole lunch to ask whatever you wanted in a relaxed yet confidential setting, please sign up for our Boston CTO Lunch.

Just to make sure this post doesn’t end up on the lightweight category, I’ll link to a good post for emailing admins tracebacks when exceptions occur. In case you don’t notice, the post is now pointing to a new feature in AppEngine that stores, keeps counters and generate cool reports called ereporter. Unfortunately, I have not had much luck configuring that one. However, I did get a hook into Django 1.1 finally to capture exceptions, but not as advertised on this AppEngine Django article, instead I ended up using a Django Middleware’s process_exception interface.

09-26 08:38PM 11.392 / 500 28360ms 1400cpu_ms 0kb Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9,gzip(gfe)
24.x.x.x - - [26/Sep/2009:20:38:39 -0700] "GET / HTTP/1.1" 500 0 - "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9,gzip(gfe)" "www.abtests.com"

E 09-26 08:38PM 39.749
: 
Traceback (most recent call last):
  File "/base/data/home/apps/.../3.336622978473149430/main.py", line 29, in 
    appengine_django.InstallAppEngineHelperForDjango()
  File "/base/data/home/apps/.../3.336622978473149430/appengine_django/__init__.py", line 517, in InstallAppEngineHelperForDjango
    InstallAuthentication()
  File "/base/data/home/apps/.../3.336622978473149430/appengine_django/__init__.py", line 407, in InstallAuthentication
    from django.contrib.auth import tests as django_tests
  File "/base/python_lib/versions/third_party/django-1.1/django/contrib/auth/tests/__init__.py", line 2, in 
    from django.contrib.auth.tests.views \
  File "/base/python_lib/versions/third_party/django-1.1/django/contrib/auth/tests/views.py", line 9, in 
    from django.test import TestCase
  File "/base/python_lib/versions/third_party/django-1.1/django/test/__init__.py", line 6, in 
    from django.test.testcases import TestCase, TransactionTestCase
  File "/base/python_lib/versions/third_party/django-1.1/django/test/testcases.py", line 1, in 
    import re

• Create Load Balancer (configure health check, etc)
• Launch Instances (but wait until they’re ready)
• Register with LB

Simple, right? The issues arise, when things start to go wrong. For example, the output all of those commands use those pesky instance ids. If I want to ssh into one of those machines, I have to look up the hostname. Annoying. If I want to check URLs from those machines (too many steps to write here). Annoying. If I want to drop into distributed shell (dsh), I have to create a group file. Annoying. If I want to get some metrics or check the health of the instances, I have to setup more command line tools. Annoying. I think by now you get the point. Enter EzCloud.

[elias@manta] ~/envs/boto $ ezcloud
Welcome to ezcloud!

# List available/configure load balancers
>>> cloud.lbs()
LoadBalancers:lookery-lb1,lookery-lb2,...

# Show their status. If all in service, then just show OK.
>>> cloud.lbs().status()
Load Balancer:  lookery-lb1
Everything is OK.
Load Balancer:  lookery-lb2
Everything is OK.

# If there's a problem, show full details.
>>> cloud.lbs().status()
Load Balancer:  lookery-lb1
Everything is OK.
Load Balancer:  lookery-lb2
1 instances out of service.
ec2-XXX.compute-1.amazonaws.com wrong availability zone

# Now to the good stuff. If I want to see instances, I want to see HOSTNAMEs!
>>> cloud.lbs()[0].instances
ec2-XXX.compute-1.amazonaws.com	us-east-1b	2009-07-16T22:54:28	running	ami-AAA
ec2-YYY.compute-1.amazonaws.com	us-east-1b	2009-07-16T22:54:28	running	ami-AAA
...

# Metrics, you ask me? Here you go.
>>> cloud.lbs()[1].metrics()
Metrics(lookery-lb1):[u'requestcount', u'healthyhostcount', u'latency', u'unhealthyhostcount']

# Functions under metrics, take start, end, period, zones.
>>> cloud.lbs()[1].metrics().requestcount()
Time Sum
2009-07-17T03:12:00Z 1149389.0
2009-07-17T02:12:00Z 1222028.0
2009-07-17T01:12:00Z 1266844.0
...

# VS:
# bin/mon-get-stats RequestCount --start-time 2009-07-17T00:00:00 \
#      --end-time 2009-07-18T00:00:00 --period 3600 --namespace "AWS/ELB" \
#      --statistics "Sum" --dimensions "LoadBalancerName=lookery-lb1, AvailabilityZone=us-east-1a, AvailabilityZone=us-east-1b"
#
# You decide.

# Now, what if I want to do a quick HTTP GET check on all of the machines (besides the health check)
>>> cloud.lbs()[1].instances.get('/some-test-uri-path')
(200, u'ec2-XXX.compute-1.amazonaws.com')
(200, u'ec2-YYY.compute-1.amazonaws.com')
(200, u'ec2-ZZZ.compute-1.amazonaws.com')
...

# OK, I want the whole response
>>> cloud.lbs()[1].instances.get('/some-test-uri-path').debug()
----------------------------------------
(200) ec2-XXX.compute-1.amazonaws.com
content-length : 118
x-powered-by : PHP/5.2.6-1+lenny3
vary : Accept-Encoding
server : Apache/2.2.9
connection : close
date : Fri, 17 Jul 2009 16:14:52 GMT
content-type : text/plain
{"error":"bad_request","description":"Missing required parameter. Please pass either format or valid redirect url."}
...

# Filter by response code, anything that doesn't return 200, please show me.
>>> cloud.lbs()[0].instances.get('/some-test-uri-path').expect(200)
ec2-NNN.compute-1.amazonaws.com	us-east-1b	2009-07-16T22:54:28	running	ami-AAA

# Many operations return EzCloud's InstanceList, so you can do things like write a dsh group file to do some debugging
>>> cloud.lbs()[0].instances.get('/some-test-uri-path').expect(200).dsh('foo')
user@ec2-NNN.compute-1.amazonaws.com
>>> ^Z 
[1]+  Stopped                 ezcloud
(boto)[elias@manta] ~/envs/boto $ dsh -g foo -M -c -- 'hostname'
user@ec2-NNN.compute-1.amazonaws.com: ec2-NNN

# You can even do some basic filtering/orderby (placement equals ZONE)
(boto)[elias@manta] ~/envs/boto $ fg
>>> cloud.lbs()[0].instances.get('/some-test-uri-path').expect(200).orderby('placement')
ec2-X.compute-1.amazonaws.com	us-east-1a	2009-07-16T22:33:32	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1a	2009-07-16T22:33:32	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1a	2009-07-16T22:33:32	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1b	2009-07-16T22:54:28	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1b	2009-07-16T22:54:28	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1c	2009-07-16T22:24:47	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1c	2009-07-16T22:24:47	running	ami-A
ec2-X.compute-1.amazonaws.com	us-east-1c	2009-07-16T22:24:47	running	ami-A

I have a couple of functions left (like actually launching instances, waiting for them and register with LB, hehe), but this is already useful to me as it is. It even saves your history and supports some TAB completion. This is my first DSL and it’s something I just hacked in a couple of hours. It’s not well thought out, but it was fun learning how to use python’s InteractiveConsole and as always it’s a pleasure to use boto.

NOTE: you must get boto trunk for this to work. Enjoy the ezcloud source.

Hadoop Timelines is a Web service built using App Engine and a Python script using Dumbo that will take care of everything to replicate Arun’s Task Timelines for your own Hadoop jobs. My goal with this project is to raise the awareness of Hadoop developers in understanding job execution and performance, maybe even crazier, that we collaborate and analyze together individual job performance through comments on specific graphs.

If you’re already comfortable with Hadoop, using Timelines should be really easy. The first thing you’ll need to do is follow Klaas’ tip for collecting job logs into HDFS using a simple cron job. If you don’t already have Dumbo and want to keep your dev environment clean, you can follow another excellent post from Klaas on using virtualenv with dumbo. Once you are up and running with Dumbo, download my dumbo/timelines.py job script that will process the joblogs.txt and submit them for public viewing to Hadoop Timelines.

WARNING: Please be aware that very basic information on your job tasks will be uploaded for public viewing. Anybody will be able to see the number of tasks, job duration, start and end time but nothing else. Please take a look at an example job if you want to be sure you’ll be comfortable uploading the same information for your jobs. The entire source for the project is available as well.

Now that I wrapped this little side project up, I’m going to start looking into my very own scary looking job graphs and possibly will be blogging whatever lessons I extract from them in the near future. Many thanks to Arun and Owen for uploading their code and data to compute the TeraSort graphs, Klaas for his amazing Dumbo and everyone else who makes writing these type of projects so much fun.

At Lookery, we have been working very hard to transform most of our data processing tasks into batch-oriented workflows in order to deal with growth. For example, we were already using Hadoop to compute our index and data files for our largest database, but the process of serving that information took place over too many network hops (load balancers, reverse proxies and Amazon S3). Therefore, as soon as I learned that Project Voldemort supported offline building of distributed stores, I decided to try it and we’re now running it in production.

If you’re interested in either sharing or attending please drop me a comment, @eliast, mail, whatever so I can gauge the interest level on this.

Fool me once, shame on me and you can ask Bush for the rest, but the first thing that came to mind was to write a cron job that saves a hash of my template files, checks it daily and mails me whenever it changes. And don’t worry, I was thinking of you too, WordPress-using reader. So I wrote the function as a WordPress plugin so others can benefit from it, until the spammers catch up with it. It’s very simple and to the point, but I made sure it was as hands-off as possible.

Instructions

• Make sure your WordPress installation and template are safe.
• Download the code to your WP plugins folder and rename the file extension to .php
• Activate the plugin.
• That’s pretty much it.

The first time it runs it will compute a signature based on your template files’ content. Then, it will check the content hasn’t changed once a day. In the case your theme get hacked and the contents of your files change, my plugin will discover that and mail you immediately. In fact, it will continue emailing you every day until you fix it. Additionally, it’ll show you a nice notice in your admin panel to warn you that someone changed your theme. If you want to reset it the warning, simply deactivate/activate the plugin to start fresh.

I must say that as much as I hate having to figure out how WordPress works on its entirety every time I write one of these, the end result is short, sweet and very powerful.

My next step would be to do the same for Project Voldemort since I already had discussed it with Jay Kreps, but I am waiting for the Protocol Buffers implementation of their network protocol to be stable enough. It should be a one-nighter instead of two. Anybody interested in hacking this one?

For example, Brett shows how to implement a scalable counter and (nearly) ordered comments using App Engine Megastore. For the former, shard the counter to get write scale and sum them on read.

Imagine millions of log lines containing ‘timestamp+TAB+userid’, the goal is to keep a count of how many calls in total and how many unique users we saw during that timeframe. Using Hadoop Aggregate Package all you have to do is emit records like these: “LongValueSum:Key+TAB+Count” or “UniqValueCount:Key+TAB+Value” and Hadoop will use a built-in combiner/reducer to return either “Key+TAB+TotalSum” or “Key+TAB+UniqueCount”. But in the case you run into Hadoop throwing out of memory exceptions when performing unique counts, all you have to do is “Shard It!” Simply put, hash your value and extend the key to include the partition. I use a ‘#’ separator to know when I sharded a key and by performing a second iteration on the job I can then LongValueSum all of the individual partitions to get my final count. Below is a simplified Python+Dumbo script to show the idea. In production, you just let Hadoop take over the reducer/combiner function.

Beware though that I was lazy to actually find the issue with Hadoop, since it shouldn’t need much memory to perform a unique count if the results are sorted, but at least I solved my problem and got to write a blog post about it.

#!/usr/bin/env python
import sys

def mapper1(key, value):
    key, value = value.split('t')
    yield "LongValueSum:TotalCount", 1
    yield "UniqValueCount:%d#Uniques" % (hash(value) % (2 << 14),), value

def mapper2(key, value):
    key, value = value.split('t')

    if key.find("#") > -1:
        h, key = key.split('#')
    
    yield "LongValueSum:%s" % (key,), value
        
def aggregate_combiner(key, values):
    agg, subkey = key.split(":")
    if agg.startswith('LongValueSum'):
        yield key, sum(values)
    elif agg.startswith('UniqValueCount'):
        uniq = set()
        map(uniq.add, values)
        for value in uniq:
            yield key, value

def aggregate_reducer(key, values):
    agg, subkey = key.split(":")
    if agg.startswith('LongValueSum'):
        yield key[13:], sum(values)
    elif agg.startswith('UniqValueCount'):
        uniq = set()
        map(uniq.add, values)
        yield key[15:], len(uniq)

if __name__ == "__main__":

    import dumbo
    job = dumbo.Job()
    job.additer(mapper1, aggregate_reducer, aggregate_combiner)
    job.additer(mapper2, aggregate_reducer, aggregate_combiner)
    job.run()

• If necessary setup a shell script to create ssh tunnels to all of your load balancers
• Create a local HTML file with iframe elements pointing to each of the tunnels
• Hack with CSS for 20 minutes until you have reached the layout you wanted (I used overflow:hidden, position:absolute and z-index to get the desired look)
• Install Fluid and create a new application pointing to your local html file
• Make sure you uncheck the ‘Fluid attempts to show badge lables’ preference in YourFluid.app so it doesn’t conflict with your own userscript
• Create a new userscript to update the badge label so it includes current activity if above a certain treshold
• Visit the Flick Fluid Icon Pool to pick a cool transparent icon for your dock

At first I tried using Dashboard and the Web Clip function in Safari to create a widget for each server so I wouldn’t have to lay them out myself, but the widgets kept resizing themselves and I wasn’t getting the dock badge labels. But that’s up to you and your skillz.