Niall's Weblog

MasterSlaveConnection: Works in Pymongo 1.6
In my previous article Python, MongoDB and Pylons – Connection handles and all that lark I described the code and configuration we use at Snaptic for working with Pylons and MongoDB. We use the MasterSlaveConnection class to automatically handle read/write splitting, because we expect to be running a single master/multiple slaves configuration in production.

Broken in Pymongo 1.7
Well, yesterday I upgraded from Pymongo 1.6 to Pymongo 1.7 (just released) and found that unfortunately all the MongoDB stuff breaks, throwing TypeError exceptions when issuing even a simple find_one() query. You will see an error like this one:

TypeError: 'Database' object is not callable. If you meant to call the 'document_class' method on a 'Collection' object it is failing because no such method exists.


But fixed in tip
After talking with Mike Dirolf (Pymongo author) on the MongoDB IRC channel, he came up with a fix within a few minutes. Although I don’t think its in a release yet, if you grab this commit to Pymongo from github, MasterSlaveConnection should work again. He also filed this bug to come up with a more general solution. Cheers to Mike for his super quick turnaround on fixing the bug!

I’ve been doing a bunch of hacking with Pylons and MongoDB recently for some backend stuff at Snaptic. Right now we are using Paster as the webserver and the Pymongo driver. This all works fine and is pretty straightforward to set up – but there are a couple of subtleties.

MongoDB vs SQLAlchemy

If you’ve ever used Pylons with SQLAlchemy, then you’ve probably noticed the integration is quite good. All of the glue to get access via the global Session object is done for you. With MongoDB and Pylons, the integration isn’t quite there yet. You have to set this up yourself.

Gotta have some replication

MongoDB doesn’t give you the same kind of single-server durability guarantees that a RDBMS like MySQL or PostgreSQL does, so you pretty much have to use some kind of replication in production. I’m expecting a master/slave configuration, and Pymongo has native support for read/write splitting (writes go to the master, reads go to the slave(s)), so I’ve used that in my Pylons integration code.

Getting a handle to the database

The basic idea is that you define a list of masters and slaves in your Pylons config file, and some code in lib/app_globals.py sets up a global handle to the connection object.

Here is what I have at the moment, in the __init__ method in lib/app_globals.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# ... non-pymongo imports omitted
from pymongo import Connection
from pymongo.master_slave_connection import MasterSlaveConnection
from pymongo.errors import ConnectionFailure
 
# ... class definition, __init__ method etc omitted for brevity ...
 
# assumption is a master/slave setup for initial production deployment.
# writes should go to the master
# reads should go to the slave(s)
# with sharding, this may change.
def make_conn(host, port, database, username=None, password=None,
        slave_okay=False):
    try:
        conn = Connection(host, port, slave_okay=slave_okay)
    except ConnectionFailure:
        raise Exception('Unable to connect to MongoDB')
    if username and password:
        auth = conn.authenticate(username, password)
        if not auth:
            raise Exception('Authentication to MongoDB failed')
    return conn
 
username = None
password = None
if 'mongo.username' and 'mongo.password' in config:
    username = config['mongo.username']
    password = config['mongo.password']
master_conn = make_conn(config['mongo.master_host'],
        int(config['mongo.master_port']), username, password)
slaves = []
for i, slave_host in enumerate(config['mongo.slave_hosts'].split(',')):
    slave_port = int(config['mongo.slave_ports'].split(',')[i])
    slave_conn = make_conn(slave_host.strip(), slave_port,
            username, password, slave_okay=True)
    slaves.append(slave_conn)
 
self.db_conn = MasterSlaveConnection(master_conn, slaves)
self.db = self.db_conn[config['mongo.db']]

Then in development.ini I have:

1
2
3
4
5
6
7
8

# in production, we will write to the master and read from the slaves
mongo.master_host = localhost
mongo.master_port = 27017
# to supply multiple slaves, use comma as the separator.
mongo.slave_hosts = localhost
mongo.slave_ports = 27017
 
mongo.db = foo

Now, to get a handle to the db object from a controller context, you just do the following:

1
2
3

db = self._py_object.app_globals.db
# now I can go wild and run:
# db.fooCollection.find_one()

If you need a handle to the connection object instead, no problem:

1
2
3

db_conn = self._py_object.app_globals.db_conn
# now I can run:
# db_conn.end_request() if I want.

There might be more elegant ways to do this, but this seems to work fine for us. I will write about connection pooling and end_request() next time. Feel free to comment if I left anything out or you have questions!

Over at Snaptic we just announced our Move Your App! developer challenge.

The Challenge

The idea is simple: write an Android app which inspires people to move around.

3 Reasons to Enter

1. Actually benefit society

Obesity is a huge problem – 64% of adults and 25% of children and teenagers are overweight or obese today in the USA. While there are many contributing factors, the most obvious is simply that we do not move around enough.

Obesity has been cited as a contributing factor to approximately 100,000–400,000 deaths per year in the USA alone.

This is an awesome opportunity to get some attention in this area and contribute something genuinely positive to society.

2. Earn some very good publicity and kudos

We have gotten some very high profile judges and sponsors for the challenge. Mårten Mickos, former MySQL CEO, Amy Novogratz, TED prize director and Pam Omidyar of HopeLab to name just a few. These are awesome folks to have reviewing your app.

3. Win an all-expenses-paid trip to TED Global 2010 and more

The top prize is a trip to the exclusive TED Global 2010 conference. I’m sure you’ve watched some TED Talks in the past and know what a fantastic, high-level event TED is. We are also going to be giving away a Mac Book Pro, Android phones, and more.

Get Help with your app

What kind of application can you write to encourage people to move? Imagine a game where you earn points based on activity – say through location checkins or something like that. Or how about a simple “distance traveled” application where you can log and graph your movement each day. There are tons of things you could do.

We are also happy to help any developers working on the challenge – check out our developers page for full details, but you can hop on #snaptic on irc.freenode.org or reach us on Twitter at @snapticdev!

And hey if you need help setting up the Android SDK on your machine, maybe my own Android SDK on Ubuntu post could be of assistance.

Register Now!

What are you waiting for If you are at all interested, go ahead and register!

Man this log is way overdue. I’ve been super busy since leaving my job at Metaweb, taking a short break, and starting up at Snaptic. While I was on my break between jobs, I got a few days of awesome, tough hiking in at Henry Coe State Park and some solid training too.

I’m about a month and a half behind in logging all my workouts so I’m not going to bother doing that for this final post before the NorCal Sectionals. If you are interested in all the gory details, they’re all posted on the CrossFit KMSF blog in any case. Instead I’ll give a few highlights and talk about where I’m at after three months of very intense training, and how I think the competition will go.

Weight & diet

I weighed myself first thing this morning and I’m just over 171lbs. Thats a gain of about 6lbs since my first training entry 3 months ago – and if anything I’m a fair bit leaner now. So I’ve definitely gained some functional mass. This despite eating what feels like a huge amount of food – after starting with Paleo Zone I switched to basically un-measured Paleo. I’m proud to say I cheated very little on the dietary front and am feeling great!

Strength & power

I’m definitely feeling a lot stronger, especially in the back squat and Olympic lifts. I mentioned in my last post that I’d done some Oly-lifting coaching down at Catalyst Athletics. That session was focused on the Snatch, and I subsequently went down and did a second session focusing on the Clean and Jerk. I’m still a complete novice weightlifter, but relative to where I was 3 months ago, I’ve improved a huge amount in form. I got a 135# snatch (squat, not power) the other day, a very modest weight, but I don’t think I could’ve even managed a proper snatch three months ago. Similarly my clean has gotten much better – 185# used to be maxing out my horrible power clean but now a 185# clean with a reasonable squat technique is fairly comfortable.

Most importantly of all I’m really enjoying practicing the Olympic lifts now and am looking forward to continuing to improve at them after the Sectionals!

Conditioning

We’ve been working a lot of short sprints, and stuff like kettlebell swings – and I feel pretty good here. Pretty fast, with decent endurance and recovery. On the day of the competition, a lot of it is going to be mental, to just keep pushing!

Workout A

In Workout A you have 6 minutes to run 800m and then get as many reps of over head squats @ 115# as possible. Score is based on reps.

We’ve run through this a few times and its a very interesting workout. Over head squats are not a particular weakness of mine, neither is an 800m sprint – so it should be fine. I feel that this will come down to concentration and sheer will power. How many reps can you knock out before you have to drop the bar? I have a number in my head and I’m going to do my damndest. That bar gets HEAVY after 10-15 reps, and you really have to fight to hold on! This will be a fun challenge.

I feel it will be key to stay calm and not to overdo it on the 800m too.

Workout B

In Workout B you have 10 minutes to do as many rounds as possible of: 7 thrusters @ 115#, 12 KB swings @ 1.5 pood, 7 chest-to-bar pull-ups. Score is number of rounds completed.

Thrusters are not a strength of mine, not least because of my shoulder flexibility, but I actually feel good about them at this weight. 115# is not too bad, and 7 reps should be fine. Kettlebell swings are rest. The chest-to-bar pull-ups are tough. I wish I had known about the standard earlier so I could have locked it in more, but oh well. Past few weeks I’ve been focusing a lot on keeping the technique solid. 7 reps should be ok for the first few rounds, hopefully my form won’t break down too badly towards the end. They will definitely be the toughest component.

A great team

It has been awesome training with all the folks at Crossfit KMSF. Thanks to coaches Kat, Lake and Cip for their dedication. Of course I’ve been lucky to have such a cool group of friends at the gym, a big shout out in particular to David, Robert, Tadhg, Zac, John and Matt who are also competing. We are going to kick some ass!

After the Sectionals

I’m really looking forward to the contest. Its been a great motivation for my training and I’ve improved a lot. I am positive about how I’m going to perform and I plan to do my best and have fun too.

A week after the sectionals, I’ll actually be going to China for a month to visit my girlfriend Frances Wang who is doing field work with the UN in Beijing. I won’t be training super hard while I’m there but I do plan to stay active and hit the gym at least a few times a week. But mostly I plan to enjoy my awesome trip and eat great Chinese food!

This update is a bit overdue, I’ve been pretty busy finishing up things before I left my job and planning a backpacking trip before I start my new one. Still been training though!

Friday 2010-02-05
Rest day.

Saturday 2010-02-06
WOD: 5 rounds of: 10 burpees, 5 deadlift @ 275. My time: 3:55. Deads start feeling pretty heavy!

Also went down to Catalyst Athletics in Sunnyvale for some Olympic lifting coaching. There were four of us in the group and we had an hour long session with Steve Pan. It was very useful coaching. We worked on snatch technique. The two big pointers I took away from it were a) holding that solid starting position in the initial stages of lift (e.g. a snatch-deadlift) and b) moving feet lightly, avoiding any “donkey kick”. I will be going down for coaching on the clean & jerk soon.

Sunday 2010-02-07
Rest day. I really needed a break.

Monday 2010-02-08
Did some shoulder-strength/rehab type work. A bunch of Turkish get-ups at 1.5 pood, presses and behind-the-neck presses. The behind-the-neck presses seem to have a very therapeutic effect. Strangely my hand and wrist pain has decreased greatly – in addition to lessened shoulder pain – since working them. I do pretty light weight (65#).

Tuesday 2010-02-09
WOD: 3 rounds of: 3 Muscle Up, 5 SDHP @ 95#, alley sprint (100m uphill). my time: 6:09.
Went into this one thinking I’d get stuck on the muscle ups, but in fact I felt much stronger doing them and had no problems. Felt a lot easier to get through the transition point.

Wednesday 2010-02-10
WOD 5 rounds of: 10 box jumps, 7 deadlifts @ 185#, 5 hand-stand push ups (on parallettes). My time: 6:49.
HSPU were definitely the slowest part of this for me – its tough to do them for reps. The deadlifts did start to get tough toward the end. Box jumps were easy.

Thursday 2010-02-11
Rest day.

Friday 2010-02-12
More shoulder work – Turkish getups, behind-the-neck presses, etc.

Saturday 2010-02-13
Run 5k. My time: 26:30 (urban route – lots of stopping at lights etc).

-Rest-

3-3-3-3-3
Overhead Squat

I did pretty well on the OHS: 125-135-140(f)-145-155. 155# for 3 reps is a PR!

Sunday 2010-02-14
Turkish get-ups, behind-the-neck presses, etc.

Monday 2010-02-15
WOD:

8x100m sprints, rest in between.
2x400m run, 2 mins rest in between.
1x800m run.

The alley sprints were hard. Very difficult to go all-out eight times like that.

- Rest -

Shoulder press 3-3-3-3-3. I did: 115-125-135-140-145(f). I think 140# for 3 reps is not too bad.

Tuesday 2010-02-15
WOD:10 Rounds: 10 Kettlebell Swings 1.5 pood, 10 Pull-ups. My time: 9:11. After about round 5, grip starts weakening. Final few KB swings took everything I had just to hold onto the damn thing!

rest

3-3-3
Snatch Balance. Just went light with this, working speed. 65-75-95.

3-3-3
Bent Over Row. Still getting used to this exercise. 95-135-155.