This weekend, I had a chance to play with two platforms that have been on my “to-play-with” list: Twilio and Fwix.
The result is a simple app I’ve made called PhoneFwix. The workflow is simple. Call (415) 483-1286. Type in your zipcode. Listen to news. The Python code is only a couple hundred lines, including all of the geocoding, fetching, processing, and little utility methods. Pretty good, Fwix and Twilio. Pretty, pretty good.
The setup on the Twilio website was very straightforward, thanks to some awesome documentation. I did have to pay a minimum of $20 to open a full account, but you can barely buy lunch these days for $20, so that’s certainly a fair deal.There are a couple of improvements I’d like to see from Twilio, such as an interactive debugging mode where I can call from a developer phone number and get some on-the-fly introspection. And, of course, Google Voice integration. (Actually, I’ll just go out and predict that Twilio will be purchased by Google by the end of 2010. You heard it here first.) On the whole, I’m very pleased with Twilio and have already thought of a few more exciting things I could do with it.
A lot of the Twilio apps I’ve seen so far involve lead generation and other sales-oriented type things. Which makes sense, considering the huge savings you’d get from Twilio compared to the expensive enterprise phone systems that charge upwards of $5 per call received (whereas Twilio charges a few cents for an average call).
Fwix’s API only has a few methods, and has a ‘geo_id’ schema that’s slightly more awkward than it might be, requiring at least two calls to get anything done. And it formats integers as strings and sometimes sends strange fragments as titles or summaries. And while these can easily be automatically flagged and discarded, they could also be flagged and discarded on Fwix’s end.
But enough with the complaining, already! Fwix does a good job in the most important area: the content. All of the geographic areas are well-seeded with (mostly) relevant information, and this is quite a feat for a Chop Suey application.
The hardest part about making this application was taking a zipcode and figuring out which Fwix location was closest to it. Luckily, geopy made this task a piece of cake:
@memoize()
def get_coordinates_for_zipcode(zipcode):
from geopy import geocoders
g = geocoders.Google(GOOGLE_API_KEY)
place, (lat, lng) = g.geocode(zipcode)
return place, (lat, lng)
@memoize()
def get_closest_geo_id(lat, lng):
from geopy import distance as geopy_distance
from models import GeoID
geo_ids = GeoID.all().fetch(1000)
distances = []
for geo_id in geo_ids:
distances.append(
{'distance': int(geopy_distance.distance(
geo_id.coords(), (lat, lng)).miles),
'geo_id': geo_id.geo_id,
'place': geo_id.key().name()
})
distances = sort_by_key(distances, 'distance', reverse=False)
return distances[0]['geo_id']
get_coordinates_for_zipcode uses the Google Maps API to (drum roll) get coordinates for the given zipcode.
get_closest_geo_id is the one that really saved me from having to brush up on my Dijkstra algorithm skills. It takes two (lon, lat) tuples and automagically finds the distance. The resulting list is sorted by the ‘distance’ dictionary key, so that the first sorted geo_id is the least distance from the given zipcode.
One annoying thing about geopy is that it’s full of print statements that I had to comment out. Honestly, who does that? You can’t just print things wily nily, geopy. Then there would just be anarchy. That’s why we have logging: to avoid anarchy…and print statements.
On a slightly unrelated note, I’ve also made a simple module for compressing GAE datastire entities into their binary representations, and decompressing them. It’s inspired by Nick Johnson’s recent post about the db.entity_pb module. (By the way, if you do any development on GAE and you don’t read Nick Johnson’s blog, you, sir or ma’am, are a crazy person.)
The problem with Nick’s sample code is that, well, it’s sample code. In real life, I’m memcaching pretty much everything I can, and in lots of cases, it’s just arbitrary data.
So I made two helper methods to conditionally compress entities into their binary equivalent. If the data is a list, it figures out if the list contains entities by looking at the first few entries for a db.Model object. If so, it individually checks each item and converts it if it’s an entity.
def to_binary(data):
""" compresses entities or lists of entities for caching.
Args:
data - arbitrary data input, on its way to memcache
"""
if isinstance(data, db.Model):
# Just one instance
return makeProtoBufObj(data)
# if none of the first 5 items are models, don't look for entities
elif isinstance(data,list) and find_first(
lambda i:isinstance(i, db.Model), data[:5]):
# list of entities
entities = []
for obj in data:
# if item is entity, convert it.
if isinstance(obj, db.Model):
protobuf_obj = makeProtoBufObj(obj)
entities.append(protobuf_obj)
else:
entities.append( obj )
buffered_list = ProtoBufList(entities)
return buffered_list
else: # return data as is
return data
I use custom classes ProtoBufObj and ProtoBufList so that it’s very easy to identify the types of data that need to be decompressed:
class ProtoBufObj():
""" special type used to identify protobuf objects """
def __init__(self, val, model_class):
self.val = val
self.model_class = model_class
# model class makes it unnecessary to import model classes
class ProtoBufList():
""" special type used to identify list containing protobuf objects """
def __init__(self, vals):
self.vals = vals
def makeProtoBufObj(obj):
val = db.model_to_protobuf(obj).Encode()
model_class = db.class_for_kind(obj.kind())
return ProtoBufObj(val, model_class)
I initially didn’t have the model_class attribute of ProtoBufObj objects, but then I started getting KindError exceptions when an entity was being decompressed but the model definition import for that given entity was behind the memoize() decorator, where it wasn’t being executed.
It’s easy to rectify this issue by making sure that model imports are being called from outside of the memoized methods, and then you should be fine to remove the model_class attribute.
** Download the CacheCompress module here. **
This utility makes for a great example of the well known time/space tradeoff, since it takes a little more time to compress and decompress the entities, but saves a good amount of space in the memcache. It actually might negate the additional time required since the amount of data that needs to be retrieved from the memcache is considerably less than it would otherwise be, so it likely takes less time to complete the memcache calls. I’ll probably have to do a test with some huge amount of entities to get a definitive answer to see how it affects performance.