Lessons Learned in Creating the Pledge Drive Tracker: Part One

Recently, we put the newly revised, Pledge Drive Tracker into production use for the first time.

There have been many lessons learned in this process which I think may prove inspiring to other developers/entrepreneurs both within public broadcasting, and without. Here is a list of some of some of the major things I learned and applied throughout the process:

• Building a CMS is a red herring! Building discreet apps are more fun and more likely to succeeed
• Lowering the cost of failure (and increase the frequency at which you make your next attempt) with open source / software as a service
• Don't be afraid to scrap everything and start over
• Use a versioning system to build the app incrementally "Commit Early, Commit Often"
• Using a framework enlists a MVC (Model / View / Controller) approach lends to a higher standard of code. For code that is cleaner, more readable and more modular.
• Handling data migrations easily is important when prototyping an application. Being able to change the data model quickly and easily has been a real life saver on this project.
• Deploying with Google App Engine has its benefits (and its challenges)
• Practice Iterative Development: Involve the user early on both in the design and in fine tuning the app prior to launch

Building a CMS is a red herring

Building a CMS is a red herring (as I found from my work with the noble and innovative (all right, I'll admit it, failed) django-newsroom project). The sweet spot is in building specific apps that perform a highly specialized function. When you are building a CMS, you are going up against Wordpress; do you really think you are that good?

The trick is to build discreet apps that perform a tightly focused, more defined function which are, from my experience, more likely to succeed. They are less politically charged and make for better experiments.

Innovation happens at the edges anyway. Better to start some kind of cms that has an established developer community and an established user community and add value at the periphery, in ways that meet your specific needs.

Lowering the cost of failure (and increase the frequency at which you make your next attempt) with open source / software as a service

We've all heard it. Fail faster. Get failures out of the way sooner, then get on to discovering the successes and driving them home. Get buy-in early. Get people using your software quickly, with a much-reduced feature set. Start with what works for people in a limited and focused way, then build out from there.

What are some ways where you can increase productivity and lower risk, so that failure costs less and innovative ideas can be ramped up from prototype to production-ready quickly and reliably?

I have found that using web2py (a web application framework similar to Django) allows me flexibility to make changes (database migrations have been a snap) , allows me to rely on a core set of modules with a fairly well-documented API, and integrates nicely into Google App Engine for production.

Additionally, the web application suite I just described cost zero dollars to implement (thanks to the hard work of Google and the open source community). And when it did come down to paying for performance, we paid only for the bandwidth and computing power needed to run our app for a week ($8.50 total). At that price, you could test your ideas all year long and hardly make a dent in your web application budget.

Additionally, I believe the process and techniques used could provide a good basis for best practices in building and deploying web applications and could serve as a foundation upon which other software is built. The process used (or in a few aspects, aimed to be used) in creating this software could easily be employed in the creation of other web applications and so, for each application, a new process would not need to be learned. In this regard, this could help to increase the frequency at which you repeat well-known tasks so you can begin working on your next project.

How would I describe this process?

• Start with Python
• Create a virtual environment
• Use pip to install modules specific to your application in this virtual environment
• Use a concurrent versioning system (like git, svn or mercurial)
• Push your code up to github or Google Code (and make use of their wiki for RFPs as well as their system for feature/bug requests)
• Document your modules and document the API for your software using epydoc
• At the same time, create doctests (or if you prefer, unit tests)
• Test your software using a continuous integration tool, like Hudson
• Create detailed build scripts using a tool like fabric
• Run in-the-browser tests using tools like Selenium (or Windmill) - has the added benefit of updating indexes for GAE
• Test in Google App Engine's development server
• Publish the app on Google App Engine (cost: $0)
• Get input from users (rinse, repeat)
• Deploy on GAE (as your traffic increases, you pay only for the bandwidth, storage, computing costs that you use).

To this end, for those of you who are interested in putting these suggestions to practical use, I have started the ongoing task of documenting this processes here.

Don't be afraid to scrap everything and start over

The fact that these steps and techniques were productive, easily reproducable, and common across frameworks (such as django and web2py) made it not too difficult of a decision to start over when I felt I needed to go a different direction. I originally started out this project using django (and while it's a sterling framework in every respect) for me, there are almost too many moving parts, too many pluggable modules. It's quite possible my thinking might change in this, but for now, I found it quite refreshing to scrap everything and use web2py. Defining the model (an important starting point whether you're using rails, django or x) was almost identical between django and web2py. After that, there were many useful tools already bundled with the framework that made my work easy. For instance, jquery is integrated into the framework which easily allowed for sorting/searching tables and incorporating useful show'hide effects into the user interface. Additionally, web2py ships with the DAL (database abstraction layer) which automatically executes queries of your database from web2py across a number of different databases; not only does this save you from writing sql code (which could vary from database to database) but it also automatically handles data migration issues (so making slight changes to your data model, especially in the early stages of the work, does not break your application).

Likewise, I think the move from django to web2py could work the other way around. I think web2py could be a good tool to create a prototype and then, if one chose, they could port it over to django once it was further defined.

Discussions of the similarities and differences of frameworks aside, because the investment was minimal, and because certain processes were similar across frameworks, the ability to start over with a different approach allowed for some surprising discoveries and gains in productivity in the end.

I see now that I can write a lot about just a few of the lessons learned during this project. I hope to touch on the other lessons learned in future blog posts.

What are your thoughts? Have you found some of these experiences to be true for you? Are you interested in learning more about a particular aspect of the processes or technologies outlined here? Do you find any of these projects interesting and would like to collaborate?

Note: this initial blog post was initially created as a Google Wave here. Some suggestions that came out of this process were the following:

• John McMellen: I would be interested in reading the technical details, though it might be tangential to this post. I've been playing around with Django on GAE and found you have to jump through some hoops. Just wondered if you had a good tutorial on Web2py on GAE.
• Jack Brighton: I completely agree, and for example adding value at the periphery might most usefully take the form of developing addons/modules for an existing CMS. So if you want to add some specific functionality, it might be worth it to target Drupal, ExpressionEngine, and WordPress in particular although there might be others also. This might require a "loose team" approach for example involving players who already develop addons for those CMSs. What could happen if we had such a team, and used this approach to solve common pubmedia problems?