Explaining Technical Debt

I’ve been thinking about technical debt and how it builds in software projects. Brace yourself, this is a depressing topic so I’ve included a picture of a turtle. Hopefully that makes you feel better.


Good? Good.

It’s very easy to come up with a specific example of technical debt and then say, if we fixed that problem the project would be much easier to modify and therefore cheaper to maintain. The real trouble with technical debt is explaining how important it is to someone without a technical background. As a rule, it’s very easy to justify the next great feature in terms of sales. It’s a measurable win and makes everyone very happy very fast.

Technical debt, true to it’s name, is another beast entirely. It builds up slowly over time. Everyone knows it’s there and can see the disease spreading but it isn’t until it’s too late that you can convince people to take action. It’s too late when you need to rewrite your software to add the next essential feature. For the economically literate, this is when your marginal cost exceeds your marginal revenue.

The end result is a long and gruelling recovery in the best case and shutting down the business in the worst. There’s nothing much that can be done about the temptation to trade long term cost savings for short term revenue. But with a useful analogy we may be able to shed some light on the dragon lurking in the darkness of the treasure cave.

Technical debt is very much like debt, but with a different currency. In the world of software we trade time for features. In an ideal world, the cost of a feature is static. It doesn’t matter if you implement the feature this cycle or five years from now, it costs the same amount of time. This is probably how management thinks software development works. They’ll probably also get confused when that’s no longer the case.

Now let’s add a bit of complexity. Your boss asks for a feature that is kinda pricey for this cycle. Normally you’d tell them no, but instead you decide to borrow some time from the next release by cutting corners. You’re hailed as a technical hero for meeting the deadline. You think to yourself, things will be okay if you spend a bit of time next release to pay back the time you borrowed.

However, now you’ve set a precedent and the credit genie is out of the bottle. The next release comes along and again you’re strapped for time and so you borrow more time. This time it’s only moderately worse so again you figure you’ll spend an all nighter or two to get it done right in the next release.

Unfortunately, debt compounds over time. When the debt is low the problem is small but when the debt is high, no amount of all nighters during a release can help you. What you need to do is spend some serious time paying down the debt by fixing the small compromises you’ve made over past releases.

Up to this point, you’ve hit all of your deadlines. How do you tell management that we need to spend a whole cycle rewriting code that already works? The reality is that it’s already getting too late because if you try to add features this release, it wil get worse. Instead let’s look at what we can do before things spin out of control.

Do you necessarily have to borrow? If you put together a killer team and invest heavily on testing you’ve solved the problem right? Well, not really. Any sufficiently complex project has unexplored problems and your project team needs to learn the best ways to solve these problems. That often means borrowing refactoring time from the future to try something new today.

But having a killer team helps, right? It does, but not in the way you might think. It doesn’t reduce your overall debt as that is really part of your source code. Having a good team is like having a credit card with a cheap interest rate. You can borrow less time from future releases with clever solutions today, but you’re still borrowing. If you’ve already accumulated debt in your source code, you still need to pay that debt no matter which team you have.

Sometimes you get lucky and a good developer will find a clever solution to a core problem that clears a bunch of debt. Sometimes you get unlucky and you’ll find that you’ve accidentally built your software on a mud bank that is washing away.

Anomalies aside, the place to be is where you can service your debt. That is, you are clearing enough of it to keep the amount of time between releases to an acceptable level. If the time between releases starts to go up quickly over every release, you’ve got a problem. It’s time to stop and refactor.

Good teams refactor constantly and are always assessing whether their designs fit their requirements. The rule of thumb I’ve seen thrown around is about one developer on one refactoring task per cycle per 5-10 developers. This number will depend very much on your industry and quality requirements.

So let’s summarise the analogy. Building software is like spending time for features. Sometimes you have to borrow time from a future release to buy a feature. If you do, your team determines the interest rate you have to pay and your source code is your balance sheet. If you don’t pay back your debt, you will no longer be able to modify your software and bad things will happen.

The scariest part is that most management teams are often spending credit as though there were no interest rates. The only people that can warn them are the engineers, so it’s time to start complaining. Your weekends depend on it.

Posted in Uncategorized | Leave a comment

sqlpyzer: IPython + sqlite in html5

What is a sqlpyzer?

sqlpyzer is an experimental plugin I wrote for the IPython  Notebook. It’s much easier to describe what it is in a video, so here goes:

You can find the source and the installation instructions here:


The Name

Sadly all of the good SQL names were taken. I was going for sqlizer but things fell apart quickly when I tried to add py.


There are installation instructions on the github page, but I’ll go into a bit more detail here. I haven’t tested these steps, so please comment with any corrections. I’m currently running Ubuntu 12.04 32bit desktop. I’ve tested the plugin using Python 2.7 and Chromium. I’m going to assume that you have your github repository located in ~/Repo. You’ll need a specific build of IPython; one that supports jsplugins. You can find such a build at:


Be sure to check out the “json” branch. Once you have ipython checked out you can then run the notebook by using the ipython.py script in the root directory. This script will then launch the notebook and open a browser.

user@computer:~/Repo/ipython$ python ipython.py notebook

Next, we’ll install sqlpyzer (again, sorry for the name). Close the notebook for now, we’ll get back to it later. I recommend you use pip and virtualenv. If you’re not familiar with these tools, check this out:


Install sqlpyzer:

user@computer:~/Repo/sqlpyzer$ pip install -e .

By using a develop egg above, you’ll be able to modify sqlpyzer without having to reinstall it. Next, we’ll have to install the sqlpyzer javascript. You’ll need to find the ipython local configuration folder. In Ubuntu:

user@computer:~$ cd .config/ipython/profile_default/static/jsplugins/

Simlink the Javascript files so that you can modify the Javascript in your repository:

user@computer:~/.config/ipython/profile_default/static/jsplugins% ln -s /home/user/Repo/sqlpyzer/js/* .

Finally, run the notebook again and open the notebook located here to test the plugin:


Repo Structure

There are three things of interest in the sqlpyzer repo. The sqlpyzer folder contains all of the python responsible for getting the data out of python and marshalled into json. The js folder contains all of the javascript that creates the sqlite database and marshals the json into the database.



The load.py module contains one function. It is called by the load_ext function in IPython. Its job is to tell IPython how to take a data object and display it. Everything comes together on this line:

formatter.for_type_by_name(module, name, serializer)

The serializer function is called every time you issue the display command on a sqlpyzer.Data object.


This module contains a simple validator and data type. Besides validation, the only real purpose this data type servers is to be able to associate Data objects with a specific serializer function that tells IPython which Javascript handler to use.


The function in this module turns the Data object into a json string. The resulting object has two properties. First is the data object. Note that the “data” key can really be called anything. Second is the handler. The handler is special as it tells IPython which Javascript function to use to interpret the json being passed in.


In order for IPython to be able to load a module as an extension, the module needs to have a load_ipython_extension function. The __all__ list tells python that it should load the load_ipython_extension when the module is loaded, as an attribute of the module. If you’re not familiar with __all__, take a look at:


Everything Else

constants.py has some constants like the Javascript callback name. test.py contains tests. To run the tests use:

user@computer:~/Repo/sqlpyzer/sqlpyzer$ python test.py


Disclaimer: I know more about Python than I do Javascript.


Ths script takes the id of a div as an argument, and creates the database object with the UI. The UI should automatically resize its controls to fit in whatever container you put it in. The display function is the interesting part in this module. It demonstrates how to retrieve data from a sqlite query. Once the console is created, it returns an object containing the database, input and output fields.


Possibly the largest single piece of source in this project. If you’re looking for the Javascript that associates a the handler name with the Javascript callback, look all the way at the end of the file:

IPython.json_handlers.register_handler('sqlpyzer_data', sqlpyzer.data);

The sqlpyzer.data callback takes two parameters; the json object that we serialised in from python and the new element created under the display function. It then creates a console and uses the json data to populate the database.

When reading the create_table code, it’s important to remember that Javascript doesn’t block when executing SQL. Instead, you need to specify callbacks that are called if a query succeeds or fails. You can get away with calling them in parallel in a single transaction, if your queries are independent as with this for loop. In the background, javascript is queuing the SQL execution to happen as a set of callbacks, once the for loop is complete.

// Create all the tables.
console.database.transaction(function (tx) {
     for (table in json.data) {
         sqlpyzer.create_table(tx, table, json.data[table]);

If your queries depend on other queries like row insertion depending on table creation, things start getting interesting. You need to specify a callback that will insert all of the rows once the table has been created. You’ll notice that the create_table function has two functions called create_schema and insert_row for this reason. create_schema exposes a callback called next that then inserts all of the rows in parallel:

// Pull it all together to create the table.
create_schema(tx, function(tx) {
    for (var index = 0; index < table.length; index++) {
        var row = table[index];
        insert_row(tx, row);

Closures make this sort of work a easy. They basically set the context for the callbacks used to create the schema and the individual rows. Otherwise, we would have to pass all of the table level parameters into both callbacks, which would be messy. If we were doing this sort of work sequentially with a language like python, your context containers would be your for loops:

# Table level context.
for name, table in tables
    # Create the schema.
    schema = get_schema(table)

    # Row level context.
    for row in table:
        # Create the row.
        insert_row(schema, row)

Once you insert a row, how do you get the id of the new row? I’m glad you asked! My initial thought was to use the last_insert_rowid function:

select last_insert_rowid() as id;

However in Javascript you cannot control the order of your callbacks and so you may end up with the wrong ID. Fortunately, the result set has a handy property called insertId. You can see it at work here:

// Generate the insert query.
var query = 'INSERT INTO '+name+'('+columns.join(', ')+') ';
query += 'VALUES ('+params.join(', ')+')';
tx.executeSql(query, values, function (tx, data){
    // The inserted row id.
    var new_id = data.insertId;


Unfortunately IPython has a nasty tendency to eat exceptions that happen in both the serialise.data function in Python and in the sqlpyzer.data function in Javascript. In Python be sure to completely test your serializers. Unfortunately, I wrote no unit tests in Javascript. Instead, I found modifying this file:


to this effect helped a bit:

JSONHandlers.prototype.call_handler = function(key, json, element) {
    var handler = this.handlers[key]
    if (handler !== undefined) {
        try {
            handler(json, element);
        } catch(err) {
        handler(json, element);


That’s about it! You know about as much as I do about making IPython talk with sqlite via html5. I’m not entirely sure what that means, but I had fun. If you end up doing anything interesting with this experiment please let me know! Again, sorry about the name…

Posted in Uncategorized | Leave a comment