Skip to content
Jan 3 / PK Shiu

Django database migration tool: south, explained

Introduction

If you are using Django for production level application, you will need to use south. Requirement changes, and therefore your data model will change over time. South is a great tool, but it is complicated. You do not want to make a mistake when migrating an application in production. This is a detail look at how it interacts with your application so that you can understand and use it better.

Interaction

South interacts (reads and writes) with four different items in an application. This is the most confusing part to me. The four items are:

  1. models.py — South reads this to determine your current data model.
  2. migrations/*.py — South creates a sub directory inside your app, and creates for you a migration file for each database migration generation. You can also create these by hand, but normally you will let south creates them for you.
  3. south_migrationhistory table in your database — when you install south, it creates it’s own table and use it to maintain states. Specifically it records the state of the database in this table. South assumes the application schema in the database is consistent with what it records in this table.
  4. your application’s schema in your database — south create and update the schema for you according to the database migration generation, which is the ultimate purpose of using south.

Different commands in south interacts with these items differently:

Normal Migration

Let is start with a normal database schema migration, from generation N to generation N+1. There are three steps in the migration:

1. The developer (you) change the models.py file,
   updating the application's data model.
2. Run manage.py schemamigration app_name --auto to create
   a migration file for generation N+1.
3. Run manage.py migrate app_name to update the database
   schema and migrationhistory table to generation N+1.

This diagram shows what are the inputs and outputs to each step. (Note, in step 2 it reads all of the migration files from all previous generations):

Initial Migration

Let’s add the very first step when using south on a new applicatoin. The initial migration obviously does not have any previous south information. So there are still three steps, but the argument passed to schemamigrate is a little different:

1. The developer (you) creates the first models.py file,
   defining the  application's data model.
2. Run manage.py schemamigration app_name --initial to
   create a migration  file for generation 1.
3. Run manage.py migrate app_name to create the database
   schema and  migrationhistory table to generation 1.

This diagram shows what are the inputs and outputs to each step, adding to the previous diagram:

Converting an Application

Converting an application is a little different because the database schema and the models are already in sync. Somehow we need to “trick” south into creating the other two items, the migration file and the migrationhistory table entry. We also need to deal with the conversion on the first instance vs other instances differently. For the first instance you will create the migration file, and for all other instances you will use the migration file to simply create the migration history table entry.

On the first app instance:

1. Run manage.py convert_to_south app_name to create the migration
   file for generation 1, and also to create the migrationhistory entry.

On other app instances:

1. Run manage.py migrate app_name 0001 --fake to create
   the migrationhistory without changing the database schema.

Diagram:

Testing and Trying Migration before

There are several different things you can do to “dry run” your migrations.

Schema Migration Generation Test Run

./manage.py schemamigration my_app_name --auto --stdout

The -stdout argument will have South generate and print out the migration code on the console instead of writing it out to the migrations directory. You can do this before you do the actual database migration schema generation. Note that schemamigration only creates the next migration script in the migrations directory. Worst case is you can just delete that file if you decided the migration schema is not needed.

Migration Dry Run

The migration operation can also be tested before actually running to migrate a database:

 ./manage.py migrate myapp --db-dry-run

This will run the migration except the database is not changed, and the migration is not recorded in the history table in the database.

Useful Commands

Another useful command is the list command. It shows all the migrations defined in the migration files, and whether they have been applied (by reading the migrationhistory table entries):

manage.py migrate --list

Deleting Migrations

Sometime I need to reset a migrated environment. This is particularly useful during initial development of an app. This special command, migrating to the “zero” state, will remove all migration history in the database, but leaving any migration files intact.

manage.py migrate my_app_name zero

Bonus

This is my initial sketch for my explanation diagrams:

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Twitter
  • Google Bookmarks
  • Add to favorites
  • email
  • FriendFeed
  • Google Buzz
  • LinkedIn
  • Tumblr
  • Reddit

25 Comments

Leave a comment
  1. Oli Warner / Jan 4 2011

    Might want to go over this and wrap inline code sections in backticks to stop WP replacing double – with –

  2. PK Shiu / Jan 4 2011

    Thanks Oli ! Didn’t even notice WP was doing that. Anyone copy and pasting the commands will be pulling their hair out. Thanks for the tip.

  3. Jeff Bauer / Jan 5 2011

    OT: What tool are you using to generate the diagrams?

    • PK Shiu / Jan 5 2011

      Hi Jeff,
      I use omnigraffle to draw those diagrams. I have a love/hate relationship with Omnigraffle. While it is *the* diagramming tool for Macs and it is very nice, it still has quite a learning curve. I am slowly getting there.

  4. Nicolas / Jan 5 2011

    For pasting code with WP I’d like to recommend to install the wp-syntax plugin and to disable the so-called visual editor in your user prefs. Thank you for the great article!

    • PK Shiu / Jan 5 2011

      Hi Nicolas,
      I use “syntax highlighter evolved” normally. Since those are just commands I didn’t want them to look like “code”. But maybe I’ll give it a try. Thanks!

  5. Daniel / Jun 10 2011

    Most of the database migration tools I’m aware of are integrated into existing Web frameworks, such as django, web2py. What are the other available and efficient libraries when you are not using such Web framework, but, for instance, Tornado. I rapidly took a look at SQLAlchemy migrate tool. Any good or bad experience with it?

    • Kerr / Oct 27 2011

      I haven’t used SQLAlchemy, but if you’re looking for a standalone schema migration product, have a look at Liquibase. It’s written in Java (fair warning for those Javaphobes), and integrates well with deployment tools like Ant and Maven.

      http://liquibase.org

    • Skylar / Aug 23 2013

      I like mariposa / django-mariposa (I wrote the django bit and contributed to mariposa). Very simple.

  6. PK Shiu / Jun 10 2011

    Hi Daniel,

    Sorry but the only migration tool that I actively use is south with Django.

  7. becker / Jul 2 2011

    whats about production and development database? Do I have manage both by hand? Do south understand which migrations are missing on both sides?

  8. PK Shiu / Jul 2 2011

    @becker — south does not need to know production vs development. If you are using south in your project, everyone/every instance need to be using it too. For example, we run personal (development) instances with all different variations, a mixture of sqlite3, OS X, postgresql and ubuntu, but we then we deploy into several layers of acceptance testing environments.

    South keeps track of what steps each database (per app actually) is at and will migrate from the correct point onwards. django-admin.py migrate –list will tell you where the current database is at and how many more migration steps are needed, if any.

  9. Abd Allah Diab / Oct 22 2011

    Thanks for this simple tutorial,
    I think that in the initial migration the command should be `manage.py schemamigration app_name –initial` instead of `manage.py schemamigrate app_name –initial`

    ;)

  10. vibelab / Jun 14 2012

    Hey, great post. Should it really be this easy to use south / migration? I usually peek at it when I start another django app, but then discrad the idea of using it for beeing “too difficult to work into”.

    i am definetely trying this next time around.

  11. Fady / Aug 6 2012

    Nice article, well explained and easy to understand, Thanks !!

  12. Daniel Sokolowski / Oct 1 2012

    Seriously great post, I always did my migrations manually but now will revisit using South on next project. Have you had any experience with django-evolution?

    Daniel – I have followed you on Twitter.

    • PK Shiu / Oct 1 2012

      Thanks Daniel,

      I have not tried django-evolution.

      P.K.

  13. Manjunath / Mar 11 2013

    Awosome tutorial.really saved my time

  14. Ofer / Mar 11 2013

    Great post!
    How would you sync a newly created Django app with a legacy database?
    And a little different, how do you handle database schema changes that originate in the database?

    Thanks in advance!

  15. Pavan Verma / Sep 7 2013

    Great article! South is one of the most useful 3rd party app available for Django. However, it’s tutorials and documentation are not the best IMO. The official tutorials do not cover the simple use cases very well. I have always been ended up being confused when going through their docs and then trying things out. South can take help from a simpler getting started guide and this article does just that.

Trackbacks and Pingbacks

  1. Tweets that mention Django database migration tool: south, explained | DjangoPro -- Topsy.com
  2. Conocimiento Abierto » Blog Archive » Tutorial básico de django south
  3. Adding new columns in Django models - Django Solutions - Developers Q & A
  4. thetawelle » Blog Archiv » django – some first impressions - frontier of science, technology & future
  5. Migrations | Diary of a Web Noob
Leave a comment