{"id":5907,"date":"2018-04-27T17:14:53","date_gmt":"2018-04-27T17:14:53","guid":{"rendered":"https:\/\/blog-stg.cheesecakelabs.com\/blog\/keeping-data-integrity-django-migrations\/"},"modified":"2022-07-01T17:24:15","modified_gmt":"2022-07-01T17:24:15","slug":"keeping-data-integrity-django-migrations","status":"publish","type":"post","link":"https:\/\/blog-stg.cheesecakelabs.com\/br\/keeping-data-integrity-django-migrations\/","title":{"rendered":"Keeping data integrity with Django migrations"},"content":{"rendered":"

Django<\/a> built-in migrations were designed as a way of propagating the changes you make on the models to the database. Migrations exist mostly to keep your code in sync with your schema and to provide a good way of versioning it. We just need a couple of commands to get our apps up and running with an updated version of our database, or <\/span>not<\/span><\/a>.<\/span><\/p>\n

<\/p>\n

Unfortunately, there are some situations when applying a migration by itself is not enough. What would you do if you needed to add a required field to an existing table? How about changing a many-to-many relationship to just a foreign key without losing pre-existing data? It may be painful and critical to make these changes without the right tool; that’s why we have Data Migrations<\/strong>! In this post I’ll use data migrations to approach these two scenarios.<\/span><\/p>\n

First Scenario<\/b><\/h1>\n

Let’s say you have to design a tracking system for a company that sells computers. This system should register the computers bought by each customer. We can migrate the following models:<\/span><\/p>\n

GitHub<\/span><\/a>
\n<\/p>\n

\n
from<\/span> django.db<\/span> import<\/span> models\n\n# Create your models here.<\/span>\nclass<\/span> Client<\/span>(models.<\/span>Model):\n    name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n\nclass<\/span> Computer<\/span>(models.<\/span>Model):\n    brand =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    bought_by =<\/span> models.<\/span>ForeignKey(Client, related_name=<\/span>'computers'<\/span>)\n<\/pre>\n<\/div>\n
Now, for some reason, we’ll need to fetch a list with the information of how many computers each customer has bought. Somehow, this is incredibly expensive to compute and your queries would take extra time if you use <\/span>annotations<\/span><\/a> with a Count aggregation. One possible solution would be to add a column on the Customer model to keep track of the amount of computers sold to that specific customer. To keep data integrity, we’ll make this field required for all new customers.<\/span><\/p>\n
To provide the initial data to  this new computed required field on your model, you can use a data migration. It basically works by executing a custom function in between schema migrations.<\/span><\/p>\n
Step 1.<\/b> First we add the field to our model as a non-required field and run the <\/span>makemigration<\/span> and <\/span>migrate<\/span> commands:<\/span><\/p>\n
GitHub<\/span><\/a>
\n<\/p>\n
\n
from<\/span> django.db<\/span> import<\/span> models\n\n# Create your models here.<\/span>\nclass<\/span> Client<\/span>(models.<\/span>Model):\n    name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    bought =<\/span> models.<\/span>IntegerField(null=<\/span>False<\/span>)\n\nclass<\/span> Computer<\/span>(models.<\/span>Model):\n    brand =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    bought_by =<\/span> models.<\/span>ForeignKey(Client, related_name=<\/span>'computers'<\/span>)\n<\/pre>\n<\/div>\n
Step 2. <\/b>Then, we’ll add a blank migration file; here we’ll add the custom function to perform the data migration. We use the following command:<\/span><\/p>\n
<\/p>\n
\n
python manage.<\/span>py makemigrations --<\/span>empty store --<\/span>name prepopulate_bought_field\n<\/pre>\n<\/div>\n
Note: <\/span>store<\/span> is the Django app you’re working on and you can use the <\/span>–name<\/span> parameter to get more informative names for your migrations.<\/span><\/p>\n
Step 3. <\/b>Now you’ll need to write the function for this migration (this is called the forwards function):<\/span><\/p>\n
GitHub<\/span><\/a>
\n<\/p>\n
\n
# -*- coding: utf-8 -*-<\/span>\nfrom<\/span> __future__<\/span> import<\/span> unicode_literals\n\nfrom<\/span> django.db<\/span> import<\/span> migrations\n\ndef<\/span> add_bought<\/span>(apps, schema_editor):\n    Client =<\/span> apps.<\/span>get_model('store'<\/span>, 'Client'<\/span>)\n    for<\/span> c in<\/span> Client.<\/span>objects.<\/span>all():\n        c.<\/span>bought =<\/span> c.<\/span>computers.<\/span>count()\n        c.<\/span>save()\n\nclass<\/span> Migration<\/span>(migrations.<\/span>Migration):\n    dependencies =<\/span> [\n        ('store'<\/span>, '0002_client_bought'<\/span>),\n    ]\n\n    operations =<\/span> [\n        migrations.<\/span>RunPython(add_bought, reverse_code=<\/span>migrations.<\/span>RunPython.<\/span>noop)\n    ]\n<\/pre>\n<\/div>\n
Note that for using our <\/span>Customer<\/span> model, we can’t just simply import it; this would get the most recent model that could be already changed by someone else (e.g. adding a field). To avoid this, we use <\/span>apps.get_model<\/span> to fetch the current migration version.<\/span><\/p>\n
This operation can also have a rollback (or reverse function) if you want or need to, simply add it as the <\/span>RunPython’s<\/span> parameter <\/span>reverse_code<\/span> – see more <\/span>here<\/span><\/a>. In this example, we use <\/span>migrations.RunPython.noop<\/span> to make the migration reversible without executing any code on the rollback.<\/span><\/p>\n
Step 4. <\/b>After computing and storing all data, we can change the <\/span>bought<\/span> field to <\/span>required<\/span> in our model by setting its <\/span>null<\/span> property to <\/span>False<\/span> (it’s also possible to  do this change on the data migration file if you want to save a migration). This step will generate another migration and you may see something like this in your screen after running <\/span>makemigrations<\/span> command:<\/span><\/p>\n
<\/p>\n
\n
You are trying to change the nullable field 'bought' on customer to non-nullable without a default; we can't do that (the database needs something to populate existing rows).<\/span>\nPlease select a fix:<\/span>\n1) Provide a one-off default now (will be set on all existing rows with a null value for this column)<\/span>\n2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)<\/span>\n3) Quit, and let me add a default in models.py<\/span>\nSelect an option:<\/span><\/pre>\n<\/div>\n
You only need to select <\/span>2<\/span> because that’s exactly what you did: you added a RunPython operation to handle null<\/span> values in a previous data migration.<\/span><\/p>\n
Second Scenario<\/b><\/h1>\n
Now, say we have an address list and we thought it would be nice to have a many-to-many relationship so we wouldn’t need to repeat rows in address table. A few days later you realize that when users change their addresses, it changes for other users with the same address too. You now want to store a full address for each customer, even if it comes with some cost. <\/span>
\n<\/span>The current state of our models looks like this:<\/span><\/p>\n
GitHub<\/span><\/a>
\n<\/p>\n
\n
from<\/span> django.db<\/span> import<\/span> models\n\n# Create your models here.<\/span>\nclass<\/span> Address<\/span>(models.<\/span>Model):\n    street_name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n\nclass<\/span> Client<\/span>(models.<\/span>Model):\n    name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    addresses =<\/span> models.<\/span>ManyToManyField(Address)\n<\/pre>\n<\/div>\n
For this data migration, we’ll need an intermediary model to store addresses data before deleting it! We change our models to something like this:<\/span><\/p>\n
GitHub<\/span><\/a>
\n<\/p>\n
\n
from<\/span> django.db<\/span> import<\/span> models\n\n# Create your models here.<\/span>\nclass<\/span> Address<\/span>(models.<\/span>Model):\n    street_name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n\nclass<\/span> Client<\/span>(models.<\/span>Model):\n    name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    addresses =<\/span> models.<\/span>ManyToManyField(Address)\n\nclass<\/span> NewAddress<\/span>(models.<\/span>Model):\n    street_name =<\/span> models.<\/span>CharField(max_length=255<\/span>)\n    client =<\/span> models.<\/span>ForeignKey(Client)\n<\/pre>\n<\/div>\n
Similarly to what we’ve done in the previous scenario, we create a migration and write write a custom function (or <\/span>forwards_func<\/span>) to move all data that was store on <\/span>Address<\/span> table to <\/span>NewAddress<\/span> table. It looks like this:<\/span><\/p>\n
GitHub<\/span><\/a>
\n<\/p>\n
\n
# -*- coding: utf-8 -*-<\/span>\nfrom<\/span> __future__<\/span> import<\/span> unicode_literals\n\nfrom<\/span> django.db<\/span> import<\/span> migrations, models\nimport<\/span> django.db.models.deletion<\/span>\n\ndef<\/span> migrate_data<\/span>(apps, schema_editor):\n    Client =<\/span> apps.<\/span>get_model('address_book'<\/span>, 'Client'<\/span>)\n    NewAddress =<\/span> apps.<\/span>get_model('address_book'<\/span>, 'NewAddress'<\/span>)\n\n    for<\/span> c in<\/span> Client.<\/span>objects.<\/span>all():\n        for<\/span> a in<\/span> c.<\/span>addresses.<\/span>all():\n            NewAddress.<\/span>objects.<\/span>create(street_name=<\/span>a.<\/span>street_name, client=<\/span>c)\n\nclass<\/span> Migration<\/span>(migrations.<\/span>Migration):\n\n    dependencies =<\/span> [\n        ('address_book'<\/span>, '0001_initial'<\/span>),\n    ]\n\n    operations =<\/span> [\n        migrations.<\/span>CreateModel(\n            name=<\/span>'NewAddress'<\/span>,\n            fields=<\/span>[\n                ('id'<\/span>, models.<\/span>AutoField(auto_created=<\/span>True<\/span>, primary_key=<\/span>True<\/span>, serialize=<\/span>False<\/span>, verbose_name=<\/span>'ID'<\/span>)),\n                ('street_name'<\/span>, models.<\/span>CharField(max_length=255<\/span>)),\n                ('client'<\/span>, models.<\/span>ForeignKey(on_delete=<\/span>django.<\/span>db.<\/span>models.<\/span>deletion.<\/span>CASCADE, to=<\/span>'address_book.Client'<\/span>)),\n            ],\n        ),\n        migrations.<\/span>RunPython(migrate_data, reverse_code=<\/span>migrations.<\/span>RunPython.<\/span>noop)\n    ]\n<\/pre>\n<\/div>\n
We can now remove our old <\/span>Address<\/span> model, then run <\/span>makemigrations <\/span>and rename your <\/span>NewAddress<\/span> model to <\/span>Address<\/span>. It’s important to run <\/span>makemigrations<\/span> in between these two operations, otherwise Django will treat <\/span>NewAddress<\/span> as if it was the old <\/span>Address<\/span>.<\/span><\/p>\n
Note that even using <\/span>migrations.RunPython.noop<\/span>, this migration is not reversible because we’re deleting and renaming models. If you need it to be reversible, you should write another custom function that does the rollback.<\/span><\/p>\n
Conclusions<\/b><\/h1>\n
These operations can be very harmful to your data if used improperly, so it’s always a good idea to have a database dump before doing something like this in production. These scenarios we used as example are a bit silly, but it’s enough to get the idea of how to apply data migrations to solve data consistency issues.<\/span><\/p>\n
You can also make <\/span>fixtures<\/span><\/a> for loading some initial data to your apps or just importing a text file on any of your data migrations and running a forwards function to get them in your database.<\/span><\/p>\n
As Sir Oliver Wendell Holmes would say, <\/span>The young man knows the rules, but the old man knows the exceptions<\/span>. So if you know any exceptions, please let us know, help is very much appreciated.<\/span><\/p>\n
The code used in this blogpost is available on <\/span>GitHub<\/span><\/a>.<\/span><\/p>\n
Good references<\/b><\/h3>\n