{"id":4919,"date":"2017-06-27T20:46:43","date_gmt":"2017-06-27T20:46:43","guid":{"rendered":"https:\/\/blog-stg.cheesecakelabs.com\/?p=4919\/"},"modified":"2022-07-01T18:35:26","modified_gmt":"2022-07-01T18:35:26","slug":"really-annoys-django-migrations","status":"publish","type":"post","link":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/","title":{"rendered":"What really annoys me about Django migrations"},"content":{"rendered":"

Automated database migrations have been a convenient way of dealing with schema changes for a long time in Django. It’s been only 3 years since migrations have been incorporated into Django<\/a> but South<\/a> had been the de-facto solution since 2008.<\/span><\/p>\n

The same way an ORM allows us to forget about SQL when writing queries to the database, migrations make sure we don’t write a single ‘ALTER TABLE’ in<\/span> our schema changes. Some may argue that’s bad: we “lose control” over a critical part of our infrastructure, we don’t know how to write SQL anymore when needed, we’re not sure how that operation is really translated into SQL, etc, etc. Ok, these points are actually valid. However, Django<\/a> migrations module is more than just a way of automatically generating and applying SQL statements, it’s also a transparent API to write your own database changes in Python<\/a>. It comes with wheels for those who need it (or trust enough) and tools for those who like to get their hands dirty.<\/span><\/p>\n

<\/p>\n

At this point, you must have noticed already on which team I play (it’s the pro-automated-migration one, if it was not clear). Still I have a critical point of view when it comes to it. It’s no secret that working with migrations on medium-to-large-sized teams can be quite annoying, having to resolve migration conflicts on each deploy sometimes. And this is old news, I remember even South library had a dedicated section: <\/span>“Part 5: Teams and Workflow”<\/b><\/a>, explaining best practices to avoid and resolve conflicts.<\/span><\/p>\n

In these almost 10 years of Django migrations, there’s plenty of literature on the Internet with solutions to this topic, which summarizes to: <\/span>avoid working on the same app, –merge them, manually change the migration dependencies or rollback and re-apply migrations<\/span><\/a>. Actually, there’s also data migrations and database locks and downtime, but I won’t go over them, because this is not what annoys me the most.<\/span><\/p>\n

The problem<\/span><\/h2>\n

This is what annoys me the most about Django migrations:<\/span><\/p>\n

\"ProgrammingError<\/p>\n

Let me try to put it in words:<\/span><\/p>\n

    \n
  1. I was in branch <\/span>bernardo\/remove-favorite-color<\/b>.<\/span><\/li>\n
  2. Removed the field <\/span>“favorite_color”<\/b> from my <\/span>Profile<\/b> model on <\/span>core<\/b> app.<\/span><\/li>\n
  3. Generated the migration:
    \n<\/span> python manage.py makemigrations core<\/code><\/li>\n
  4. Applied it:
    \npython manage.py migrate core<\/code><\/span><\/li>\n
  5. Committed:
    \ngit add . && git commit -m \"Remove favorite color from profile\"<\/code><\/span><\/li>\n
  6. And switched to branch <\/span>bernardo\/some-other-feature<\/b>.<\/span><\/li>\n<\/ol>\n

    Then you start making some code changes on another app, nothing related to your Profile model, spend a couple of minutes doing it, open Django admin to human-test it and remember you have to create another profile. Nothing wrong with that, right? Nope, you get an error.<\/span><\/p>\n

    Some may say I’m overreacting, because it’s a simple change to fix it:<\/span><\/p>\n

      \n
    1. Stash your current changes:
      \ngit add . && git stash<\/code><\/span><\/li>\n
    2. Go back to <\/span>bernardo\/remove-favorite-color<\/b><\/li>\n
    3. Get the migration name:
      \npython manage.py showmigrations core<\/code><\/span><\/li>\n
    4. Rollback the migration:
      \npython manage.py migrate core 0002_auto_20170618_1549<\/code><\/span><\/li>\n
    5. Go back to <\/span>bernardo\/some-other-feature<\/b><\/li>\n
    6. Unstash changes:
      \ngit stash pop && git reset<\/code><\/span><\/li>\n
    7. Profit<\/span><\/li>\n<\/ol>\n

      Yet the key point is having to stop all your line of thought, rollback the migrations and get back to it. That’s not only annoying, it’s counter-productive. What I wanted is something simple as:<\/span><\/p>\n

      $ ~ git checkout bernardo\/some-other-branch\n$ These migrations are applied to the database and are not on your code:\n$ - core\/migrations\/0002_auto_20170618_1549.py\n$    - RemoveField(model_name='profile', name='favorite_color')\n$ Do you want me to resolve this issue? [y\/n]\n<\/code><\/pre>\n
      One solution<\/span><\/h2>\n
      One day I stumbled upon this guy: <\/span>django migrations – workflow with multiple dev branches<\/span><\/a>. It’s a StackOverflow question asking whether there’s a git-hook to deal with these sort of things. I did find some solutions, but not quite what I was looking for (mainly <\/span>this gist<\/span><\/a> and <\/span>this lib<\/span><\/a>). Talking to other devs here at Cheesecake Labs we thought this deserved some attention. My first try was actually to do something similar to the question’s accepted answer suggestion:<\/span><\/p>\n
      1. Just before switching [branch], you dump the list of currently applied migrations into a temporary file mybranch_database_state.txt
      \n2. Then, you apply myfeature branch migrations, if any
      \n3. Then, when checking back mybranch, you reapply your previous database state by looking to the dump file.<\/p><\/blockquote>\n
      Git has only a <\/span>post-checkout <\/b>hook, so no good to the first item on the list, it’s not possible to know the operations of applied migrations after the checkout happened. Django stores applied migrations on a database table called <\/span>django_migrations<\/b>, but all we have there is a migration file name and app, without the migration operations it had applied. We wanted the hook to be a bit smarter: some migration operations don’t raise ProgrammingErrors, we wanted to provide this information to the developer and avoid unnecessary rollbacks.<\/span><\/p>\n
      Our solution<\/span><\/h2>\n
      To have the desired output mentioned above, we started off with these set of short-term goals:<\/span><\/p>\n
        \n
      1. Find out whether there are applied migrations on the previous branch, that are not present on target branch.<\/span><\/li>\n
      2. Read the contents of these migrations and find all operations.<\/span><\/li>\n
      3. Warn the user, when the checkout is done, that a set of migrations are applied and not tracked.<\/span><\/li>\n<\/ol>\n
        After <\/span>git checkout<\/b> finishes, it calls a <\/span>post-checkout<\/b> hook passing as argument the two branch references and whether it was a branch checkout: <\/span><\/p>\n
        $ post-checkout <previous-ref> <target-ref> <is-branch-checkout><\/code><\/pre>\n
        And then our hook has to follow these steps:<\/span><\/p>\n
          \n
        1. Find the closest commit ancestor between the two references passed by the post-checkout hook. Which translates basically to: find where in the history the two branches have diverted. We use <\/span>git-merge-base<\/b> command for that.<\/span><\/li>\n
        2. With a <\/span>git-diff<\/b> command, list all files that have been changed between the previous branch and the ancestor commit.<\/span><\/li>\n
        3. From this list, filter all files that are migration modules. By default, Django creates a <\/span>migration<\/b> module on each app, but the name can be changed using the <\/span>MIGRATION_MODULES <\/b>settings, so we take this into consideration.<\/span><\/li>\n
        4. With all migrations in hand, check which of them are <\/span>applied to the database<\/b>. We do that by querying the database, using Django’s <\/span>MigrationRecorder<\/b> class, which already provides an applied_migrations<\/strong><\/span> <\/b>helper function.<\/span><\/li>\n
        5. Now with the final migration files list, we can<\/span> execute <\/strong>their code and <\/span>discover operations for each migration<\/b>.<\/span><\/li>\n<\/ol>\n
          We’ve created a library called <\/span>django-nomad<\/b>. So far, we have implemented steps 1 to 5 and a git-hook installer. The code is still on the first stages, but <\/span>it’s open on Github<\/span><\/a> and accepting suggestions and Pull Requests.<\/span><\/p>\n
          What’s next?<\/span><\/h2>\n
          On the long term, we would like to <\/span>automatically resolve<\/b> the mentioned issues. But this raises a couple of concerns, since simply rolling back the previous migrations may cause data loss, and despite being only development<\/a> environment, it can be undesired. <\/span><\/p>\n
          When there’s a problem, there are also possibilities (quoting my friend Jonatas on his post about a Serverless architecture<\/a>):<\/span><\/p>\n
            \n
          1. We could create a separate schema and override the database connector to switch between them based on the branch, whenever a clash happens.<\/span><\/li>\n
          2. Instead of dumping the whole schema, we could maintain only a separate table with different migrations applied, and update the <\/span>db_table<\/b> attribute on each Model.<\/span><\/li>\n<\/ol>\n
            These are only in the field of ideas and were not put into thorough consideration to find pros, cons or blockers yet. Anyway they are a good start.<\/span><\/p>\n
            Thoughts? Suggestions? We’re doing something wrong? Help is always welcome. Thanks!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
            Automated database migrations have been a convenient way of dealing with schema changes for a long time in Django. It’s been only 3 years since migrations have been incorporated into Django but South had been the de-facto solution since 2008. The same way an ORM allows us to forget about SQL when writing queries to […]<\/p>\n","protected":false},"author":65,"featured_media":4951,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[471,344],"tags":[],"class_list":["post-4919","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engenharia","category-opiniao"],"yoast_head":"\nWhat really annoys me about Django migrations<\/title>\n<meta name=\"description\" content=\"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"pt_BR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What really annoys me about Django migrations\" \/>\n<meta property=\"og:description\" content=\"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/\" \/>\n<meta property=\"og:site_name\" content=\"Cheesecake Labs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cheesecakelabs\" \/>\n<meta property=\"article:published_time\" content=\"2017-06-27T20:46:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-07-01T18:35:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2017\/06\/Banner_database2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2000\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Cheesecake Labs\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@cheesecakelabs\" \/>\n<meta name=\"twitter:site\" content=\"@cheesecakelabs\" \/>\n<meta name=\"twitter:label1\" content=\"Escrito por\" \/>\n\t<meta name=\"twitter:data1\" content=\"\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. tempo de leitura\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutos\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/\",\"url\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/\",\"name\":\"What really annoys me about Django migrations\",\"isPartOf\":{\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/#website\"},\"datePublished\":\"2017-06-27T20:46:43+00:00\",\"dateModified\":\"2022-07-01T18:35:26+00:00\",\"author\":{\"@type\":\"person\",\"name\":\"Bernardo Smaniotto\"},\"description\":\"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.\",\"breadcrumb\":{\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/#breadcrumb\"},\"inLanguage\":\"pt-BR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What really annoys me about Django migrations\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/#website\",\"url\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/\",\"name\":\"Cheesecake Labs\",\"description\":\"Empresa de desenvolvimento e design de aplicativos mobile & web que est\u00e1 reinventando o desenvolvimento de produtos com times remotos. N\u00f3s desenvolvemos aplicativos iOS, Android e aplica\u00e7\u00f5es Web com as melhores empresas dos EUA, do Brasil e do mundo.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"pt-BR\"},{\"@type\":\"Person\",\"name\":\"Bernardo Smaniotto\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-BR\",\"@id\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2016\/11\/bernardo-300x300.jpg\",\"contentUrl\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2016\/11\/bernardo-300x300.jpg\",\"caption\":\"Bernardo Smaniotto\"},\"description\":\"10 years of experience in Marketing and Sales in the Technology sector. My main purpose is help, support and structure efficient operations and also develop independent and multidisciplinary teams.\",\"url\":\"https:\/\/blog-stg.cheesecakelabs.com\/br\/\/autor\/bernardo-smaniotto\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What really annoys me about Django migrations","description":"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"pt_BR","og_type":"article","og_title":"What really annoys me about Django migrations","og_description":"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.","og_url":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/","og_site_name":"Cheesecake Labs","article_publisher":"https:\/\/www.facebook.com\/cheesecakelabs","article_published_time":"2017-06-27T20:46:43+00:00","article_modified_time":"2022-07-01T18:35:26+00:00","og_image":[{"width":2000,"height":720,"url":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2017\/06\/Banner_database2.png","type":"image\/png"}],"author":"Cheesecake Labs","twitter_card":"summary_large_image","twitter_creator":"@cheesecakelabs","twitter_site":"@cheesecakelabs","twitter_misc":{"Escrito por":null,"Est. tempo de leitura":"7 minutos"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/","url":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/","name":"What really annoys me about Django migrations","isPartOf":{"@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/#website"},"datePublished":"2017-06-27T20:46:43+00:00","dateModified":"2022-07-01T18:35:26+00:00","author":{"@type":"person","name":"Bernardo Smaniotto"},"description":"In these 10 years of Django migrations, there's plenty of literature with solutions to this topic, which summarizes to: avoid working on the same app.","breadcrumb":{"@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/#breadcrumb"},"inLanguage":"pt-BR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/really-annoys-django-migrations\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog-stg.cheesecakelabs.com\/br\/"},{"@type":"ListItem","position":2,"name":"What really annoys me about Django migrations"}]},{"@type":"WebSite","@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/#website","url":"https:\/\/blog-stg.cheesecakelabs.com\/br\/","name":"Cheesecake Labs","description":"Empresa de desenvolvimento e design de aplicativos mobile & web que est\u00e1 reinventando o desenvolvimento de produtos com times remotos. N\u00f3s desenvolvemos aplicativos iOS, Android e aplica\u00e7\u00f5es Web com as melhores empresas dos EUA, do Brasil e do mundo.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog-stg.cheesecakelabs.com\/br\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"pt-BR"},{"@type":"Person","name":"Bernardo Smaniotto","image":{"@type":"ImageObject","inLanguage":"pt-BR","@id":"https:\/\/blog-stg.cheesecakelabs.com\/br\/#\/schema\/person\/image\/","url":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2016\/11\/bernardo-300x300.jpg","contentUrl":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2016\/11\/bernardo-300x300.jpg","caption":"Bernardo Smaniotto"},"description":"10 years of experience in Marketing and Sales in the Technology sector. My main purpose is help, support and structure efficient operations and also develop independent and multidisciplinary teams.","url":"https:\/\/blog-stg.cheesecakelabs.com\/br\/\/autor\/bernardo-smaniotto\/"}]}},"_links":{"self":[{"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/posts\/4919","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/comments?post=4919"}],"version-history":[{"count":1,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/posts\/4919\/revisions"}],"predecessor-version":[{"id":10472,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/posts\/4919\/revisions\/10472"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/media\/4951"}],"wp:attachment":[{"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/media?parent=4919"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/categories?post=4919"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog-stg.cheesecakelabs.com\/br\/wp-json\/wp\/v2\/tags?post=4919"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}