Markus Holtermann

Writing Safe Database Migrations in Django

2021-06-06T00:00:00+02:00

Five years ago, at DjangoCon Europe 2016 in Budapest, I gave a talk Don’t Be Afraid Of Writing Migrations. Back then, while migrations in Django weren’t particularly new, they shipped in 2014, I saw many folks struggle and worry about touching the migration files. And even fewer wrote migrations by hand. Mind you, I wouldn’t recommend writing entire migration files by hand. Django has tools to generate them for us. But I do recommend looking at them, understanding what they do, and then see if they need to be adjusted. Because, in the end, as an engineer, you will always know more about your application than Django will ever be able to.

As an example, when you add a non-nullable field to a model and run makemigrations, Django will ask you for a one-off default value. That value would be set for all existing rows. This is the correct behavior when there’s data in that table already. But if you know that the table is empty everywhere, it can easily appear to be useless. As an engineer, you can make a distinction there. As an engineer you can decide that it’s fine not to have a default value. As an engineer, you know more about your project than Django does. But Django must be conservative to ensure the default way works for everyone.

Migrations! How? When?

I want to start this post with some general considerations. Questions like how and when do you apply migrations? How are migrations related to deployments? What are dos and don’ts that follow from that?

If you run a somewhat serious site, you are aiming for something called zero-downtime deployments. On common way for that is something called a rolling upgrade or a staged rollout. On a high level, it works like this.

In the beginning, all your servers run the old version of you application. They are indicated by the blue circle. Then you start rolling out the new version to a few servers (indicated by the orange part). You keep this state for a short while, to ensure everything keeps working. Over time, you deploy the new version to more and more servers. Until the new version is running everywhere.

The clear benefit of this rolling upgrade or staged rollout is the ability to notice issues early on, way before the new version is running everywhere. The benefit is in addition to the fact that your site remains fully available during the entire time.

But these benefits come at a price.

As you can easily imaging, running two versions of you application means, they need to be compatible. Essentially, whatever the new version does, the previous version needs to be able to keep functioning the way it did before. This is not always easy. And, unless this becomes a habit, this requirement can easily be forgotten. At least until the next deployment fails because of that.

But while it’s often far from trivial to ensure that the backwards compatibility is kept for the business logic, it becomes even more tricky when we involve databases. Whatever you do with objects in the database, you need to remember that there can always be this one server that has not been updated yet and thus runs the previous version.

Which brings me back to one of the opening questions.

When Do We Deploy Migrations?

Since you never really know when the last server was updated, but it’s somewhat easy to figure out when you start a deployment, I’d recommend to always apply migrations right before you start deploying your code.

Now, this comes with some serious implications. If you do so, you cannot remove a model or a field from a model in the same release as the one that contains the migration. If you were to do that, because migrations run first, you’d remove a database table or a column from a table while some servers may still try to use it.

If you want to remove something, you will need to make two releases. The first one removes the usage of the model or field from all your code. The release is deployed everywhere, the second release can remove the table or field from the database. This is now safe since there’s no code anymore that may access them.

In other words: when you add something to a model, or loosen the constraints, such as the maximum length of a char field, you must do that before the deployment. When you tighten the constraints or remove something, you must have two releases, the first one that works with the new constraints and the second one that then puts them in the database or removes a field or table.

Which leaves the question: how do rename a field? Short answer, you don’t. The long answer is you add a new field, copy the data, and remove the old one. Something similar to the third migration recipe from my previous talk.

Which brings me to the next question.

How Do We Deploy Migrations?

While I’d love to present the perfect solution to you, I can’t. And I can’t, for several reasons.

For once, everyone, every team, every company has their own processes for releasing new versions of their software.

The spectrum of how these processes work is shier endless. There’s that super hip startup, that has a CI/CD pipeline that automatically deploys to a staging system where integration and end-to-end tests are run. And if all of them pass, the CI/CD automatically deploys to production. And the release pipeline runs for each merged pull request.

On the other hand, you have enterprises where a database administrator types in the SQL statement manually to make a change to the database. And that process is preceded with a change request process where 3 department heads have to sign off. Or something like that.

If we look a little bit below the surface of the variety of those processes, we can see that they all have something in common.

Only Go Forwards & Never Look Back

They all only make changes in one direction: forwards.

If you think about it, it actually makes sense. If you made a change to the database, you might not be able to undo it. Data that was removed is gone. You can’t magically undo a DROP TABLE statement once a transaction is committed. What you can do, however, is recreate that table and restore the data from a backup that you obviously took before applying the migration and that you obviously also tested. Because that’s what one does, isn’t it?

The thing is, the moment something is shared with others, and that’s kind of the point of databases, you have to expect that somebody else is using it.

So, why does Django provide a way to unapply a migration? Well, I don’t know. But I do know that it’s quite a useful tool during development. But when it comes to databases for staging and production and such, I’d really recommend to not go backwards. Depending on when exactly the migration occurs compared to when new code is deployed, you may even be running code that expected a migration to be applied.

And, given two migrations that depend on each other, but where the first one has additional, unrelated changes to the second one, how would you roll back those unrelated changes? The answer to that is, a new migration that rolls back the corresponding changes.

This “only go forward” and “apply migrations before deployment” has gone so far for the Django projects I maintain, that the entrypoint script for my Docker containers is this:

#!/bin/sh

set -e

cmd="$@"

until django-admin dbshell -- --command '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

>&2 echo "Postgres is up - executing command"

django-admin migrate -v 2

exec $cmd

I’ll first try to connect to the database, PostgreSQL in this case, until it succeeds. Once done, I apply all migrations in the project. And then execute the actual command, such as running gunicorn.

This approach works very, very well for me.

There’s a small gotcha, though. Since applying the migrations is part of the entrypoint of each Docker container, Django will attempt to apply migrations each time a container starts, which adds to the startup time. However, if no migrations need to be applied, the migrate command is like a no-op. However, when you think back about the staged rollout, you must make sure that the very first stage is exactly one Docker container.

Now, after all this theory, let’s look at something more hands-on.

Adding A Field Is Harmless

Our database models evolve over time. And one of the most frequent changes we do to our models, is adding field. And doing so seems rather harmless, doesn’t it?

We have two models.

from django.db import models

class AddFieldModel1(models.Model):
    name = models.CharField(max_length=10)

class AddFieldModel2(models.Model):
    name = models.CharField(max_length=10)

In the first one, we add a nullable field, in the second one, we add a field with an explicit default value. This seems fine, right?

from django.db import models

class AddFieldModel1(models.Model):
    name = models.CharField(max_length=10)
    field = models.CharField(max_length=10, null=True)

class AddFieldModel2(models.Model):
    name = models.CharField(max_length=10)
    field = models.CharField(default="aaaaaaaaaa", max_length=10)

First, let’s look at the migration that Django creates

from django.db import migrations, models

class Migration(migrations.Migration):

dependencies = [
    ("add_field", "0001_initial"),
]

operations = [
    migrations.AddField(
        model_name="addfieldmodel1",
        name="field",
        field=models.CharField(max_length=10, null=True),
    ),
    migrations.AddField(
        model_name="addfieldmodel2",
        name="field",
        field=models.CharField(default="aaaaaaaaaa", max_length=10),
    ),
]

For those of you who have looked at migration files before, this is nothing new. For everyone else, let me briefly explain what you can see here:

First, this migration depends on another one, namely migration 0001_initial form the app add_field. Which means, this migration can only ever be applied to the database, when that dependency has been applied. Or in reverse: when you are applying this migration, that dependency will be applied before.

Second, you see a list of operations. An operation is Django’s abstraction around some so called database instructions that alter your database, such as adding and removing database columns, creating and removing database tables, and more.

The two operations here, each add a field called field to the models addfieldmodel1 and addfieldmodel2, respectively. The field that is added is then describe there.

We can now use Django’s sqlmigrate command to get the underlying SQL commands.

BEGIN;

--
-- Add field field to addfieldmodel1
--
ALTER TABLE "add_field_addfieldmodel1" ADD COLUMN "field" varchar(10) NULL;

--
-- Add field field to addfieldmodel2
--
ALTER TABLE "add_field_addfieldmodel2" ADD COLUMN "field" varchar(10) DEFAULT 'aaaaaaaaaa' NOT NULL;
ALTER TABLE "add_field_addfieldmodel2" ALTER COLUMN "field" DROP DEFAULT;

COMMIT;

All of these commands still look fairly harmless, don’t they?

Well, you might have guessed it, the answer is no!

The first ALTER TABLE is kind of okay, but the second one can cause you some real headache.

To understand why, we need to understand how databases handle these types of schema alterations.

Adding a nullable column, as we do in the first case, is nothing more than some metadata update. The so called table header will include the new column, a flag that its nullable, and that’s it. None of the existing records will need to be updated. Any new record that has a non-null value, will include that value.

For our second case, however, the database will not only need to add the column to the table header, but it will also need to go through all database records in that table and set the default value. And this can take quite some time, if you have a table with a lot of records.

Additionally, since your database will take a fairly heavy lock on the table, you might even render your site inaccessible, in case the table you’re modifying is used rather frequently. Because both read and write queries might be blocked.

That is, unless you use PostgreSQL 11 or newer, which also deals with the second case in a clever and very efficient way. However, since you might not know which database your code is running on, for example, because you’re writing a reusable Django app, it’s a good idea to always take approach number one and scratch the idea of adding a default value out of your head.

But I Want A Default Value!

Well, okay. You can get a default value. The migration recipe number two in talk linked before gives you step-by-step instructions.

However, I’d only recommend that approach for tables with a fairly small amount of records.

That is, because Django runs each migration inside a transaction. If you’re updating a hundred million records at ones, depending on what your application, or rather its users, might be doing during that time, you can easily get to a point where the transaction needs to be rolled back. Imaging going through 99 million records and then the transaction fails. That’s more than annoying. To ensure that doesn’t happen, you’d need to get a write lock on all records in the table, which can again lead to an unavailability of your entire site.

So, how do you deal with this?

Write a management command and run that after applying the migration:

from django.core.management.base import BaseCommand
from django.db import transaction

from safe_migrations.add_field.models import AddFieldModel1

CHUNK_SIZE = 5000


    class Command(BaseCommand):
    def handle(self, *args, **options):
        updated = CHUNK_SIZE
        while updated >= CHUNK_SIZE:
            with transaction.atomic():
                ids = (
                    AddFieldModel1.objects.filter(field__isnull=True)
                    .select_for_update()
                    .values_list("id", flat=True)[:CHUNK_SIZE]
                )
                updated = AddFieldModel1.objects.filter(id__in=ids).update(
                    field="bbbbbbbbbb"
                )

The management command will lock at most 5000 objects at a time, and then update their field value.

By using select_for_update() for each chunk, you can be sure that the field value for those objects won’t be overridden by anybody else in the meantime.

Sure, running this command will take longer than updating all records at once while locking your table. But it allows you to keep your site operational. Which, very often, I guess, is more important.

But coming back to what I said earlier, as an engineer you know more about the project than Django does, this applies here as well. If you know that the table you’re adding a field to is small or maybe even empty, it’s absolutely okay to add a default value.

Which brings me to another topic. Databases are usually pretty good a retrieving data very efficiently. So much so, that, until a certain threshold, a full table scan is more efficient than looking up rows in an index. But at some point, your table outgrows that point and you need an index.

Adding An Index

Modern Django versions provide not just one but two ways to do so:

from django.db import models

class AddIndexModel1(models.Model):
    name = models.CharField(max_length=10)

class AddIndexModel2(models.Model):
    name = models.CharField(max_length=10)

Firstly, the old way that’s been around forever. You can set db_index=True on a field and Django will create an index.

Secondly, since Django 1.11, you can define class based indexes in a model’s Meta class. They are far more flexible, and powerful. And since Django 3.2 you can even add indexes on expressions, also known as functional indexes.

from django.db import models

class AddIndexModel1(models.Model):
    name = models.CharField(max_length=10, db_index=True)

class AddIndexModel2(models.Model):
    name = models.CharField(max_length=10)

    class Meta:
        indexes = [
            models.Index(fields=("name",), name="my_idx")
        ]

There’s actually a third option. The index_together / unique_together attributes in the model’s Meta class allow you to create indexes on multiple columns. Personally, I’d consider them outdated as well. Additionally, for the example at hand, I’m going to ignore them. Because they behave identically to db_index and can be replaced with class-based indexes.

Looking at the auto generated migration, you can see an AlterField which adds the db_index=True, as well as an AddIndex operation.

from django.db import migrations, models

class Migration(migrations.Migration):

dependencies = [
    ("add_index", "0001_initial"),
]

operations = [
    migrations.AlterField(
        model_name="addindexmodel1",
        name="name",
        field=models.CharField(db_index=True, max_length=10),
    ),
    migrations.AddIndex(
        model_name="addindexmodel2",
        index=models.Index(fields=["name"], name="my_idx"),
    ),
]

A downside of the AlterField operation is, that you don’t really see on the Python level what changed on the field. You need to search for the last migration operation involving a field in order to be able to tell that the index was added.

In contrast to that, the AddIndex operation is clear in what it does: it adds an index.

When we now look at the generated SQL, we can see something very interesting:

BEGIN;

--
-- Alter field name on addindexmodel1
--
CREATE INDEX "add_index_addindexmodel1_name_adf72323" ON "add_index_addindexmodel1" ("name");
CREATE INDEX "add_index_addindexmodel1_name_adf72323_like" ON "add_index_addindexmodel1" ("name" varchar_pattern_ops);

--
-- Create index my_idx on field(s) name of model addindexmodel2
--
CREATE INDEX "my_idx" ON "add_index_addindexmodel2" ("name");

COMMIT;

Firstly, db_index not only adds a single index, but it adds two. The first one is the one that we all expect. The second one, however, is one that Django adds to make LIKE queries efficient.

Secondly, the name for the auto-generated db_index indexes is unpleasant to look at. The 8 random characters are part of an MD5 hash over several attributes to uniquely identify the index.

Using the class based index, we can, however define out own index name, which makes it so much more pleasant to look at. Using meaningful index names has the added benefit that it’s easier to debug database issue. The index name can carry additional context that then allows the database administrators to debug certain issues more effectively. But it’s important to know that some databases, among them PostgreSQL requires an index name to be unique within a database. Using my_idx as I did in the example here, is probably not the best idea. But it’s short to read and makes the code fit on the slides.

Now, if you go ahead and apply this migration on your database, you’ll be fine when there’s not really any load on it and when a table doesn’t have a lot of records. However, as with the ADD COLUMN example earlier, this operation can lock your table for quite a while.

And the worst thing, using db_index, it does so twice. Once for each index. Even if you’ll never use the one for LIKE queries.

I got to admit, though, using a CharField as an example here is the worst example I could give. If you set db_index on an IntegerField Django will only create one index. But this demonstrates that it’s a good idea to look at the migration files and see what they’ll actually do.

So, how do we fix the table lock issue?

Well, PostgreSQL can build indexes concurrently, while allowing access to the data in the underlying table. That, however, comes with the downside that this needs to run outside of transactions.

Since each migration runs within a transaction, we need to set atomic=False. Then we can use AddIndexConcurrently to turn our class-based index into one that’s added concurrently.

from django.contrib.postgres.operations import AddIndexConcurrently
from django.db import migrations, models

class Migration(migrations.Migration):

atomic = False
dependencies = [
    ("add_index", "0001_initial"),
]

operations = [
    migrations.AlterField(
        model_name="addindexmodel1",
        name="name",
        field=models.CharField(db_index=True, max_length=10),
    ),
    AddIndexConcurrently(
        model_name="addindexmodel2",
        index=models.Index(fields=["name"], name="my_idx"),
    ),
]

Let’s look at what actually changed on the SQL level:

--
-- Alter field name on addindexmodel1
--
CREATE INDEX "add_index_addindexmodel1_name_adf72323"
ON "add_index_addindexmodel1" ("name");
CREATE INDEX "add_index_addindexmodel1_name_adf72323_like"
ON "add_index_addindexmodel1" ("name" varchar_pattern_ops);

--
-- Create index my_idx on field(s) name of model addindexmodel2
--
CREATE INDEX CONCURRENTLY "my_idx" ON "add_index_addindexmodel2" ("name");

As you can see, the BEGIN and COMMIT statements are gone. And the last CREATE INDEX statement now has an additional CONCURRENTLY.

Now, if you’re asking yourself how you deal with that on MySQL and MariaDB, I got to disappoint you: you don’t. Because luckily, you do not even need to, because adding indexes there happens without locking the whole table.

Test Your Migrations

Even with all these suggestions and tips, one thing remains. You should test your migrations. I’m not necessarily talking about unit tests. Yes, maybe, it depends. No, I mean, you should test your migrations in a production-like environment. Have some test scenarios available that you can refer to when migrations touch a particularly large table or one that’s accessed frequently. See and try out how the database behaves.

But it’s important to understand, that this level of testing of migrations is not something I’d do for each migration. But it’s something that can help you understand how your database works and what impact on the production environment you might see. But in the end, whatever you do in a testing environment, it’s not your production environment and thus _will_ behave slightly differently. Even if it’s just for the users that behave different than usual.

Summary

Which brings me to the end of this talk.

Let me briefly summarize what we’ve seen today:

It’s usually a good idea to apply migrations before you deploy and run new code. While not trivial, it’s relatively easy to wrap one’s head around it. CreateModel and AddField can go into the same release as the code; DeleteModel and RemoveField need a separate release. Renaming is a combination of add and remove.

It’s a good approach to only ever go forwards. Rolling back database migrations can lead to additional unexpected behavior, in addition to the one you’re facing already.

When adding fields to existing models, make it a habit to add nullable columns without a default value. It’s a good pattern that’s always safe.

If you want default values, that’s fine, but populate existing rows manually.

When you add indexes, try to do that concurrently. Again, especially on bigger tables.

A Pony On The Move: How Migrations Work In Django 🐎

2021-06-05T00:00:00+02:00

Django’s migration framework has been around since version 1.7 in 2014. From the outset, it had a long journey ahead, considering all the learnings and experience Andrew Godwin had with the 3rd party package South. However, Django’s migrations are not to be considered a rewrite of South as they work fundamentally different.

This talk will give an insight view into the components that make up Django’s migration framework. The talk will lay-out how these components fit and work together. It will outline major challenges and problems that were encountered and overcome along the way.

One of the goals of this talk is to reduce the entry barrier for potential Django contributors to the rather complex migration framework. There are some parts that are tricky and non-trivial and non-obvious. Thus, this talk is absolutely aimed at prospective contributors, regardless if they are first-time Django contributors or ones with a year-long track record.

Introduction

These 17 blocks are the main parts that make Django’s migration framework. And I want to try to bring some order to them. You might notice that I didn’t include the management commands. Nor did I include the individual database operations. The management commands use some of these components, but they are not particularly complex, compared to the rest. The individual migration operations are grouped together as “Operations” in the second row to the right.

I think the best way to structure this pile of boxes is to not sort them alphabetically as it is done here.

Instead I am going to arrange them with some arrows, showing which component is using which other component. That means, “A uses B” will be shown by an arrow pointing from A to B.

As you might notice, and also might have expected, there is a lot going on here. And I do mean A LOT! I’ve also omitted some arrows to keep it readable.

This talk will only cover the core parts of the migration system; other parts are ignored. But just because they aren’t covered here it doesn’t mean they’re not important. Each of these boxes is fundamental to the overall working of the migration framework.

So, here’s what this post is going to cover. It’ll be about the core of the migration framework, the applying and un-applying of migrations and how Django figures out what you changed in your models to then create new migration files.

I will highlight the boxes as they’re being covered, and gray out all others that weren’t covered yet. Essentially, we’re going to be the pony that’s migrating through this graph of boxes.

ModelStates, ProjectStates, StateApps

I want to start with these three pieces: ModelStates, ProjectStates, and StateApps. If you want to define a hierarchy of importance among all components making Django’s migration framework, these three are by far the most important ones. Nothing else works without them.

When we talk about migrations, we’re almost always talking about a timeline of your project. And we look at the state of the database and the corresponding models at some point in the life of your project. A ProjectState is essentially a snapshot at an arbitrary point in time on the timeline.

A ModelState represents a single model at some point on the timeline. In many ways, it looks like a Django model. But it isn’t. It’s a lightweight class without a lot of magic or logic. Because this class is instantiated hundreds, often thousands, and sometimes even millions of times. It contains a model’s name and app label. It’s fields, indexes, and constraints. And stuff like that.

The StateApps are pretty much an App registry as you may know from django.apps.registry.Apps. The place where all models of your Django project are registered. Its purpose inside the migration framework becomes more clear when we look at the Migration Executor later on. For now, let’s say, it’s an App registry with some models at some point in time, much like the ProjectState.

I just said, a ProjectState is a snapshot of models at some “point in time”. What that means is, whenever you or Django changes something in the models, such as deleting a field or adding a model, that is a new state of your project. Because your models before and after these events are different.

In Django’s migration system terminology, these “events” are called migration operations. And that’s what we’re looking at next.

Migrations and Operations

There are about two dozen migration operations. They are for example AddField and RemoveField, CreateModel and DeleteModel, but also less commonly used ones, like AlterOrderWithRespectTo, RemoveConstraint, or for example, PostgreSQL’s specific one, CreateExtension.

Operations have two tasks. They look at an instance of a ProjectState and then mutate the ModelStates that are in the ProjectState. It’s important to know that there’s only ever a “mutate forward”. The method on the Operation classes for that is called state_forwards().

The second thing operations do, is deciding on what to do with the database. For that, and unlike for the state, there are the database_forwards() and database_backwards() methods. They talk to the SchemaEditor and ask it for example to create a new table or add a column to an existing table.

A Migration is a bundle of these Operations. All Operations within a migration are executed in a single database transaction. That is, if the underlying database supports that.

A single migration is also always bound to a single Django app. Which means, all operations within the migration work on the same Django app. But, and I can’t stress that enough, each operation has access to the whole ProjectState, and thus to all apps and models. That is important when migrations deal with ForeignKeys. Because, let’s say, you change a model’s primary key from an Integer to a String. Then the AlterField operation will go through all models and check if there’s a field that points to the model you just change. And if there is, it will change that field’s database column’s data type as well.

Now that we know how the migration system keeps track of state and how to mutate the state and talk to the database, how does the framework know in which order the changes are applied?

Migration Graph

The answer to that is the MigrationGraph. And I’ll only briefly cover this one. In a mathematical sense, it’s a so-called directed acyclic graph.

A Migration has two, one might argue three, attributes that the graph considers for ordering:

First, and most commonly used: dependencies. That’s a list of other migrations that need to be applied before the migration in question can be applied. Let’s say, you have a migration that creates a model, and another migration that adds a ForeignKey on another model to that model. Well, The target model has to be created first. Otherwise the foreign key constraint can’t be fulfilled.

Second, barely known and less often used is the attribute run_before. It works exactly like dependencies. Just the other way around.

And third, there is the replaces attribute on a migration. Which comes into play when you squash migrations. And it essentially replaces a set of migrations with a single other one.

Now, the question arises, how does Django know about the migrations in your project? For that, there’s the migration loader

Migration Loader

The potentially obvious task for the MigrationLoader, is loading all migration files from all apps in your Django project and adding them to the graph that we just talked about. As part of that, the loader is actually going to import the Python file that contains the migration and instantiate the Migration class with the app_label. This is where the migration gets the app_label from, since it’s not an attribute on the class.

But building the graph is easier said than done. Because, remember how I just quickly skipped over the replaces attribute on Migrations, and how it’s used with migration squashing? Well, building the _actual_ graph in the migration loader will need to account for that. The graph may go and replace some migrations with a replacement. But if some of the replaced migrations have already been applied, then the replacing one cannot be used. Instead, the remaining replaced migrations need to be used.

The loader does a few more things. For example, it checks that the migration history is sound. If you have applied migrations but some of its dependencies haven’t been applied. It’s going to yell at you.

And lastly, the loader provides an interface, to create a ProjectState from the underlying graph.

At this point, let’s recap what we have talked about so far. We know how to load migrations. We know how to turn them into a graph that defines in which order to do what. And we know the underlying data types that represent a database state at some time.

Which brings us to our next component.

Migration Executor

It’s the brain of applying and unapplying migrations. From the outside, there’s only one important method: migrate(). The method takes one required argument: targets. Targets are the names of the migrations that you want to have applied or unapplied at the end of the method call. Which means, if you want to apply your whole project, it is a list of all the last migrations in each app. These migrations are also called leafs.

From those targets, the executor will attempt to build a migration plan. If you’re curious, you can use the showmigrations management command with the --plan option to see what the plan would look like when you apply your whole project.

As part of building the plan, the executor will look at which migrations have already been applied and which ones are still outstanding. Or vice versa, the migrations that are applied and need to be unapplied.

Now, when migrating forward, that is, applying migrations, the executor is going to start with a fresh ProjectState and then iterates over all migrations in the plan and calls mutate_state() on each migration. Each migration will then in turn call state_forwards() on each migration operation. That builds up the representation of your database operation by operation. And it does that, up until the point when the first migration will need to be applied. Which means, up until the point when operations will need to talk to the database through the SchemaEditor.

At that point, the most crucial part of the migration process occurs: rendering models. After the rendering, the executor is going to call the apply() methods of each migration. Which will mutate the state forwards operation by operation, and also applies the database changes operation by operation.

Before I go into the unapplying part I want to demystify the “rendering of models”. Remember how, in the beginning I said that ModelStates are “just like a model”? And how a ProjectState knows about all ModelStates at a given time?

Model Field References

Model classes and their fields have references to each other and among each other. For example, you can use a model’s _meta API and get all fields on that model. And each field will have a model attribute, pointing back to that model.

And if you for example add a ForeignKey on model A, pointing to model B, then Django will automatically add a reverse ForeignKey, which is a ManyToOneRel, from B to A. Which is effectively a field on B. And the fields themselves will have the attributes related_model and remote_field which point to the corresponding part on the other side of the relation. So, adding a ForeignKey to A not only changes A but also B. And just with a few models and ForeignKeys you will end up with millions of pointers that the migration framework would need to keep track of.

Performance Considerations

To avoid that, the migrations work on ModelStates, which do not have these references. And because the SchemaEditor only works with model classes, the ModelStates need to be converted into them. And that’s called “model rendering”. And is a far better approach than working with model classes to start with. Because keeping track of these pointers is pretty much impossible. And you can trust me on that, because I’ve spend literal days debugging issues inside the migration framework where pointers where stale pointers were the cause of a problem.

There’s an infamous ticket #23745 which is about caching the rendered model classes. For those of you who’ve been around long enough, that is from Django 1.7 onwards, migrations got a significant speed boost in 1.8 with that ticket. There’s also a lot more into the rendering of ModelStates and ProjectStates that I’d love to write about but don’t have the time for. Like, for example, figuring out when to evict some cached model class. Which is enough content for a whole post.

Unapplying Migrations

Now for the unapplying part: It starts off like the applying part, by generating a plan to follow. But since ModelStates can only ever be mutated _forwards_, the executor will cache all intermediate ProjectStates for each migration that will be unapplied. If you’ve ever encountered a huge memory load during unapplying migrations, this is why. Once the ProjectStates have been cached, the executor is going to call a migration’s unapply() method. And it’s doing that in reverse order of the plan.

Migration Autodetector

As the last trip of our journey through Django’s migration framework, I want to briefly look into the MigrationAutodetector.

The autodetector looks at your current models and the ProjectState represented by all migrations in your project, and then tries to figure out what operations need to be added in order to get your current ProjectState to the state that resembles the models in your project.

At this point, I was originally going to say “nothing is as simple as that”. Because, on the surface, it looks kind of simple. But the devil is in the details. There’s this method _detect_changes() which synchronously calls a dozen different methods to generate the changes needed for new and removed models, added and removed fields, and everything else one can do to a model. And each of the generation methods more or less works similar: they are iterating over all models in the ProjectState represented by migrations and the ProjectState``representing all models in your Django project. And for each model they then do their thing. For example, they compare the list of fields on a model and then decide that two fields were added and are not in your migrations yet. Thus, two ``AddField operations will be added. And it’s actually these generating functions that hide the complexity. And some of them are a few hundred lines long.

Summary

This concludes our journey through Django’s migration framework. As you’ve seen, there are numerous parts involved. And I can only encourage each and everyone of you to take a look at the code and explore it. And if you’ve been hesitant or scared away by its complexity so far, I hope this post made you curious to learn more.

FAQ

Why does changing verbose_name cause a migration?

A question that pops up repeatedly is around the field attributes. Such as, why does changing the verbose_name or choices cause a migration. With the post a hand, I hope you now know the answer now: because the ModelState for the corresponding model changes. And the reason why we can’t filter out some attributes is, that nobody knows which attribute somebody may or may not use inside some migration. Like, choices for example, it could be used to validate data in a data migration. Even though the attribute doesn’t have any effect on any database query in migrations.

Why can’t I remove old field classes?

Another question is about old field classes, and why they can’t be removed. And again, because of the ModelStates. If there’s only one migration in your Django project that refers to that field, well, the field class needs to stay around. You can use migration squashing to possibly get rid of the field. And if that doesn’t work automatically, maybe try writing the squashed migration by hand. But as long as there’s a single migration that still imports and uses the field, the field class needs to stay around.

Why doesn’t the SchemaEditor use ModelStates?

When I explained what “model rendering is”, one might ask, why does the SchemaEditor not “just work” with ModelStates. The answer to that is two fold.

Firstly, the SchemaEditor is part of the database backend and doesn’t “know” about the internals of the migration framework. Making it work with ModelStates would mean opening up some of the internal API, such as the ProjectState and ModelState. Which would not necessarily be an issue, because the benefits almost certainly outweigh the costs in this case.

Secondly, because that change needs to happen in a backwards compatible way. Which is where it gets fairly tricky. It’s not about the built-in database backends. The SchemaEditor has a publicly documented API. We will therefore need to provide a proper migration path covering one LTS cycle. Which adds a lot of maintenance burden. But there is ticket #29898 that is precisely about this: Adapting the SchemaEditor to operate on ModelStates instead of rendered models.

All Hands on Deck — Handling Security Issues

2019-12-06T00:00:00+01:00

Building a service that unintentionally exposes millions of people’s user data or is open to all kinds of other security issues is, unfortunately, very quickly done. Reason enough to look at who’s going to be involved when responding to an issue.

A Historic Event

On April 10th, 1912, the R.M.S. Titanic on April 10, 1912 was on its maiden voyage from the port in Southampton to New York City. At that time, the Titanic was the largest ocean liner in service. There were about 2,224 people on board.

Four days into the journey, on April 14, at about 9 am ship time, Captain Edward Smith received this message from steamship Caronia:

West-bound steamers report bergs, growlers, and field ice […]

A few hours later, at 1:42 pm, this message from steamship Baltic made it to the captain:

[…] Greek steamer Athenia reports passing icebergs and large quantities of field ice today […]

The captain acknowledged both messages. But it wasn’t uncommon to see icebergs and field ice in that area in April. And it was also a general assumption that ocean liners weren’t at risk when hitting an ice berg.

Throughout the day, four more messages were received by the radio operators on board Titanic. At 1:45 pm:

Amerika passed two large icebergs […]

At 7:30 pm, a message from the Californian to the Antillian:

[…] Three large bergs 5 miles southward of us. […]

And at 9:40 pm from the Mesaba to the Titanic:

Ice report […]. Saw much heavy pack ice and great number large icebergs. Also field ice.

These three messages got lost and never made it to the captain.

The sixth and final message which was about to be received by the radio operators on the Titanic at 11 pm from the Californian. The Californian was another steam ship which stopped nearby in the middle of the ice. However, the radio operator on duty at that time was still busy processing a backlog of messages from the day before. Instead of responding he, shut up the sending operator with:

Shut up! Shut up! I’m working Cape Race.

Cape Race was a radio station on shore, close by.

Ten minutes later, at 11:40 pm, after sighting an iceberg right ahead, the Titanic hit it starboard. Over the next 2 hours and 40 minutes, more and more water made its way into the hull.

The Titanic sank at 2:20 am on April 15, 1912, killing more than 1500 people.

A Present Company

Receiving Security Issue Report

Not as dramatic is receiving a notice of a possible security issue in a tech company or department. Usually, nobody’s life is on the line. But it may very well be in case of — for example — hospitals.

Support teams at companies receive several messages a day, be it support requests, spam, or reports of possible security problems. The people who receive those messages need to sort through them, assess and triage them, and process them appropriately.

When their triage of a security issue report is off, the report will not be classified as a potential security issue. It will instead pile up among the other bug reports.

But even when the triaging is right, the report needs to reach the team who is going to deal with it next: the security team, project managers, or product managers. And they need to act upon it.

Freeing Up Resources

Product and project managers will need to budget and plan for engineers to work on the problem. But the report might also go to a dedicated security team that will do a more in-depth analysis of the issue and will then communicate with the product and project managers or fixes it themselves.

Once assessed, if there is an immediate workaround that mitigates the issue in the short term, all other customers should be informed accordingly. That will again involve customer support or key account management.

Fixing The Security Issue

When the developers came up with a security fix for the issue, other parties in the engineering department will need to get involved as well: testing and quality assurance, technical writers and documentarians, and whoever else in the company that is involved in any regular change to a product.

Getting A Fix To Customers

Most of the positions mentioned so far are part of the “Engineering Department.” But they are not the only ones involved.

Looking at proprietary products where customers usually need to pay for an update, the sales department needs to get involved and decide if they are going to hand out an update for free or if they are going to charge customers for it despite the vulnerability. The product managers, in conjunction with the sales department, should also decide if and how much time and money they invest in backporting a security fix to older versions that may still be used by customers.

Dealing With The Fallout

Imagine one of these leaks that exposed millions of user data records. That’s a PR nightmare. The public relations office will have their hands full. And so does the marketing department. A product slogan like “your secure thing-y” can quickly become a mockery.

Furthermore, the legal department is going to be involved. The data protection officer will need to talk to the authorities and inform them about the leak.

In the case of a production company, if the bad press about the issue causes sales of products to go down, then the purchasing department should probably think about reducing the number of purchased parts.

What Do We Learn From The Disaster?

(Re)evaluate Requirements

While standards, requirements, guidelines, and such are great things, they can also be incomplete, incorrect, or inadequate.

At the time when the Titanic sank, British vessels over 10,000 tons needed to carry 16 lifeboats. The Titanic carried 20 and was thus well above the requirements. But the lifeboats only provided space for 1.178 people, just about half of the number of people on board the Titanic. And only a third of the total number of people fitting on the Titanic. But that should have been fine. Because lifeboats were meant to get people from one ship onto another that was in close proximity. Nobody expected an ocean liner to sink within 2 hours.

In today’s world, just because sending TANs for online banking via SMS is common practice, it doesn’t mean it’s the right thing to do.

Practice Makes Perfect

It’s also been documented that a lot of crew members had no understanding of the evacuation procedures. This is not only because of a lack of communication. But more importantly, because of the lack of practice. Even with 40 years on the job, captain Smith appeared to be paralyzed when he grasped the enormity of the problem.

Translating this into the tech world means we need to train ourselves for the case of a security incident. A documented procedure is excellent. But without practice, people will act in all kinds of ways, possibly irrationally.

Communication Is Crucial

And lastly, and most importantly, communication is crucial.

It’s been documented that captain Smith ordered his first and second officers to “put the women and children in [the lifeboats] and lower away.” But the officers interpreted it differently. One put men next to the woman and children when no women or children were around. The other one lowered lifeboats with empty seats.

As you can see, there can be numerous people involved when it comes to handling a security issue. And a lot of people mean a lot of communication. Clear communication is important in any company. But it’s crucial when it comes to handling security issues.

Resources

The Oceanus magazine, volume 28, number 4, winter 1985/86. This edition gives an in-depth insight into what happened around the Titanic tragedy.
And Agile Application Security by Laura Bell and others. It’s about how agile teams can deal and handle security issues. And they can do that despite their ability to iterate quickly.
Slides
Video recording

Logging Rethought

2019-04-12T00:00:00+02:00

The Current State Of Logging

We build services, potentially used by hundreds, thousands or even millions of people. And despite all the testing we do, some operations and interactions with these services will not work out the way we expect or hope. Wouldn’t it be great to be able to reconstruct and understand what led to a problem? And then analyze if the same problem occurred at other times to possibly find the root cause of the problem?

Historically, we’ve used some form of plain text or prose-style logging within the applications we build and operate. Code like the following is not uncommon:

logger.error(
    "Login failed because the connection to the authentication provider "
    "%s timed out.",
    provider_name
)

This message is comprehensible and provides a lot of valuable information to a human reader. The message will either be written to a file, the standard output or some service that will do something with it.

Often, these log messages will not be looked at until something is broken. Typically, at that time, an engineer will need to sort through and scroll through thousands of lines of text until they eventually find this one message among a billion other messages:

[2019-04-01T10:47:04.139+00:00] [ERROR] [srv01] Login failed because the connection to the authentication provider google timed out.

It is then up to them to understand and deduce that probably a user tried to log in using credentials for an external authentication provider called google. They can also understand that the login did not succeed because the connection between the application and the authentication provider was too slow or otherwise interrupted and thus timed out.

Our Logging Is Broken

While the message mentioned above is understandable and provides some valuable information to a human reader, it does not provide any helpful information for the problem at hand at all. In retrospect, nobody will easily be able to figure out the problem given this message.

To understand what information would be helpful, we’d need to look at the situation from the perspective of the one dealing with the incident: What IP and host did the server try to connect to? What was the timeout limit at the time? How many other attempts were made to talk to that authentication provider? Where other outgoing connections affected by timeout errors as well? Were other servers affected by the same problem?

Tracing Events

The scenario above was rather well constrained: a single application having trouble talking to some external service. However, most systems these days are not that simple anymore; they’re built with a macro-service or microservice architecture in mind. There are half a dozen to hundreds of individual services doing work.

One of the key problems that come with a macro- and microservice architecture is its complexity in dependencies. One service depends on another that depends on another that depends on another. When one service causes problems, chances are, other services are not running smoothly either.

Often, communication between individual services is handled via HTTP requests. To better understand how requests — or more generally events — relate to each other, what request triggered what other request and what failed request caused another request to fail, one can tag each request with a Request ID or a Trace ID and provide that identifier along with log messages. Additionally, these trace IDs are passed on to sub-requests. This concept is generally known Event Tracing.

Tracing Errors

When combined with error tracing tools, such as Sentry, attaching a trace ID to the error event would allow whoever investigates it to look up all log messages related to that event. And with that additional context, they are better equipped in identifying, understanding, and resolving the problem than without it.

There Is More Out There Than HTTP

But event tracing does not need to stop at the HTTP request/response level. Messages put into a queue can contain the trace ID as well. This allows workers processes to pull the trace ID from the message when they start their work and attach that trace ID to every log message they are writing. When looking at the collection of log messages for a given trace ID from all different services, the log messages from workers would show up as well and give viable information why, for example, some service was talked to or why not.

Tracing Users And Other Objects

While tracing events, especially across service boundaries, is already a considerable improvement and significant benefit compared to typical logging behavior, it’s not the only thing that can be traced. Anything with an identifier can be traced.

To give an example, a request that comes into one service will cause a request to an authentication service, which will return the authenticated user to the first service. Log messages within the first service that are written before the user was authenticated must clearly identify it as such, and so must log messages after a user was authenticated. That allows for a much easier understanding of why something failed: because the user in question hadn’t been authenticated yet.

Similarly, one could trace any object by its identifier throughout the entire service architecture.

Adding Structure To Our Logs

Let’s look back at the log message from above:

logger.error(
    "Login failed because the connection to the authentication provider "
    "%s timed out.",
    provider_name
)

As already mentioned, this log message does not really give any meaningful information. It’s at least missing the target IP address, the timeout limit. Either can be added easily to the message:

logger.error(
    "Login failed because the connection to the authentication provider "
    "%s @ %s timed out after %s ms.",
    provider_name,
    provider_ip,
    timeout,
)

This will then result in a message much like this:

[2019-04-01T10:47:04.139+00:00] [ERROR] [srv01] Login failed because the connection to the authentication provider google @ 8.8.8.8 timed out after 5 ms.

Now, this log message is already far more helpful. Firstly, the IP of the authentication provider seems to be 8.8.8.8. Everybody knows this IP address points to Google’s public DNS service and not some authentication provider. Secondly, the timeout is five milliseconds, that’s hardly enough time for a PING round-trip on a dodgy WiFi connection (or about 1500km return — 750km one way).

The problem here is, though, attaching all this information to some prose-style messages makes them rather tedious and hard to read and terrible to understand. Additionally, questions regarding the locality of the problem, its frequency, etc. are still not easy to answer.

A solution to solve all these troubles that the traditional logging approaches seem to have can be structured logging. With structured data, when appropriately presented to an individual, it contains not only the explicit information that’s already part of the message above, but the presentation itself can convey additional information.

This post is going to use structlog for demonstration purposes.

Similar to before, the application will need to log events. But instead of using a prose message an event key or event identifier is used [1]. Additionally, further information is passed along as keyword arguments.

import structlog
logger = structlog.get_logger("structlog")
logger.error(
    "auth_provider_failed",
    provider_name=provider_name,
    provider_ip=provider_ip,
    timeout=timeout,
)

When looking at how the event is recorded internally, it may look like this:

{
    "event": "auth_provider_failed",
    "timestamp": 1554115624.139,
    "level": "error",
    "provider_name": "google",
    "provider_ip": "8.8.8.8",
    "timeout": 5,
}

On top of the data given, there is a timestamp and a level item. They were added by structlog automatically.

Now that there is structured data, the event record can be printed out to something a human can easier grasp. For that, structlog provides a simplistic KeyValueRenderer and a more fancy, colorizing, ConsoleRenderer. Using them, once can expect an output that’s close to this:

2019-04-01 10:47:04.139 [error ] auth_provider_failed [structlog] provider_name='google' provider_ip='8.8.8.8' timeout=5

While the KeyValueRenderer and the ConsoleRenderer are great for development, they still provide almost unparsable output and thus make the whole point of having structured logging mood. In deployed environments, a renderer like the JSONRenderer will come in handy. It serializes the dictionary shown above into a JSON string:

{"event": "auth_provider_failed", "timestamp": 1554115624.139, "level": "error", "provider_name": "google", "provider_ip": "8.8.8.8", "timeout": 5}

WYSIWYU – What You See Is What You Understand

As we all know, JSON is a format understood by pretty much every application out there. When the JSON object is written to a log file that file can be watched by tools like fluentd or filebeat. These services can ingested the log records into some long-term storage like CrateDB. Visualization tools like Grafana can then be used to turn the events into a graph which a human eye is better at recognizing than at finding a log message among a thousand others.

As an example, when looking at the following graph, one can see that there is something happening out of the ordinary around the 12:50pm. With timezones in mind, 12:50pm Berlin time is about the same time as 10:47 UTC on April 1st, 2019, and thus around the time of the event that was recorded above.

Without even looking at any logs, one would be able to say that something is broken.

A Picture Says More Than A Thousand Words

Consulting two additional graphs — the first one shows the number of log messages per log level over time, the second is about the number of started, finished and failed requests over time — additional information is revealed.

At some time around 12pm Berlin time, an increase in failed requests and error messages can be seen. Together with the previous findings, there is a high likelyhood that the error with the authentication provider at 12:47pm was caused by something that happened around 12pm.

As an engineer, this piece of information is vital. Because that at the log messages around 12pm could then reveal that e.g. a deployment happened, or that somebody changed some configurtaion value.

Furthermore, the first graphs also shows that the error about the authentication provider is not an isolated incident but part of a bigger problem.

Structuring My Django Logs

All the above is nice and fancy and shiny and seems to be the unicorn everybody wishes for. But how can structured logging be used in a real application?

There is a repository with a bunch of different logs on my GitLab account. It comes with a docker-compose setup of a couple of different services that interact with each other. It comes with a fluentd setup that writes log messages into a CrateDB instance. And a Grafana dashboard showing some graphs.

Logging For European Users

When logging information, potentially data about users and customers ends up in those logs. Since th EU has graciously provided the beloved GDPR, there is obviously no way to just log whatever one wants.

To keep this section short, a few tips on what to avoid when logging:

Never log any secrets. EVER!
Ensure your log data store is secured and not world-readable! NO, REALLY!

Explicitly log whatever should end up in logs:

# DO
logger.error("some_event", foo="bar", lorem="ipsum")

# DON'T DO
logger.error("some_event", **some_object.__dict__)

Come up with a naming scheme and convention for event names before you begin.
Rather log a bit too much than too little.
Log object IDs as well as their attributes where it makes sense. The attributes may change over time.
Improve as you learn. Add missing information when you notice it would be great to have it.

[1]	It’s usually a good idea to think about the different events that could be logged now and in the future and to come up with a naming scheme and naming convention before logging events.

Less Obvious Things To Do With Django’s ORM

2019-03-23T00:00:00+01:00

Django’s ORM is powerful. And it has been for ages. It only became even more powerful over the last years. Django’s ORM can be intimidating because it can do so many things. And it can also be confusing when trying to find out how to do things.

Our Models

Before we begin with the examples we need to define our database layout. Throughout this post I will be using these three database models:

from django.db import models


class Author(models.Model):
    name = models.CharField(max_length=200)

    def __str__(self):
        return self.name


class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(
        Author, on_delete=models.CASCADE, related_name="books"
    )
    votes = models.PositiveIntegerField()
    genres = models.ManyToManyField("Genre", related_name="books")

    def __str__(self):
        return self.title


class Genre(models.Model):
    name = models.CharField(max_length=200, unique=True)

    def __str__(self):
        return self.name

You can input whatever data you want. My talk-orm repository on GitHub contains the full Django project as well as some sample data I sourced from Goodreads.

Counting objects

Let’s start with something basic. We’ll count how many records we have in the database:

>>> from literature.models import Author, Book, Genre

>>> Author.objects.count()
1142
>>> Book.objects.count()
3578
>>> Genre.objects.count()
412

We access the attribute objects on a database model class. objects is a “Model Manager”. Django uses them to connect database model classes and QuerySets. We then access the count() method on a manager which uses the underlying, auto-generated queryset to create the SQL that returns the number of records for that model.

Fetching single objects

Great, let’s fetch some random author from the database:

>>> Author.objects.get()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File ".../django/db/models/query.py", line 403, in get
    (self.model._meta.object_name, num)
literature.models.Author.MultipleObjectsReturned: get() returned more than
one Author -- it returned 1142!

Well, that obviously didn’t go as planned. We got an exception that multiple objects got returned. That’s what get() is all about. It returns exactly one object. Not zero, not two or more. One!

Alternatively, we could use first():

>>> Author.objects.first()
<Author: Bill Bryson>

The difference here, first() may return None in case there is no record to return. We can see that when we try to select me as an author:

>>> Author.objects.filter(name="Markus").first()
>>> Author.objects.filter(name="Markus").get()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/query.py", line 399, in get
    self.model._meta.object_name
literature.models.Author.DoesNotExist: Author matching query does not exist.

While first() returned None, get() raised a DoesNotExist exception.

Advanced filtering

We’ve just seen how we can filter on the exact value of a model field when querying the database. But there’s more. For example, we can select all authors whose name starts with "Lisa":

>>> Author.objects.filter(name__startswith="Lisa")
<QuerySet [<Author: Lisa Unger>, <Author: Lisa Scottoline>, ...]>

Similarly, when we don’t care about upper and lower case and only care if the character sequence is part of a record, we can use __icontains:

>>> Author.objects.filter(name__icontains="tom")
<QuerySet [
    <Author: Robert Tombs>,
    <Author: Tom Barbash>,
    <Author: Tom Sweterlitsch>,
    ...
]>

All these Field Lookups are fully documented.

Following related objects

Let’s say we want to print a list of all books with their corresponding author. The first approach we will make may very well look like this:

>>> books = Book.objects.all()
>>> for book in books:
...     print(f"Title: {book.title} -- Author: {book.author.name}")
Title: Harry Potter and the Sorcerer's Stone -- Author: J.K. Rowling
Title: Harry Potter and the Prisoner of Azkaban -- Author: J.K. Rowling
Title: A Short History of Nearly Everything -- Author: Bill Bryson
Title: The Lord of the Rings -- Author: J.R.R. Tolkien
Title: The Fellowship of the Ring -- Author: J.R.R. Tolkien
Title: One Hundred Years of Solitude -- Author: Gabriel García Márquez
Title: Angels & Demons -- Author: Dan Brown
...

That works, we end up with a lot of database queries. Specifically, we end up with 1 + $number_of_books queries. Why is that?

First, we’re selecting all the books. That’s one query. Then, in the for loop, we make one query per book. In case you’re wondering: that is bad! It may be “just fine” for two to five books, but when you have more, you will end up with performance problems.

Sidebar: Inspecting database queries

When you’re developing your Django project or app, it can be helpful to check the recent database queries quickly. For that, Django tracks them on the database connection:

>>> from django.db import connection

>>> connection.queries
[
    {
        'sql': 'SELECT COUNT(*) AS "__count" FROM "literature_author"',
        'time': '0.000',
    },
    {
        'sql': 'SELECT COUNT(*) AS "__count" FROM "literature_book"',
        'time': '0.000',
    },
    ...
]

Following one-to-one and many-to-one relationships

Getting back to where we left off, we need to find a way to optimize our 1 + $number_of_books database queries. And Django has just the right tool for that: select_related(). This queryset method tells Django to fetch forward relationships when making the initial query:

>>> books = Book.objects.select_related("author").all()
>>> for book in books:
...     print(f"Title: {book.title} -- Author: {book.author.name}")
Title: Harry Potter and the Sorcerer's Stone -- Author: J.K. Rowling
Title: Harry Potter and the Prisoner of Azkaban -- Author: J.K. Rowling
Title: A Short History of Nearly Everything -- Author: Bill Bryson
Title: The Lord of the Rings -- Author: J.R.R. Tolkien
Title: The Fellowship of the Ring -- Author: J.R.R. Tolkien
Title: One Hundred Years of Solitude -- Author: Gabriel García Márquez
Title: Angels & Demons -- Author: Dan Brown
...

Now we have only 1 query. Exactly what we wanted.

I wrote above that select_related() is for forward relationships. That means, it only ever works when there is at most one object on the other end of the relationships. “At most,” because that related object could also be None, e.g. when you have a ForeignKey with null=True. In other words, you can use select_related() when the current model has a ForeignKey or OneToOneField, or if the current model is the opposite end of an OneToOneField. It will not work for ManyToManyFields or the reverse of a ForeignKey.

Following one-to-many and many-to-many relationships

When there are one-to-one and many-to-one relationships, there are probably one-to-many and many-to-many as well. And indeed, there are. You use them when you have ManyToManyFields or when you follow a ForeignKey backward.

Given our database schema, we have that scenario when we want to list all the books for each author. The naïve approach will look a bit like this:

>>> authors = Author.objects.all()
>>> for author in authors:
...     print(f"Author: {author.name}")
...     for book in author.books.all():
...         print(f"  - Title: {book.title}")
Author: Bill Bryson
- Title: A Short History of Nearly Everything
Author: Edith Wharton
- Title: Ethan Frome
Author: Patricia A. McKillip
- Title: In the Forests of Serre
- Title: The Bards of Bone Plain
...

As you might imagine, this has similar problems as the example I had above. We now have 1 + $number_of_authors queries: one for the list of authors, and one for each author to get the books. We can optimize this to exactly two database queries:

>>> authors = Author.objects.prefetch_related("books")
>>> for author in authors:
...     print(f"Author: {author.name}")
...     for book in author.books.all():
...         print(f"  - title: {book.title}")
Author: Bill Bryson
- Title: A Short History of Nearly Everything
Author: Edith Wharton
- Title: Ethan Frome
Author: Patricia A. McKillip
- Title: In the Forests of Serre
- Title: The Bards of Bone Plain
...

Django will construct a prefetch query under the hood which pretty much equals to:

books = Book.objects.filter(author_id__in=...)

The filter on author_id will automatically be populated by Django and limit the books to the set of authors selected in the first query.

If you want to limit the books queryset further, you can do so using Prefetch objects:

>>> from django.db.models import Prefetch

>>> prefetch_qs = Book.objects.filter(title__startswith="H")
>>> authors = Author.objects.prefetch_related(
...     Prefetch("books", queryset=prefetch_qs)
... )

Aggregating data

All the things above are somewhat basic and something everybody using Django will come across eventually. The next topic about data aggregation and annotating database objects with additional information is something that’s still common, but it may take some time for this to come up in a project.

Let’s `COUNT` again

Let’s start by counting the number of books per author:

>>> from django.db.models import Count

>>> authors = Author.objects.annotate(book_count=Count("books"))
>>> for author in authors:
...     print(f"Author: {author.name}: {author.book_count}")
Author: Jen Wang: 10
Author: Sarah MacLean: 12
Author: Charles Soule: 12
Author: A.S. King: 1
Author: Jesmyn Ward: 1
Author: Victor LaValle: 2
...

And at this point, it’s interesting to start to look at the SQL Django generated:

SELECT
    "literature_author"."id",
    "literature_author"."name",
    COUNT("literature_book"."id") AS "book_count"
FROM "literature_author"
LEFT OUTER JOIN "literature_book"
    ON "literature_author"."id" = "literature_book"."author_id"
GROUP BY
    "literature_author"."id",
    "literature_author"."name"

The critical puzzle piece in this SQL statement is the JOIN between the author and book tables with the COUNT in the SELECT clause. Django shifts the entire work to calculate the sum of books per author to the database.

Let’s `SUM` it up

What we have is already great. Now, let’s look into finding the top five authors with the most votes across all books:

>>> from django.db.models import Sum

>>> authors = Author.objects.annotate(
...     sum_votes=Sum("books__votes")
... ).order_by("-sum_votes")[:5]
>>> for author in authors:
...     print(f"Author: {author.name}: {author.sum_votes} votes")
Author: J.K. Rowling: 10354107 votes
Author: Suzanne Collins: 10270371 votes
Author: Rick Riordan: 5860434 votes
Author: John Green: 5694398 votes
Author: Stephen King: 5181285 votes

Instead of a simple count, we sum up all the votes for all books an author has written. We then sort the authors by the sum of votes in decreasing order and limit the amount of records returned from the database to five.

The SQL for this query is not too different to the previous one. Instead of COUNT there is a SUM function. There is now a ORDER BY column with a LIMIT. All as expected.

SELECT
    "literature_author"."id",
    "literature_author"."name",
    SUM("literature_book"."votes") AS "sum_votes"
FROM "literature_author"
LEFT OUTER JOIN "literature_book"
    ON "literature_author"."id" = "literature_book"."author_id"
GROUP BY
    "literature_author"."id",
    "literature_author"."name"
ORDER BY "sum_votes" DESC
LIMIT 5

Annotating “arbitrary” data

The annotations shown above are not the only thing Django can do. There’s a lot more:

>>> from django.db.models import CharField, Value
>>> from django.db.models.functions import StrIndex, Substr

>>> substr_exp = Substr(
...     "name",
...     1,
...     StrIndex("name", Value(" ")) - Value(1),
...     output_field=CharField(max_length=100),
... )
>>> Author.objects.annotate(
...     first_name=substr_exp
... ).annotate(
...     book_count=Count("books")
... ).order_by("-book_count")
<QuerySet [<Author: Stephen King>, <Author: Peter Meredith>, ...]>

This query will count the books per author, but will also attach the “first name” to each model instance.

If we add a values() queryset method after the first annotate() method we effectively group on the first name for the counting:

>>> Author.objects.annotate(
...     first_name=substr_exp
... ).values(
...     "first_name"
... ).annotate(
...     book_count=Count("books")
... ).order_by("-book_count")
<QuerySet [
    {'first_name': 'Lisa', 'book_count': 56},
    {'first_name': 'David', 'book_count': 53},
    ...
]>

I’m well aware that this is a lot to digest and understand. The Django documentation has a whole chapter on aggregations that I can highly recommend to read through and have a look at whenever you need to deal with aggregations and annotations, because I haven’t even covered half of it.

Top-k selects

The last thing I want to cover is something that’s generally not easy to express in SQL and also computational wise rather heavy. It’s about selecting the top-k elements for something else.

The typical approach to this problem, across all databases, is the use of subqueries. A subquery is a full SQL query that will run as part of a “main” database query. Some database have additional capabilities, though, that make top-k selection relatively easy. On PostgreSQL, that would be a LATERAL JOIN.

Let’s start by first selecting the top three books by the number of votes per author, and then the top three books by votes per genre.

Top three by author

When we discussed the prefetch_related method before, we already looked into the Prefetch object. We will leverage that here. Let’s build this query piece by piece.

We want authors and a set of books that belong to each author. For now, the books_qs won’t be doing much:

>>> books_qs = Book.objects.all()
>>> authors = Author.objects.prefetch_related(
...     Prefetch("books", queryset=books_qs)
... )

With this, we will prefetch all books that belong to an author. As a next step, let’s sort the list of books by votes in descending order:

>>> books_qs = Book.objects.order_by("-votes")

The last step is to limit the number of books to k. The approach everybody takes will be this:

>>> books_qs = Book.objects.order_by("-votes")[:3]

However, this will cause Django to raise an AssertionError:

>>> list(
...     Author.objects.prefetch_related(
...         Prefetch("books", queryset=books_qs)
...     )
... )
Traceback (most recent call last):
...
AssertionError: Cannot filter a query once a slice has been taken.

If one thinks about that, Django will take the book_qs and apply a filter() call on the author_id to limit the books to the list of authors selected before. So, we need another approach. There’s already a feature request ticket on the Django bug tracker.

Instead, we need to look into Subquery and OuterRef.

First, we’ll select the primary key of the top k books while filtering on an outer reference to an author_id. This queryset will not work on its own. It will only ever work in the context of a subquery that knows about an author_id.

We then put that book_sub_qs into a subquery. With that, the inner query “knows” about the author_id. If we were to iterate over books_qs, we’d get a list of books, the first book having the most votes, and not more than 3 books per author:

>>> from django.db.models import OuterRef, Subquery

>>> book_sub_qs = Book.objects.filter(
...     author_id=OuterRef("author_id")
... ).order_by("-votes").values_list("id", flat=True)[:3]
>>> books_qs = Book.objects.filter(
...     pk__in=Subquery(book_sub_qs)
... ).order_by("-votes")

With that, we can now go back to our Prefetch() object and combine authors and books:

>>> book_sub_qs = Book.objects.filter(
...     author_id=OuterRef("author_id")
... ).order_by("-votes").values_list("id", flat=True)[:3]
>>> books_qs = Book.objects.filter(
...     pk__in=Subquery(book_sub_qs)
... ).order_by("-votes")
>>> authors = Author.objects.prefetch_related(
...     Prefetch("books", queryset=books_qs)
... )
>>> for author in authors:
...     print(f"Author: {author.name}")
...     for book in author.books.all():
...         print(f"  - {book.title}")
Author: Doris Kearns Goodwin
  - Team of Rivals: The Political Genius of Abraham Lincoln
  - No Ordinary Time: Franklin and Eleanor Roosevelt: The Home Front in World War II
  - The Bully Pulpit: Theodore Roosevelt, William Howard Taft, and the Golden Age of Journalism
Author: Nathaniel Philbrick
  - In the Heart of the Sea: The Tragedy of the Whaleship Essex
  - Mayflower: A Story of Courage, Community, and War
  - The Last Stand: Custer, Sitting Bull, and the Battle of the Little Bighorn
Author: Terry Pratchett
  - A Blink of the Screen: Collected Shorter Fiction
Author: Timothy B. Tyson
  - The Blood of Emmett Till
Author: Harper Lee
  - To Kill a Mockingbird
  - Go Set a Watchman
Author: Mark Kurlansky
  - Salt: A World History
  - Cod: A Biography of the Fish that Changed the World
  - The Basque History of the World: The Story of a Nation
Author: James Hilton
  - Random Harvest
...

SELECT
    "literature_book"."id",
    "literature_book"."title",
    "literature_book"."author_id",
    "literature_book"."votes"
FROM "literature_book"
WHERE
    "literature_book"."id" IN (
        SELECT
            U0."id"
        FROM "literature_book" U0
        WHERE
            U0."author_id" = "literature_book"."author_id"
        ORDER BY
            U0."votes" DESC
        LIMIT 3
    )
    AND "literature_book"."author_id" IN (7, 16, 25, 40, ..., 18885860)
ORDER BY
    "literature_book"."votes" DESC

Top three by genre

We can use the very same pattern we have above for top-k by an author when we want to select the top-k by genre.

The notable difference between the Book-Author and Book-Genre relationship is that one of them is a many-to-one (Book-Author) and the other one is many-to-many (Book-Genre).

Due to the relationship being a many-to-many one, we need to make one change to remove duplicate books — the book_qs gains a distinct() call.

>>> book_sub_qs = Book.objects.filter(
...     genres=OuterRef("genres")
... ).order_by("-votes").values_list("pk", flat=True)[:3]
>>> book_qs = Book.objects.distinct().filter(
...     pk__in=Subquery(book_sub_qs)
... ).order_by("-votes")
>>> genres = Genre.objects.prefetch_related(
...     Prefetch("books", queryset=book_qs)
... )
>>> for genre in genres:
...     print(f"Genre: {genre.name}")
...     for book in genre.books.all():
...         print(f"  - {book.title}")
Genre: Fiction
  - Harry Potter and the Sorcerer's Stone
  - The Hunger Games
  - To Kill a Mockingbird
Genre: Romance
  - The Hunger Games
  - The Fault in Our Stars
  - Pride and Prejudice
Genre: Cultural
  - The Poisonwood Bible
  - Things Fall Apart
  - The Paris Wife
Genre: Australia
  - The Forgotten Garden
  - The Dry
  - Force of Nature
...

SELECT DISTINCT
    "literature_book_genres"."genre_id" AS "_prefetch_related_val_genre_id",
    "literature_book"."id",
    "literature_book"."title",
    "literature_book"."author_id",
    "literature_book"."votes"
FROM "literature_book"
INNER JOIN "literature_book_genres"
    ON "literature_book"."id" = "literature_book_genres"."book_id"
INNER JOIN "literature_book_genres" T4
    ON "literature_book"."id" = T4."book_id"
WHERE
    "literature_book"."id" IN (
        SELECT
            U0."id"
        FROM "literature_book" U0
        INNER JOIN "literature_book_genres" U1
            ON U0."id" = U1."book_id"
        WHERE
            U1."genre_id" = "literature_book_genres"."genre_id"
        ORDER BY
            U0."votes" DESC
        LIMIT 3
    )
    AND T4."genre_id" IN (1, 2, ..., 411, 412)
ORDER BY
    "literature_book"."votes" DESC

Setting up a private & secure IRC channel

2018-11-03T00:00:00+01:00

The Internet Relay Chat (IRC) was create in 1988 by Jarkko Oikarinen. And it’s stil around and used. At least today there are a ton of configuration options one has at their disposal to configure their nick, channels, or their client.

In this post I’ll show you how you can create and register a private and secured IRC channel that restricts who has access and can post to it.

This post assumes that you have registered and authenticated at the IRC network.

First things first, you need to create a channel. Let’s call it #project:

/join #project

You then want to make sure that channel is registered to yourself:

/msg ChanServ REGISTER #project

You also want to enable the ChanServ guard so you can reclaim access and have it keep the channel in the state you want it to be, if you ever are kicked out:

/msg ChanServ SET #project GUARD ON

Now, you need to define who has full access to that channel. That means they could add others who have full permissions. And they’d be able to kick you out and revoke your access:

/msg ChanServ FLAGS #project USERNAME +AFRefiorstv

Next up, you can hide the channel from channel lists and user profiles. We’ll also enable a verbose logging mode that shows who changed what:

/msg ChanServ SET #project PRIVATE ON
/msg ChanServ SET #project VERBOSE ON

We’ll then set a couple of channel modes. These are — in order — SSL only, no color, members can invite, invite only, no external messages, private, block unidentified, secret:

/msg ChanServ SET #project MLOCK +Scginprs

At this point, nobody else than the people you added above, will have access. To change that you’ll need to add them to the invite list. Replace the USERNAME part only. The $a: indicates it’s an account and needs to be part of this statement:

/mode #project +I $a:USERNAME

Lastly, you may want to check the invite list again:

/mode #project +I

Simple TOTP Bash Script

2018-08-10T00:00:00+02:00

Using Two Factor Authentication (2FA) for services is a good idea. Most services will require you to have an Android or iOS smartphone and use Google Authenticator or similar apps to generate TOTP codes.

That’s fine if you have a smartphone. If you don’t have one you can often use SMS tokens instead. But SMS for 2FA is not recommended anymore. Also, what do you do if you don’t have a smartphone or mobile phone at all? I know of at least 2 people who need to use 2FA and don’t have a smartphone.

First thing I suggested to them: use Authy, there’s a Chrome plugin you can use instead of a smartphone. But it turns out, they seem to require you to have a smartphone. I guess for backup or security reasons.

Since both people have access to a Linux system, I figured I might as well find a Linux client that allows them to generate TOTPs. But all tools or scripts I could find seemed bloated. Until Emma Delescolle pointed me to oath-toolkit. Thanks!

With the included oathtool one can generate a TOTP:

$ oathtool --base32 --totp ONSWG4TFOQYTEMZUGU3DOOBZ
785263

Great. Now I only needed a way to securely store the secret. I figured a symmetrically encrypted file using GnuPG would probably do just fine:

$ echo ONSWG4TFOQYTEMZUGU3DOOBZ | gpg --symmetric --out ~/.secret-file

And loading that file isn’t hard either:

$ secret="$(gpg < ~/.secret-file)"
$ oathtool --base32 --totp "$secret"
785263

Great. Now I only needed to tie those things together in a small script that lets me add new secrets for various services and gives me a TOTP back when I need one:

#!/bin/bash

# Copyright (c) 2018, info AT markusholtermann DOT eu
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of the <organization> nor the
#       names of its contributors may be used to endorse or promote products
#       derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

set -e
set -o pipefail

SOURCE_DIR=~/.config/2fa

function init() {
    mkdir -p $SOURCE_DIR
    chmod 0700 $SOURCE_DIR

    if ! hash gpg 2>/dev/null ; then
        echo "Please ensure that GnuPG is installed!"
        exit 1
    fi
    if ! hash oathtool 2>/dev/null ; then
        echo "Please ensure that oathtool is installed!"
        exit 2
    fi
}

function add_key() {
    echo "Adding a new key"
    if [ "x$1" != "x" ] ; then
        identifier=$1
    else
        echo "What's the identifier?"
        read -r identifier
    fi
    echo "What's the secret?"
    read -r secret
    echo "$secret" | gpg --quiet --symmetric --out "$SOURCE_DIR/$identifier"
}

function get_totp() {
    if [ "x$1" != "x" ] ; then
        identifier=$1
    else
        echo "What's the identifier?"
        read -r identifier
    fi
    secret="$(gpg --quiet < "$SOURCE_DIR/$identifier")"
    oathtool --base32 --totp "$secret"
}

function list() {
    ls -1 "$SOURCE_DIR"
}

function help() {
    echo "Setup a new TOTP account or generate a new TOTP token from an existing account."
    echo
    echo "Usage: totp.sh [--add|--list|--totp] [IDENTIFIER]"
    echo
    echo -e "--add     Will ask for an identifier (i.e. 'google', 'slack', ...) and\\n" \
            "         then for the secret provided by the service provider."
    echo -e "--list    Will list all available identifiers."
    echo -e "--totp    Will ask for an identifier (i.e. 'google', 'slack', ...) and\\n" \
            "         then return the TOTP token."
}

init

case $1 in
    --add)
        add_key "$2"
        ;;
    --list)
        list
        ;;
    --totp)
        get_totp "$2"
        ;;
    *)
        help
esac

How do I use the tool? Let’s say we want to add 2FA to our Google account. You copy the 2FA key from Google’s website and add it:

$ totp.sh --add google
Adding a new key
What's the secret?
abcd efgh 1234 ijkl mnop 5678 90qr stuv

At this point you’ll be asked for a password and confirmation to secure the secret. Repeat the above with different identifiers for various services you want.

You can then list all available services / identifiers:

$ totp.sh --list
google

And lastly, you can get the TOTP back:

$ totp.sh --totp google
612027

Oh, I Found a Security Issue

2017-11-22T00:00:00+01:00

Django’s Security Process

When you’ve used Django and follow one of our email lists you will likely have seen an announcement of a Django security release much like this one from April 4th this year:

Date: Tue, 4 Apr 2017 08:31:25 -0700 (PDT)

From: Tim Graham <…..@gmail.com>

To: django-announce <django-announce@googlegroups.com>

Subject: [django-announce] Django security releases issued: 1.10.7, 1.9.13, and 1.8.18

Today the Django team issued 1.10.7, 1.9.13, and 1.8.18 as part of our security process. These releases address two security issues, and we encourage all users to upgrade as soon as possible: https://www.djangoproject.com/weblog/2017/apr/04/security-releases/

As a reminder, we ask that potential security issues be reported via private email to security@djangoproject.com and not via Django’s Trac instance or the django-developers list. Please see https://www.djangoproject.com/security for further information.

This post is how we, the Django team, get to the state where we are able to send out such an announcement.

Reporting the issue

The process usually starts with somebody reporting a security issue to the security team. The preferred way to contact us is security@djangoproject.com.

While we can’t give a guaranteed response time you can expect an acknowledgment within 24 hours. Please do not hesitate to send a follow-up email if you do not get a response within 2-3 days. Depending on the severity of the issue maybe even earlier.

Apart from that, do not under any circumstances, communicate your findings to anybody else. The Django security team will take care of the public announcement later.

Assessing the reported issue

Once the security team got your email, it will assess the issue. You can expect to be included in the discussion. This allows you to make sure your reported issue is addressed and that the security team is not missing the point.

Fixing the issue

The security team will also ask you if you want to supply a patch or if it should fix the issue itself.

Fixing the issue is often not hard. We are allowed to sacrifice backward compatibility in case of security issues. On top of that, history has shown that many patches are only a few lines of code checking just ONE MORE THING

Confirming the fix

At some point, there is a patch available.

If you wrote the patch the security team will review it.

If somebody from the security team wrote the patch, we kindly ask you to review the patch to ensure it actually fixes the issue you reported and we don’t miss something.

This is a very important part of the process, even with multiple people confirming that the patch works, there is still the possibility of missing something.

Pre-notification

Once the patch is “ready”, to be committed to the release branches, we send out pre-notification emails to some people and organizations. We will not make this list public. It mostly consists of distributors and organizations with lots of personal data.

Aside from the detailed, private notification, we announce the upcoming security release on the django-announce mailing list with a time frame during which the release will be made available.

Release & Announcement

About a week after the pre-notification there’s the security release.

At that point, we will commit the security patches publicly. From that point on we consider the security issue to be disclosed publicly.

After that, somebody from the release team will cut the corresponding Django releases.

We will then publish a blog post with some details on the security issue. We will not publish exploits.

We will publish Common Vulnerability and Exposure numbers — or CVEs — so that you can check against your downstream vendor’s CVE database to see if they patched the vulnerability.

Furthermore, we will announce the releases to the django-announce and oss- security OpenWall mailing list.

Implementing these steps yourself

As you can see, there’s a lot of process on handling security issues properly. At least we think we handle them properly.

How can you apply this to your library and make it more secure?

Setup reporting channel

First and foremost, communicate a way people can securely reach out to you to report a security issue.

A sentence like “If you think you found a security issue please get in touch via FOO” goes a long way. And PLEASE, including the actual email address there goes a long way.

Don’t let somebody who wants to help you find a communication channel themselves. Tell them that even a suspicion of a security issue is worth reporting privately. Even if it turns out to not be an issue.

Monitor reporting channel

I hope it goes without saying that you have to monitor whatever communication channel you choose.

If you don’t have the time to monitor 24/7, which I believe nobody in the room has, add something like “you should receive an acknowledgment within 7 days” or whatever is reasonable FOR YOU! This gives the reporter the information that they may need to wait for a short time.

Just because YOU develop an open source library in your free time doesn’t mean that you need to be around fixing it 24/7. Have a life besides it, please.

Fix the issue

Find a way to fix the bug. Communicate with the reporter. Once you think you are good with you have, ask the reporter to confirm your patch. BEFORE you make it public!

Release & Announce

Commit and release a new version of your library or program. Inform people about the release. Communicate where you are going to release such information at a prominent place so people can subscribe to updates.

My suggestion is to generally report to the oss-security OpenWall list.

Learn from it

For your own sake, accept the fact that you had a security issue in your software and don’t stress out over it. We’re all humans. We all make mistakes! Learn from it and try to not make this mistake again.

For what it’s worth, Django received over 60 CVEs since its release. And there’s a whole bunch of people reviewing code.

OWASP Top 10

One thing I want to mention here is the Open Web Application Security Project or OWASP. They provide a Top 10 of security issue classifications that people should care about. It’s often considered a “standard” but there are some loud voices that claim OWASP is not focusing on the real issues but tries to tackle them from a higher or business level. Have a look at it and decide for yourself.

Resources

The PyCon Canada illustration and logo are licensend under the MIT license.
Slides

Thoughts About Normal and Abnormal Data

2017-10-27T00:00:00+02:00

A lot of our data lives in databases. A lot of those databases are relational databases. They are called relational due to the way how objects refer to each other. But just because having references between certain records is “normal” and “what you should do” it doesn’t mean it’s the best choice.

This post is not about files. Neither in a file room nor your spreadsheet or PDF document. Everyone has their own system of how they store their files. And I think that’s a good thing.

This post is also not about document stores or document databases like MongoDB or CouchDB. In my opinion all these data stores have their reason to exist. It’s just not something I am looking into here.

What I am going to cover are relational databases, such as PostgreSQL, MySQL and a whole lot more. The key here lies on relational databases.

In order to understand what normal or abnormal data is, I need to talk about some database theory.

Database Theory

Let’s say we want to store a few person names, which planets they are from and what gender they identify with.

If you were to put that information in a database table this might very well look a bit like this:

Name	Home planet	Gender
Padmé	Naboo	Female
Luke	Tatooine	Male
Leia	Alderaan, Naboo	Female

There is a problem with this format, though. You need to store two planet names in a single cell. This is certainly possible but will eventually lead to problems. For example filtering for all people from Naboo can get tricky. If you are using PostgreSQL you can put that into an array field that can get around a few of these issues but not all.

First Normal Form

This is the time where in relational database theory the so called First Normal Form or 1NF is introduced.

First mentioned by Edgar Codd in 1971, there are a few criteria a table needs to fulfill for it to be considered in First Normal Form. Firstly, each cell must have atomic or indivisible entities. Secondly, each row needs to be uniquely identifiable.

When we apply these two rules, our table can look like this:

PersonID	Name	Home planet	Gender
1	Padmé	Naboo	Female
2	Luke	Tatooine	Male
3	Leia	Alderaan	Female
3	Leia	Naboo	Female

We introduced a PersonID that we will start to use to reference a person. We also changed the combined “Alderaan, Naboo” value into two distinct rows. However, now the PersonID itself is not uniquely identifying anymore. As such, the key to identify a person now is a combination of PersonID and Home Planet.

This is all great and we’re happy now, but we can run into a problem over time while using our data. This table suffers from Update Anomalies. Let’s look at a subset at the previous table:

PersonID	Name	Home planet	Gender
3	Leia	Alderaan	Female
3	Leia	Naboo	Male

If Leia were to change her gender and would identify herself as male, we would need to update every row in the table with the new gender. If we don’t update all rows but instead only update some, we end up with inconsistent data. And that is something you certainly don’t want to have happen to your production data.

Second Normal Form

To remedy this issue there exists a Second Normal Form or 2NF.

The Second Normal Form was defined by Edgar Codd in 1971 as well. In addition to the criteria from the First Normal Form it requires that all non-key attributes are fully functional dependent on the primary key. Now, what does that even mean? Let’s look at the table again:

PersonID	Name	Home planet	Gender
1	Padmé	Naboo	Female
2	Luke	Tatooine	Male
3	Leia	Alderaan	Female
3	Leia	Naboo	Female

The primary key is the combination of PersonID and Home Planet. Because that’s what uniquely identifies are row.

A person’s name as well as their gender are each non-key attributes. They both are also functional dependent on the primary key. But don’t depend on the FULL primary key, but only part thereof. Name and gender only depend on the PersonID, so to say. The Home Planet doesn’t influence either. As such, this table violate the Second Normal Form.

We can solve this violation, though:

PersonID	Name	Gender
1	Padmé	Female
2	Luke	Male
3	Leia	Female

PersonID	Planet name
1	Naboo
2	Tatooine
3	Alderaan
3	Naboo

If we move the Planet Name into its own table and only reference the PersonID, we comply with the Second Normal Form. So far so good.

But now we want to extend our table of planet names. We want to add the planet Dagobah to our list. However, trying to do so will give us some trouble if we don’t have at least one person in our person table that is considering that planet their home planet. If we were to insert Dagobah into our planet table we couldn’t. There’s no person in our person table that would have Dagobah as their home planet:

PersonID	Planet name
1	Naboo
2	Tatooine
3	Alderaan
3	Naboo
???	Dagobah

While this might not be an issue for fixed-data scenarios, it certainly is an issue in most situations these days.

There’s also another problem you might have already thought of: when we have trouble inserting data, we might as well have issues deleting it. And you’re right there.

Deletion Anomalies can happen as well. If we were to delete Luke from our persons table, then Tatooine wouldn’t have a person anymore that it belongs to. We’d thus need to delete that planet from the planet table as well:

PersonID	Name	Gender
1	Padmé	Female
3	Leia	Female

PersonID	Planet name
1	Naboo
???	Tatooine
3	Alderaan
3	Naboo

That’s kind of unfortunate, isn’t it? Let’s remedy this issue. You may already guess, when there is a First and Second Normal Form there probably is a Third Normal Form as well.

Third Normal Form

And you’re right about that! Like the previous two, the Third Normal Form or 3NF was defined by Edgar Codd in 1971. Simply put, in addition to the criteria for from the First and Second Normal Forms, it requires that all non-key attributes are dependent on the primary key, and only the primary key.

Looking at our table schema now, we have the person table with the primary key PersonID. Name and gender only depend on the PersonID. We could have more persons with the same name and different genders:

PersonID	Name	Gender
1	Padmé	Female
2	Luke	Male
3	Leia	Female

Likewise for the planets, although multiple planets with the same name are somewhat unrealistic. The name and the amount of surface water only depend on the PlanetID primary key:

PersonID	Planet name
1	Naboo
???	Tatooine
3	Alderaan
3	Naboo

New to the party here is the table combining PersonID and PlanetID. Those two fields also represent the table’s primary key. The table represents the m-to-n relationships between persons and planets. Thus preventing insert and deletion anomalies:

PersonID	PlanetID
1	10
2	11
3	10
3	12

Tip

Database normalization is great!

A normalized database can solves the issues of insert, update and deletion anomalies I pointed out. But should you always normalize your database? Is a fully normalized database always a good idea?

Another Example

Let’s have a look at another example, a wiki. Imagine a simple wiki with pages where each page can have an arbitrary number of revisions. From a relational database perspective this could look a bit like this:

Revision

RevisionID
PageID
Text
Date

On the one hand side we have a Page object that identifies a single page. A page has a unique ID that we use internally to refer to it. A page also has a name attribute we use for displaying. And there is a slug attribute that also uniquely identifies a page object, very much like the PageID, but it is “human readable” and may change together with the name.

On the other side there is a Revision object. That object is used to store different versions of the text of a wiki page. The date attribute identifies when the revision was created. As such, the revision object with the newest date for a page represents the last revision.

Given that schema I would want 2 tasks to be solved:

Task 1: Fetch a single page and its current revision
Task 2: Fetch all page titles and the date of their current revision

As the first task I want to fetch a single page and the corresponding last or current revision. This is a quite common scenario. A user navigates to your wiki and you want to show the recent version of a particular page. How do you do that given the database structure I just showed?

The second task is somewhat less common but still useful; fetch all page titles and the date of their current or last revision. This could happen when a user opens your wiki’s sitemap or your index and you want to list all pages with their last revision data.

Well, for the first task, this is probably the simplest SQL query you could find that works across different databases. Using common table expressions or window functions if you are on PostgreSQL could be beneficial. But this is a pure SQL standard:

SELECT
  *
FROM page
INNER JOIN revision
  ON
    page.page_id = revision.page_id
WHERE
  page.slug = 'some-slug'
ORDER BY
  revision.date DESC
LIMIT 1;

You filter for the slug of the page, join your page and revision tables on the page id, sort by the revision date in descending order, and select the top 1 result.

Similarly for the second task we are fetching all pages with their most recent revision date. Adhering to what pretty much all relational databases understand:

SELECT
  page.name, last_revs.date
FROM page
INNER JOIN (
  SELECT
    revision.page_id, MAX(revision.date) date
  FROM revision
  GROUP BY
    revision.page_id
) last_revs
  ON
    page.page_id = last_revs.page_id;

We’re selecting the maximum revision date and corresponding page id for each page from the revisions table, join the result with the pages on the page id, and return the page’s name and last revision date.

Now with those two queries, let’s look at some numbers and statistics.

Benchmarking

These scientifically absolutely unproven tests ran on an Intel i7, 8GB RAM and PostgreSQL 9.6.5. The database contained 10 thousand pages and 6 million revisions in total. Randomly created with an overall average distribution.

For my “benchmark” I ran the first query with 10 concurrent connections. Each connection fetched a fixed random subset of 1000 pages. Each page was fetched 10 times in a row. That comes down to a total of 100 thousand queries.

The second task I decided to just run the same query 10 times to get an idea of the timing. The second query is also not dependent on a particular page but on the full table.

For the first task, on average, the first call for a page took almost 20 milliseconds, whereas the subsequent calls were about 2.5 to 3 milliseconds. That’s easily explainable by cache hits for subsequent calls.

For the second task the query run time varied from about 3.5 to 7 seconds. There also wasn’t any useful caching happening. The cache misses are certainly something one can look at in details, but that’s beyond the talk and this post.

What I instead want to do is, restructure and denormalize the database structure a bit.

Denormalizing our Database

The only change compared to the previous schema is the added LastRevision column on a page which references, as you might have guessed, to the currently active revision for a page.

Revision

RevisionID
PageID
Text
Date

Obviously, our SQL will have to change.

Instead of doing ordering in descending order by date and then limiting the result set to a single record, we can now directly access to last revision:

SELECT
  *
FROM page
INNER JOIN revision
  ON
    page.last_revision_id = revision.revision_id
WHERE
  page.slug = 'some-slug';

For the second task we don’t need any subquery anymore which will make the query significantly more efficient:

SELECT
  page.name, revision.date
FROM page
INNER JOIN revision
  ON
    page.last_revision_id = revision.revision_id;

The picture the numbers for the first task draw looks quite similar. Except that the query time for the first call went from 20 milliseconds to just above 0.3 milliseconds. That is about two orders of magnitude faster for the first call.

The picture for the second task, though, is something I really like. Not only dropped the absolute query time for all calls, but subsequent calls of the same query heavily hit the cache.

While the average query time was around 6 seconds before, using the denormalized schema the query run time is about a 10th of that for the first call. Subsequent queries are about 200 times faster than the original average.

Conclusion

I guess what I want to show, normalizing your database is a good idea to prevent anomalies. It’s also a good idea to prevent data duplication and to keep your data consistent.

But what I also want to show you, there’s a time at which you should rethink your database design and may denormalize your database a bit. The important part is that you don’t start with that right in the beginning when you setup your project. You will not know how your project evolves and denormalization adds another layer of complexity. When you know your project, and when your database reaches a certain size, I suggest you look at the single bottleneck at a time.

Check what’s important for your users. And start there to optimize. But only go as far with denormalization as you feel comfortable. Because every layer of denormalization you add, needs to be maintained.

Resources

Slides

Security and Ease-of-Use With Django Templates

2017-06-24T00:00:00+02:00

I engaged in a bit of a Twitter discussion that started when Samuel Spencer showed a Django package to modify page templates through the Django admin.

This blog post is not intended to blame him for developing the package. This blog post is not intended to discourage anybody from using that package. This blog post is intended to educate about possibly risks that arise from using that package or, in general, blindly trust and run user supplied content which is something this package does to some degree.

TL;DR: We live in a dangerous world. Security is hard. Don’t trust anything.

Ok. Let’s me a bit more serious and specific. Let’s weigh the pros and cons. Let’s see what’s actually at stake.

The Deployment

Deploying Django applications is more complicated than your typical WordPress site. At least when you get started with Django. There are dozens of articles, conference talks, and mailing list discussions on that topic. Google it if you want specifics. In essence, deploying a Django project is not simply done by copying some files to a document root on a web server as it is common with PHP. You’ll probably use an application server like gunicorn and uwsgi. Or you figured out how to use mod_wsgi in the Apache HTTP server which then works similarly to PHP yet completely different. But when you managed to get all changes to your server, you still need to tell the application server to reload and restart your application. And this is a good thing ™. You’ll see why that is later.

When You Screw Up

Let’s assume you have this magic way to modify the HTML template files for your Django project in the browser. That’s great, as you can now make some quick fix online without the need to commit code and trigger your deployment pipeline. So it’s perfectly suitable to perform those changes from your smartphone given a responsive Django admin. Wrong!

Guess what happens when you modify your base.html file and have a syntax error in there. The potentially harmless case is a broken <a> tag, a messed up a list definition or whatnot. Or you accidentally remove the keywords from the <head> section so your SEO ranking goes downhill. None of these things actually are a real issue for the matter of this post. Your site will still be available. Users can still access your page and read its content. Modern browsers will cope with all kinds of broken HTML structures. So, all good, right? Wrong!

What happens if you accidentally removed some escaping from some Django template language variables, such as escapejs inside a JavaScript block? Well, the good thing, your site will still “work” and is not going to throw a HTTP 500 Internal Server Error. Probably, at least. Lucky you. But maybe not. This is one of those subtle issues you will only know about once somebody exploited an issue with unescaped JavaScript. Which is now possible due to to missing escapejs, remember?

I think I’d prefer to completely screw up my Django template syntax. At least the issue will be brutal when the entire page won’t load. You should notice that, right? Depends! Only if you have proper monitoring or error reporting.

But that’s not all. That’s just the start. Depending on which template file you just broke, you might as as well broke your Django admin template. Congratulations, you’re now forced to figure out a way to revert that. Likely without any form of web UI. Oh, and of course, you’re on your phone in the middle of nowhere with crappy internet such that an SSH connection cracks down every 5 minutes. Sucks for you, I guess.

Something Else Is Insecure

Given the example from the introduction — a view in the Django admin that lets you modify the template files — what happens if there’s a privilege escalation in the Django admin that lets arbitrary users modify those templates. We’ll, that unfortunate! (On that note, if you encounter or think you encountered a security issue in Django, please reach out to security@djangoproject.com and do not report the issue through the issue tracker).

You’re pretty much out of luck here until you there’s a patched Django release you can use. But in the meantime, users (alright, let’s start calling those users attackers now) attackers will be able to exploit the privilege escalation and have all kinds of fun. Starting with one of those “you’ve been hacked” memes, over a simple “breaking the page” by messing with the Django template language syntax over sneaking in some evil JavaScript that performs XSS attacks against you and your users. Of course nobody is going to notice the latter. Thus you’re leaking more and more data to the attackers.

Now, all previous issues assume an attacker only touches the HTML template files. While that is bad enough, slightly more subtle changes could be introduced by exploiting an issue in the online editor that would allow an attacker to modify arbitrary files. Like, the JavaScript files shipped to a user. But again, this would manifest itself instantly. The “fun” part begins when an attack would be able to view (and potentially write) Python files belonging to your project. I mean, what could possibly go wrong when Django’s SECRET_KEY or database credentials are leaked, right? Apart from being able to connect to your database they could also change the Python code and essentially do whatever they want. Well, almost. Because most (all?) Python web application servers need to be told to reload the web application and thus actually see those changes. In order to do that an attacker would thus need to restart the application server. And that’s not that easily possible unless you can already run arbitrary code. On the other hand with PHP, changes are reflected instantly (at least as far as I remember. I’ve not done any PHP development for a while).

When To Use Such Features

Well, if you want to err on the side of caution: never.

If you do use those tools, make sure you’ve come up with a threat model and risk model. If they say the usability and user experience outweighs the risk and threat, sure, go ahead. If they say it’s too risky, well, don’t use such things.

On a higher level you can probably reduce the decision making to this: Is your site on the internet, don’t use such features. Is it company internal, e.g. behind a VPN and only a limited set of users have access to it, you’re probably fine. A company usually has bigger problems when there’s a rough admin that starts XSSing its colleagues. Is it your own tool you use for yourself only and nobody else has access to it, sure, go ahead, use such tools if they make your life easier.

In the end it comes down to the Security, Functionality and Ease-of-Use triad where you can only ever actually achieve 2 of them.

Closing Remarks

These are some thoughts I had on the initially linked tweet. I wanted to write them down to not forget about them. There’s probably a lot more to consider and at stake that I didn’t mention. I’m more than welcome to continue the discussion on this topic in said Twitter thread.

To Index Or Not, That’s Not The Questions

2017-04-05T00:00:00+02:00

Some Background Theory

As databases are used to store more and more information every day, these are also a key component in every Django project. Thus it’s important to understand how they work.

Of course I can’t explain all the details about all the different databases you can use with Django. Not only, because I don’t know all of that, but also because that would make for a hell of a talk. Or probably an entire conference.

The only thing about I want to say about the theoretical background on databases is, there’s something called “relational algebra”. With that every SELECT statement you could possibly come up with can be expressed. It’s mathematically proven.

How Database Lookups Work

Instead, let’s start with how lookups in databases work. Because that is what we do most the time.

Assume we have this database table of names of people and their corresponding age when they first started programming.

Now we want to select every persons that started at age 19.

We can express that with a SQL query:

SELECT * FROM people WHERE age = 19

Now, how can we find each person matching that query?

Table Scan Lookup

Well, that’s fairly easy. We just look at each row in the table, check if the condition applies, and return the row if it does.

This is called a “Full Table Scan”

So far so good. We have 7 rows here. Thus we look at 7 rows. That’s a small number of rows, so the query is rather fast.

But imagine you have like 100 thousand, 100 million, 100 billion or even more rows. Iterating over every single row can be time consuming. And that’s not something we want or something we can afford. We want something that provides guaranteed timing to find a particular row. Independent of the number of rows.

This is where indexes join the party.

What Are Indexes?

Indexes give fast access to a single (or multiple) items without much speed reduction for an increasing amount of data. This is also called “Random Access”. You will see in a bit why it’s called that way. But first let us look into the most common index types in modern database systems.

B-Trees / B+ Trees

The most common indexes these days are B-Tree indexes. Or more precisely, B+ Tree indexes.

They are either named after one of their inventors, Rudolf Bayer, or because they are self-balancing. That’s not really clear but also doesn’t really matter. Self-balancing means that the trees have certain timing guarantees: the most interesting for B-Trees it doesn’t matter how big the index is, for any index of the same size the timing is going to be the same. You’ll see that in a bit as well.

As with every tree (both in Computer Science and in nature) you start with a root. The difference between Computer Science and Nature is, in Computer Science we put the root of a tree at the top.

Whatever. Here’s the root note of a B+ Tree of grade 3.

Grade 3 means, this node can have 3 keys. That’s what the 3 boxes at the top of that node are for. The 4 boxes underneath hold pointers to another node or a row in the database table.

Now, let’s say you have the keys 11 and 37 in this node and you don’t have a 3rd key.

Then the leftmost pointer would point to a node with keys smaller than 11. The second pointer would point to a node with keys greater or equal than 11 and smaller than 37. The third pointer would point to a node with keys greater or equal than 37.

Having an index on the age column from the table I showed in the beginning, this could then look like this.

The magic now is what happens with the pointers in this second row of nodes. Each of them points to a single row in the table that has the particular key, which is the age in our example.

But it’s not just this.

You see how the last pointer in each of the bottom nodes points to the next node? This is used for something called “Index Scan”. I’ll come back to that later.

Let’s look at the second row of nodes in a bit more detail.

You can now see, how every pointer from one of those tree nodes points to a single row in the table.

You can also see how these pointers from the tree nodes somewhat randomly point to some rows in the table. That’s why this is called “Random Access”. The database randomly jumps around in the database table.

Random Access Lookup

Let’s refresh our memory with the SQL query we had before.

SELECT * FROM people WHERE age = 19

How does an index now help to find the corresponding row faster?

Well, let’s look at the tree:

It requires 1 step to get from the first node to the second node. And a seconds step from the second node to the row in the database table.

Remember, we had to look through all 7 rows before to see if they matched the database query.

And since there’s no more key in the index with 19, we’re done.

Index Scan

Now, coming back “Index Scan”, assume you want to count the amount of people that started coding when they were in the age range from 5 to 13.

SELECT COUNT(*) FROM people WHERE age BETWEEN 5 AND 19;

SELECT COUNT(*) FROM people WHERE age >= 5 AND age <= 19;

The database will look for the key 5 and will then use the pointer to the next node to look for further keys.

And because all the information that the database needs for the query is right here in the index, the database will not look at the table at all.

Indexes Are Awesome

Let’s have them in Django.

And we already do, actually.

There is

db_index=True which you can set on a model field
index_together=(('name', 'age'),) which you can set in a model’s Meta class
ForeignKey() / OneToOneField() which use indexes for quick lookups of the data in related tables
primary_key=True which Django automatically uses on the AutoField which represents the id column on every model.

This is already great. But this feature set is somewhat limiting. There are not just B+ Tree indexes out there. There are a ton more

2016

Let’s look at 2016.

Marc Tamlyn and I had ideas for indexes. Actually, Marc had some ideas during his contrib.postgres work already. And we had thoughts about APIs. And things we’d like to have in Django. Like, let’s make Django support all the indexes.

But we didn’t have time to implement our ideas!

But we got lucky. Actually, the Django Project got lucky.

Google Summer of Code 2016

Django was, once again, accepted as an organization for Google Summer Of Code. Thank you Google!

For those of you who don’t know what that is: Google pays a Student for 3 months to work on an Open Source project while being mentored by the project’s contributors.

For the most part Tim Graham, but also Marc and I mentored a student Akshesh Doshi tackling a more generic Index support in Django. From writing down a proposal for the API etc. until eventually merging things into Django.

The major outcome of GSoC 2016 is django.db.models.indexes.Index(fields, name) (docs)

It defines the base class of all indexes. You can use them via the indexes option in a model’s Meta class.

For example like this:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=200)

    class Meta:
        indexes = [
            models.Index(
                fields=['name'],
                name='name_idx',
            ),
        ]

This will create a B+ Tree index on the name column of the database table.

Granted, that’s nothing new. That’s what you can do with db_index=True on the name field.

You can of course also define an Index on multiple columns:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=200)
    age = models.PositiveSmallIntegerField()

    class Meta:
        indexes = [
            models.Index(
                fields=['name', 'age'],
                name='name_age_idx',
            ),
        ]

Granted, that’s nothing new either. You can do with index_together already.

But you can now also do this:

from django.contrib.postgres.fields import JSONField
from django.contrib.postgres.indexes import GinIndex
from django.db import models

class Doc(models.Model):
    data = JSONField()

    class Meta:
        indexes = [
            GinIndex(
                fields=['data'],
                name='data_gin',
            ),
        ]

Define a GinIndex. That’s something PostgreSQL specific. But it’s something you could not do before. At least not reliably without too much pain.

A GinIndex can be used to index the key values inside a JSON blob. So you could filter for rows in a table where the key in a JSONB field maps to a particular value. That’s like the “NoSQL 1-0-1”.

Another built-in index type that ships with Django 1.11 is BrinIndex which, simply put, can allow for much faster computation of aggregations. Such as, finding the last time each article was purchased.

And because Indexes are parts of the database schema, they are obviously tracked through migrations. Thus the index is created when you run python manage.py migrate:

BEGIN;
--
-- Create model Doc
--
CREATE TABLE "someapp_doc" (
    "id" serial NOT NULL PRIMARY KEY,
    "data" jsonb NOT NULL);
--
-- Create index data_gin on field(s) data of model doc
--
CREATE INDEX "data_gin" ON "someapp_doc" USING gin ("data");
COMMIT;

Feature ideas

Great.

That’s what is in Django 1.11 which was released yesterday.

But what’s out there to come for Django 2.0?

What’s on the horizon?

What do we want to have eventually?

Functional Indexes

They are useful in all kinds of situations where you don’t want to index on the raw value but e.g. on a variation of it, such as lower-case of a string. I’m already working on that. I’m not quite there yet. I’d love some help from people understanding the expressions API.

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=200)

    class Meta:
        indexes = [
           FuncIndex(
                expression=Lower('name'),
                name='name_lower_idx',
            ),
        ]

`db_index=<IndexClass>`

Using indexes for a single column can be cumbersome as you’ve seen before. Thus, let’s support Index classes as an attribute to db_index.

from django.db import models

class Author(models.Model):
    name = models.CharField(
        max_length=200,
        db_index=HashIndex
    )

`Field.default_index_class`

Having a B+ Tree for some fields doesn’t make sense. As previously shown, there is a GinIndex that’s perfect for JSONField. Why not have a default_index_class per field class that’s being used when db_index=True?

from django.contrib.postgres.fields import JSONField
from django.contrib.postgres.indexes import GinIndex
from django.db import models

# Somewhere in Django’s JSONField implementation:
# JSONField.default_index_class = GinIndex

class Document(models.Model):
    data = JSONField(db_index=True)

Refactor `index_together` and `db_index`

This one is more under the hood than user facing:

It would probably make sense for db_index and index_together to use Model._meta.indexes internally, I could imagine. That’s something to investigate.

`GiSTIndex`

There’s a GiSTIndex in PostgreSQL that can be used for geo-spacial queries, such as “give me all points that have a maximum distance of 10 to a given point”. It’s not in Django 1.11. I don’t know why, but I guess because nobody added it.

Sprints

And speaking about all these features, there are Sprints on Thursday and Friday. I hope you are staying and join us.

Talk to me or anybody else on the core team or any other attendee if you want to work on any of those tickets. Some of those ideas still need tickets in our issue tracker. And some also probably need some discussion on the django-developers mailing list. Now is the time to get these discussions starting if you want to have any of this in Django 2.0.

On that note, keep in mind that Django 2.0 will not support Python 2 anymore!

Resources

Slides

2 Factor Authentication in Django

2016-09-12T00:00:00+02:00

Introduction

2 Factor Authentication — or 2FA in short — is a special version of the general Multi Factor Authentication (MFA) and generally describes the idea that is also followed by Universal 2 Factor Authentication (U2F). It generally consists of something a user possesses and something a user knows.

Disclaimer

I’m not a cryptographer. I’m somebody interested in security and cryptography.

History

Authentication is something we are confronted with every single day. At least in the modern world. For most of us it probably starts the moment we look at our phone in the morning and need to unlock it to get access to the feed reader.

But that’s today, what’s was there before? Maybe decades ago? Maybe centuries ago?

Around 300 BC, the Romans passed so called “watchwords” from to 10th maniple to 9th to the 8th and eventually to the 1st maniple in front of witnesses. The 1st maniple then passed it back to the tribune who in return knew which group of maniples didn’t passed the watchword. The watchwords were used to authenticate people during the night. [1]

Passwords evolved and In the opening days of the Battle of Normandy in 1944, paratroopers of the US used a password — flash — which was presented as a challenge, and answered with the correct response — thunder.

But history shows that not only passwords have been used to authenticate people or messages. The ancient Greek and Spartans are said to have used a scytale for encryption and for authentication. A scytale is a cylinder with parchment wound around it. Only if somebody had a cylinder of the same diameter the receiver could read the message. That was even earlier, around 700 BC.

Today

In todays life, we have lots of 2 factor situations. Think about your Debit or credit card. You posses that card. In order to use it you need to know the related PIN you know. Only if you have both you can withdraw money from an ATM.

Think about logging into your Google, Github or many other major accounts: you can set up your accounts on those services such that somebody logging in is required to provide a PIN along the password. The PIN can for example be generated by Google Authenticator.

But there are many more identifying components that could be used. Think about you fingerprints. Or the irises in your eyes. And don’t forget your voice, though modern computer simulations can imitate it fairly well.

And lastly, there’s still SMS as a second factor out there. However, if you follow the NIST (National Institute of Standards and Technology), they recently published a draft of a Digital Authentication Guideline. In that it says:

Due to the risk that SMS messages may be intercepted or redirected, implementers of new systems SHOULD carefully consider alternative authenticators. If the out of band verification is to be made using a SMS message on a public mobile telephone network, the verifier SHALL verify that the pre-registered telephone number being used is actually associated with a mobile network and not with a VoIP (or other software-based) service. [NIST]

What they are saying is, that you should verify that the receiver of an SMS is the intended person and not somebody else interfering with the public mobile phone network.

If you’re looking into implementing 2FA, I highly recommend to give this document a read before.

The basics of 2 Factor Authentication

Let’s start with some basic, yet command and secure, processes for 2FA

HOTP

The first one, and the basis for the second one, is HOTP or HMAC based one time passwords.

HOTP sits on top of HMAC-SHA1 — which is still secure despite SHA1 being broken — and some bit and byte shifting, called truncation. The full, in-depth explanation is available in RFC 4226.

HOTP is counter based. Each time you need a new one-time password you increase a counter and get a new one-time-password. The key is know to you (or more precisely your HOTP generator) and the verifying party. In the end, the verifier has to make sure that each counter value can only be used once. That’s about it.

The following code is from the django-otp package. Slightly reformatted.

import hashlib
import hmac
import struct

def hotp(key, counter, digits=6):
    msg = struct.pack(b'>Q', counter)
    hs = hmac.new(key, msg, hashlib.sha1).digest()
    hs = list(iter(hs))

    # Truncate
    offset = hs[19] & 0x0f
    bin_code = (
        (hs[offset] & 0x7f) << 24
        | hs[offset + 1] << 16
        | hs[offset + 2] << 8
        | hs[offset + 3]
    )
    return bin_code % pow(10, digits)

TOTP

The second algorithm I pointed out before is TOTP, Time-based one-time passwords.

TOTP effectively works exactly the same as HOTP with one essential difference: the counter is derived from the current time.

What the algorithm does according to RFC RFC 6238, it takes the number of elapsed seconds between now (t) and a start (t0) and divides it by the duration of how long a single token should be valid. That defaults to 30 seconds. You effectively end up with a counter that changes every 30 seconds.

Looking at code, this is about it. Nothing fancy.

import time

def totp(key, t=None, t0=0, steps=30, digits=6):
    t = int(t or time.time())
    counter = (t - int(t0)) // int(steps)
    return hotp(key, counter, digits)

YubiKey, Nitrokey

I’m not going into details about YubiKey, Nitrokey or similar systems.

They are pieces of hardware, similar to USB sticks, that do crypto for you, for example providing HOTP tokens, or other features such as SMIME or GnuPG keys.

Django Integration

Anyway, enough about the basics you got to know before you consider 2 Factor Authentication for your project. How do you actually implement it? Here are 5 code snippets. I’ll implement a TOTP “device”, a view to enter and validate the token as well as a decorator and mixin to require a 2 factor authenticated session for a particular view.

Let us start with the models.py. Looking back at the parameters for the TOTP function, you need a key, a step, the start time and last time a token was used.

In verify_token() we check that the given token is valid by computing the current token and checking the current token was generated after the last one and matches the other token.

# models.py
import binascii
import time

from django.conf import settings
from django.db import models

from .utils import new_key, totp

class TOTPDevice(models.Model):
    # Adapted from django_otp
    user = models.OneToOneField(settings.AUTH_USER_MODEL)
    key = models.CharField(max_length=20, default=new_key)
    step = models.PositiveSmallIntegerField(default=30)
    t0 = models.BigIntegerField(default=0)
    last_t = models.BigIntegerField(default=-1)

    def verify_token(self, other):
        key = binascii.unhexlify(self.key.encode())
        t = time.time()
        token = totp(key, t, self.t0, self.step)
        if t > self.last_t and token == other:
            self.last_t = t
            self.save(update_fields=['last_t'])
            return True
        return False

For the view where we enter the token we need a form that has a single integer field and validates the token against the TOTP device for a given user.

# forms.py
from django import forms
from django.core.exceptions import ValidationError

class OTPTokenForm(forms.Form):
    token = forms.IntegerField(min_value=0, max_value=999999)

    def __init__(self, user, *args, **kwargs):
        self.user = user

    def clean(self):
        if not self.user.totpdevice.verify_token(
                self.cleaned_data['token']):
            raise ValidationError('Invalid token')

The view to enter the token checks if the session is already verified — that’s a key I defined — and redirects to some other URL if it is.

Otherwise it does normal form handling. For valid forms it updates the session and redirects too.

# views.py
from django.conf import settings
from django.contrib.auth.decorators import login_required
from django.shortcuts import redirect, render
from .forms import OTPTokenForm

@login_required
def otp_token_view(request):
    if request.session.get('verified', False):
        return redirect(settings.LOGIN_REDIRECT_URL)
    if request.method == 'POST':
        form = OTPTokenForm(request.user, request.POST)
        if form.is_valid():
            request.session['verified'] = True
            return redirect(settings.LOGIN_REDIRECT_URL)
    else:
        form = OTPTokenForm(request.user)
    return render(
        request, 'otp_token_form.html', {'form': form}
    )

Now for the decorator and mixin. If the user is not authenticated redirect them to the login page. If the user is authenticated and the session has the verified flag, use the view, otherwise redirect to the 2FA view.

Warning

Mind the property behavior of is_authenticated here. That’s new in Django 1.10. If you’re on 1.9 or before you need to make it function calls.

# decorators.py
import functools
from django.conf import settings
from django.shortcuts import redirect

def is_verified(view):
    @functools.wraps(view)
    def wrapper(request, *args, **kwargs):
        if not request.user.is_authenticated:
            return redirect(settings.LOGIN_URL)
        if request.session.get('verified', False):
            return view(request, *args, **kwargs)
        return redirect('otp-token-view')
    return wrapper

class IsVerified(object):
    def dispatch(self, request, *args, **kwargs):
        if not request.user.is_authenticated:
            return redirect(settings.LOGIN_URL)
        if request.session.get('verified', False):
            return super().dispatch(request, *args, **kwargs)
        return redirect('otp-token-view')

Well, and here’s how you finally use the decorator and mixin in views: For the IsVerified mixin, keep in mind that it has to go to the left! Mixins always go left from the base view! This extends the

# views.py
from django.http import HttpResponse
from django.views.generic import View
from .decorators import is_verified
from .mixins import IsVerified

@is_verified
def my_view(request):
    return HttpResponse('This is a 2FA function view')

class MyView(IsVerified, View):
    def get(self, request, *args, **kwargs):
        return HttpResponse('This is a 2FA class view')

Well, that was a lot of boilerplate. And there’s a bunch of stuff you don’t even have yet.

You don’t have a nice key setup.
You don’t have a QRCode view that you can scan with your phone.
You don’t have backup codes in case you ever lose you phone.
You don’t have an integration into Django Admin.

Imagine to write all that yourself for every project you want to use 2 Factor Authentication in. That’s not going to be fun. Hence there are people out there that have done most of the work for you.

django-otp provides the underlying algorithms and required database models. Feel free to use it. django-otp ships with support for email, HOTP, static tokens, TOTP. There are plug-ins for YubiKey and SMS, too. Reminder, don’t use SMS if you can avoid it. There’s also a very simplistic Django admin integration.

Another library, that I co-maintain for a few months now, is django-two- factor-auth. It heavily reuses django-otp but adds a couple of more things to the party. There’s an entire key setup process, including QR code view, backup codes. There’s a separate implementation for Django admin, not tied to django-otp. I’m also working on an advanced admin integration which includes theming and so on. If you’re staying for the sprints and are looking for something to work on, I’m happy to guide you through the process.

What’s the future of 2 Factor Authentication?

Given what I’ve talked about, where can or should you go?

In my opinion, when you deal with sensitive personal information in your projects, you probably want 2 Factor Authentication. When you deal with other people’s money, you probably want 2 Factor Authentication. When you deal with sensitive infrastructure, you probably want 2 Factor Authentication. When you deal with anything that you think might be worth it, you probably want 2 Factor Authentication. When you deal with anything else, why not have 2 Factor Authentication as an opt-in at least?

How you get there is up to you. I showed you a simple way to build the basics yourself. If that’s what you prefer, sure, please use that. If you prefer to rely on other projects, great, do it and report and contribute back as a “thank you”, please.

There’s also the option for Universal 2 Factor Authentication I mentioned in the beginning. Yuriy Ackermann gave a talk about that at Kiwi PyCon 2016.

Why?

Lastly, why would you want to offer 2FA to your customers? Kenneth Reitz recently published a post how 2FA saved a couple of accounts from being taken over.

Resources

Slides
Video

[1]	http://ancienthistory.about.com/library/bl/bl_text_polybius6.htm

[NIST]

https://pages.nist.gov/800-63-3/sp800-63b.html

SSL All The Things In Python

2016-09-10T00:00:00+02:00

Introduction

Following my DjangoCon US 2016 Talk I gave a talk at PyCon Australia in Melbourne and PyCon New Zealand in Dunedin.

Most of the slides from the former talk are the same. The major difference is the replacement of Django specifics with examples for a Python client and server implementation using the ssl standard library package.

How to use SSL in Python

Client Side

The client side of SSL connections is comparably easy as opposed to the server side. That doesn’t mean you can’t easily mess up things and make the entire encryption layer you intend to have useless. But what would security be without some danger, right 😉.

First of all you create a socket. That’s the normal Python socket layer. You then create a default SSL context which has all the best practices of the installed Python version. You can furthermore drop some protocols if you feel like it and your requirements allow for that. Next you wrap the socket into an SSL socket. While you do that, make sure you check the host name! And finally you connect to the server.

import socket, ssl

HOST, PORT = 'example.com', 443

def handle(conn):
    conn.write(b'GET / HTTP/1.1\n')
    print(conn.recv().decode())

def main():
    sock = socket.socket(socket.AF_INET)
    context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1  # optional
    conn = context.wrap_socket(sock, server_hostname=HOST)
    try:
        conn.connect((HOST, PORT))
        handle(conn)
    finally:
        conn.close()

if __name__ == '__main__':
    main()

Server Side

The server side of SSL connections is more complicated and you can mess up much more. But let us start at the beginning.

You create a socket and bind it to an address and port. You, again, create a SSL default context for a server. The first thing you probably want to do is load the key, certificate and all intermediate certificates. You can put all of these into one file in that order. You can also provide them as separate arguments to the load_cert_chain() function. Again, you can exclude some protocols and update other context options based on your requirements. In the example we’re disabling TLS 1.0 and 1.1, because nobody needs them, right 😉. Lastly, you want to set a bunch of ciphers. That list comes from cipherli.st. Finally you start listening on the socket and wrap it with the SSL context for each connection.

import socket, ssl

HOST, PORT, CERT = 'example.com', 443, '/path/to/example.com.pem'

def handle(conn):
  print(conn.recv())
  conn.write(b'HTTP/1.1 200 OK\n\n%s' % conn.getpeername()[0].encode())

def main():
  sock = socket.socket()
  sock.bind((HOST, PORT))
  sock.listen(5)
  context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
  context.load_cert_chain(certfile=CERT)  # 1. key, 2. cert, 3. intermediates
  context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1  # optional
  context.set_ciphers('EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH')
  while True:
    conn = None
    ssock, addr = sock.accept()
    try:
      conn = context.wrap_socket(ssock, server_side=True)
      handle(conn)
    except ssl.SSLError as e:
      print(e)
    finally:
      if conn:
        conn.close()
if __name__ == '__main__':
  main()

Resources

DjangoCon US 2016: SSL All The Things

2016-07-19T00:00:00+02:00

Introduction

Ever since Snowden revealed that the NSA is spying on all of us the urge to more secure systems increased. At least over in Germany and plenty other European countries.

There’s also the risk of public and unencrypted Wi-Fis. You don’t want the owner of the cafe next door or the other people sitting in the cafe be able to wiretap your network traffic.

And some internet service providers even inject their own advertisement in websites you access. That’s scary. And a bad idea from a user’s perspective. Not to mention from a security perspective.

Disclaimer

I’m not a cryptographer. All examples I give here are either to the best of my knowledge or I’ve talked to skilled people I trust and asked them for their knowledge. Which still means there can be errors as we all a humans!

This blog post will not cover everything you could possibly know about SSL and TLS, by all means, that would be enough to fill an entire conference with talks.

Also, SSL 2 and 3 are broken and insecure. Do not use them! Period. TLS 1.0 and 1.1 are discouraged and superseded and should also not be used anymore as well. Unless you really need to support old browsers.

What is SSL / TLS?

SSL stands for Secure Socket Layer. TLS stands for Transport Layer Security. But that’s just words. They are both cryptographic protocols for communication systems most notably networks. Either one provides 2 of the 3 parts of “Information security”:

Confidentiality which is provided through encryption
Integrity which is provided through signing

Availability is the third part in Information Security but needs to be handled on a different layer.

Web Server Configuration

Because this is the blog post for my talk from DjangoCon US 2016 and most things we do with Django these days are website, let us have a look at how to configure web servers to handle SSL.

Apache 2 / httpd

Let’s start with Apache 2 or httpd

<VirtualHost *:443>
    ServerName  example.com
    SSLEngine   on

    # Details at https://cipherli.st/
    SSLCipherSuite         EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
    SSLHonorCipherOrder    on
    SSLProtocol            all -SSLv3
    SSLCertificateFile     /etc/nginx/ssl/example.com.crt
    SSLCertificateKeyFile  /etc/nginx/ssl/example.com.key
    SSLOpenSSLConfCmd      DHParameters   "/etc/nginx/ssl/example.com.dh"
</VirtualHost>

You need to switch on the SSL engine. You want to define which ciphers you want and make sure the connection is going to use the best cipher it can, according to your list, not the client’s.

You also want to ensure to not use SSL and only TLS and maybe go for TLS 1.2 only.

Defining a custom Diffie-Hellman Parameter is also a good idea. That is used for the initial key exchange between a client and a server.

Nginx

And this is the syntax for Nginx. It looks pretty much the same as the one for Apache 2.

host {
    listen       [::]:443 ssl;
    server_name  example.com;

    # Details at https://cipherli.st/
    ssl_ciphers                EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;
    ssl_prefer_server_ciphers  on;
    ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
    ssl_certificate            /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key        /etc/nginx/ssl/example.com.key;
    ssl_dhparam                /etc/nginx/ssl/example.com.dh;
}

What is Let’s Encrypt?

Now that we know how to configure web servers where the heck do we get the certificates from?

And what the heck is “Let’s Encrypt“?

And how does the whole SSL certificate thing work anyway??

In order to answer these questions I need to tell you a bit about how the entire certificate machinery works.

How does SSL work?

Let us start with CAs: CA stands for Certificate Authority. These are “trusted” entities we or our browsers rely on in order to establish a chain of trust

Browsers, email clients, and other programs have a pre-installed set of root certificates. These are provided by Root CAs, CAs that e.g. browser vendors — such as Mozilla and Google — trust. That database of root certificates is the “trust store”.

The Root CAs sign intermediate certificates. Most Root CAs just sign their own intermediate certificates. Some Root CAs though (in the diagram “Root CA 3”) are not in the trust store. So they are not automatically trusted by our browsers.

Trusted Root CAs can cross sign other CA’s intermediate certificates and make certificates signed by “Intermediate CA 3” trusted by a browser.

Which brings me to the next point: the certificate you can buy from well known vendors are not signed by their root certificates but are signed by the intermediate certificates.

As a result, when your browser trusts Root Cert 1, it trusts Intermediate Cert 1, and hence it trusts all certificates signed by Intermediate Cert 1.

If your browser trusts Root Cert 2, it trusts Intermediate Cert 3, and hence it trusts all certificates signed by Intermediate Cert 3.

What is Let’s Encrypt?

It’s a Root CA, that’s not in global trust stores. At least not yet. Their Intermediate certificate is cross-signed by IdenTrust which is in all common trust stores.

But Let’s Encrypt has control over their intermediate Cert, which allows them to sign arbitrary other certificates, which we will use for our servers.

Let’s Encrypt offers an API to get “unlimited” SSL certificates for your domains, free of charge.

By now they issued more than 5 million certificates through their API.

The process during which one can get a certificate is defined as ACME or Automatic Certificate Management Environment.

It’s a fairly simple JSON API with some crypto magic separated in 4 major steps.

In order to use the Let’s Encrypt API one needs three things:

An Account Key — This is most likely an RSA public key pair
A Certificate Key — That’s the key you put in your web server config
A Certificate Signing Requests (CSR) — This contains the list of domains you want to be part of the certificate

The first API endpoint is “new-reg”. This authenticates you as a “person” or “server” against Let’s Encrypt. You send your Account Key’s public key signed with the private key to Let’s Encrypt. You can optionally include a mail address to get notified about expired certificates.

Second, there is “new-authz”. You send your CSR to Let’s Encrypt signed with your Account Key. The API responds with a list of challenges. These are specially crafted URLs, with specific content, under each requested domain, you need to make available. This is the step to proof you have control over the content served under a given domain.

“challenge” is the next step. For each challenge you received, Let’s Encrypt is going to request the related challenge you made available and check for its correctness.

Lastly, “new-cert”. When all challenges succeeded you can request the certificate and put that in the place where your web server finds it.

And here’s the Apache 2 / httpd config to make those challenges available

<VirtualHost *:80>
    ServerName  example.com

    Redirect    /  https://example.com/

    Alias "/.well-known/acme-challenge/" "/srv/http/acme-challenges/"
    <Directory "/srv/http/acme-challenges">
        AllowOverride None
        Options None
        Require all granted
    </Directory>
</VirtualHost>

And the same one for Nginx.

host {
    listen       [::]:80;
    server_name  example.com;

    location /.well-known/acme-challenge/ {
        alias      /srv/http/acme-challenges/;
        try_files  $uri =404;
    }

    location / {
        return  301  https://example.com$request_uri;
    }
}

How to use Let’s Encrypt?

There’s an official client which does “all” the magic. Literally, it rewrites your Apache config file.

That’s the wrong approach, in my opinion. That’s a task for configuration and system management tools. Watch my talk from 2015th PyCon Australia for details on that topic.

There’s a script acme-tiny by Daniel Roesler which has 200 lines. Easy to understand. I Recommended to read through the code. I forked that and added support for Systemd and a few other things.

There are also a bunch of other tools. E.g. letsencrypt-aws by Alex Gaynor. I haven’t used it myself, but it probably just does what the name suggests.

And there’s Rproxy by Amber Brown

Personally and at djangoproject.com we use my fork of acme-tiny.

But all the scripts boil down to the same things: You need an account key, a certificate signing request, a place to put the challenges, and a final location for the full certificate.

$ python3 /etc/acme-tiny/acme-tiny.py \
    --account-key "/etc/acme-tiny/account.key" \
    --csr "/etc/acme-tiny/example.com.csr" \
    --acme-dir "/srv/www/acme-challenges" \
    --output "/etc/nginx/ssl/example.com.crt" \
    --combine "https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem"

Adjusting Django

Now that this is sorted, how do we use HTTPS in Django and what do you need to change?

As it turns out, not much.

You want to make cookies only accessible via HTTPS. But only on production systems. You can’t use HTTPS locally at least not without some complex steps. At least not with built-in features. There is, however, a runserver_plus command in the django-extensions 3rd party app. Hence, set the settings variables through environment variables:

import os

CSRF_COOKIE_SECURE = os.getenv('SECURE_COOKIES') == 'yes'
SESSION_COOKIE_SECURE = os.getenv('SECURE_COOKIES') == 'yes'

If you use secure cookies with the runserver command this might manifest as a HTTP 403 error when you try to login to the admin because the CSRF cookie won’t be submitted.

Apart from that, there a couple of things you can optionally enable. Have a look at the Security Topic in the Documentation.

Most things you don’t want to do in Django though, but in your reverse proxy if at all possible. It’ll be faster.

What I didn’t cover … but want to mention

Let’s Encrypt’s certificates are only valid for 90 days. If you need to revoke one before, there’s an API call for that.

If you lost your account key or it was compromised you can change it as well.

You probably want HSTS (HTTP Strict Transport Security). Really. Running a website on HTTP and HTTPS may make it possible for attackers to force HTTP requests.

Similar to HSTS there’s also HPKP (HTTP Public Key Pinning).

This feature is neat but mostly not too useful unless you’re a big organization and you are regular victim of DNS attacks.

And of course you can use the Let’s Encrypy certificates for services other than HTTPS. The only thing you need to make sure is that the domains for that certificate serve valid challenge responses on port 80 as explained above.

Things that could go wrong — An incomplete list

And because we’re talking about cryptographic tropic, here’s an incomplete list of things that can go wrong:

Both, HSTS and HPKP come with a risk of making your website unavailable for previous users. Check out how it works. You want it, but be aware of the risks!
When you deal with cryptographic keys, there’s always a possibility to leak keys. Be aware of that and have a process in place to handle the case of a leak certificate or account key!
Some people claim HTTPS is causing too much resourcen usage and they can’t use it. With not too old hardware that’s not an issue.

Resources

Reference variables in Ansible YAML

2016-04-11T00:00:00+02:00

Today I found myself in a typical DRY situation when working on an Ansible role. I was defining a slightly more complex variable to later be used in an Ansible template. The variable looked a bit like this:

backups:
  - name: primary
    output: /mnt/data/extern
    includes:
      - /etc/
      - /home/
      - /var/lib/
  - name: secondary
    output: /mnt/data/intern
    includes:
      - /etc/
      - /home/
      - /var/lib/
  - name: remote
    output: "ssh://user@host:/mnt/data/backup/client1"
    includes:
      - /etc/
      - /home/
      - /var/lib/
  - name: grab-n-go
    output: /mnt/usb-drive
    includes:
      - /home/markus/documents/important

As you can easily see, the includes item always contains the same values for the first 3 items. I looked into the possibility to define the list of directories only once and use a variable in those places.

I started off with this approach:

backup_includes:
  - /etc/
  - /home/
  - /var/lib/

backups:
  - name: primary
    output: /mnt/data/extern
    includes: backup_includes
  - name: secondary
    output: /mnt/data/intern
    includes: backup_includes
  - name: remote
    output: "ssh://user@host:/mnt/data/backup/client1"
    includes: backup_includes
  - name: grab-n-go
    output: /mnt/usb-drive
    includes:
      - /home/markus/documents/important

Turns out, that does not work. Instead, when iterating over backups.X.includes Jinja2 is iterating over the string backup_includes. That’s of course not what I want.

But YAML does provide a solution to reference other variables, which I found on StackOverflow. It’s also shown on Wikipedia. Using & to define an anchor and * to dereference it.

backup_includes: &backup_includes
  - /etc/
  - /home/
  - /var/lib/

backups:
  - name: primary
    output: /mnt/data/extern
    includes: *backup_includes
  - name: secondary
    output: /mnt/data/intern
    includes: *backup_includes
  - name: remote
    output: "ssh://user@host:/mnt/data/backup/client1"
    includes: *backup_includes
  - name: grab-n-go
    output: /mnt/usb-drive
    includes:
      - /home/markus/documents/important

Apart from & and * there’s also the <<: operator to clone an item and allow overriding of its child attributes:

backup_item: &backup_item
  includes:
    - /etc/
    - /home/
    - /var/lib/
  excludes: []

backups:
  - <<: *backup_item
    name: primary
    output: /mnt/data/extern
  - <<: *backup_item
    name: secondary
    output: /mnt/data/intern
  - <<: *backup_item
    name: remote
    output: "ssh://user@host:/mnt/data/backup/client1"
    excludes:
      - /var/lib/

Django Migrations Recipe #1

2016-04-04T00:00:00+02:00

This is the first migration recipe I publish for Django’s migration framework. I published it as part of my talk at DjangoCon Europe 2016 in Budapest. This recipe will show that migration code is just Python and Django’s built-in “makemigrations” command might not output the most efficient migration.

This recipe show how one can optimize a migration Django creates.

Take the following 2 models, a “Book” and a “Library”. Each book is associated with exactly one library:

from django.db import models

class Book(models.Model):
    title = models.CharField(max_length=200)
    library = models.ForeignKey('Library')

class Library(models.Model):
    name = models.CharField(max_length=200)

When you run makemigrations for an app with these models you get 3 operations.

The first one is a CreateModel operation and ensures the Book model exists with a primary key and a name. But the reference to the library is nowhere to be found.
The CreateModel operation for the Library looks as expected. There’s a primary key and a name.
The next operation is an AddField. And there it is, the library field for the Book model.

$ python manage.py makemigrations
Migrations for 'optimize_makemigrations':
  0001_initial.py:
    - Create model Book
    - Create model Library
    - Add field library to book

You may ask now why the ForeignKey is added as a separate operation. There are 2, maybe 3 reasons behind that:

Firstly, having references at the end of a migration when automatically creating one is always safe.
Secondly, Django’s migration framework processes all models within the same app in alphabetical order to provide a deterministic behavior. “Book” comes before “Library”, last time I checked.
Thirdly, nobody has contributed a patch yet. If you’re staying for the sprints, that might be something to look into.

from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = []
    operations = [
        migrations.CreateModel(
            name='Book',
            fields=[
                ('id', models.AutoField(...)),
                ('title', models.CharField(max_length=200)),
            ],
        ),
        migrations.CreateModel(
            name='Library',
            fields=[
                ('id', models.AutoField(...)),
                ('name', models.CharField(max_length=200)),
            ],
        ),
        migrations.AddField(
            model_name='book',
            name='library',
            field=models.ForeignKey(to='app.Library'),
        ),
    ]

Now, that we have this example, how can it be optimized? I guess some of you already have an idea:

from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = []
    operations = [
        migrations.CreateModel(
            name='Library',
            fields=[
                ('id', models.AutoField(...)),
                ('name', models.CharField(max_length=200)),
            ],
        ),
        migrations.CreateModel(
            name='Book',
            fields=[
                ('id', models.AutoField(...)),
                ('title', models.CharField(max_length=200)),
                ('library', models.ForeignKey(to='app.Library')),
            ],
        ),
    ]

The answer to that is, to re-order the CreateModel operations and merge the AddField into the CreateModel for the Book.

Resources

Django Migrations Recipe #2

2016-04-04T00:00:00+02:00

This is the second migration recipe I publish for Django’s migration framework. I published it as part of my talk at DjangoCon Europe 2016 in Budapest. This recipe will show how to add a non-nullable column to a table with already existing data.

Adding columns is a common behavior when we write our software. Requirements change and so do our models. Sometimes we’d like to add a column that we don’t want to be nullable on database level.

The following recipe works only when you have some data source where you can get the value for existing values from. That could be a CSV file, an API that you query or anything else.

Let’s start with an Author model this time. The additional “create” classmethod is for demo purposes only and reflects all places in your code where you create an Author instance:

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=50)

    @classmethod
    def create(cls, name):
        return cls.objects.create(name=name)

Before we add a field we need to create the migration for the current state. We’ll also create a migration for initial data:

$ python manage.py makemigrations
Migrations for 'non_nullable_field':
  0001_initial.py:
    - Create model Author
$ python manage.py makemigrations non_nullable_field --empty --name initial_data
Migrations for 'non_nullable_field':
  0002_initial_data.py:

We’ll use the empty migration to create some initial data:

from django.db import migrations

def forwards(apps, schema_editor):
    Author = apps.get_model('non_nullable_field', 'Author')
    Author.objects.create(name='Author 1')
    Author.objects.create(name='Author 2')

def backwards(apps, schema_editor):
    Author = apps.get_model('non_nullable_field', 'Author')
    Author.objects.filter(name='Author 1').delete()
    Author.objects.filter(name='Author 2').delete()

class Migration(migrations.Migration):
    dependencies = [
        ('non_nullable_field', '0001_initial'),
    ]
    operations = [
        migrations.RunPython(forwards, backwards),
    ]

Each time this migration is applied the “forwards” function is being called and two authors are being created. Each time the migration is unapplied the “backwards” function is being called and the two authors are being deleted.

We can then apply all migrations

$ python manage.py migrate non_nullable_field
Operations to perform:
  Apply all migrations: non_nullable_field
Running migrations:
  Rendering model states... DONE
  Applying non_nullable_field.0001_initial... OK
  Applying non_nullable_field.0002_initial_data... OK

Now we can proceed with adding an author’s homepage. This is fairly easy by adding an URLField! Note that this is nullable for the moment! We will change this to null=False a bit later:

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=50)
    homepage = models.URLField(null=True)

    @classmethod
    def create(cls, name):
        return cls.objects.create(name=name)

But first, let’s create and run migrations! Again

$ python manage.py makemigrations
Migrations for 'non_nullable_field':
  0003_author_homepage.py:
    - Add field homepage to author
$ python manage.py migrate non_nullable_field
Operations to perform:
  Apply all migrations: non_nullable_field
Running migrations:
  Rendering model states... DONE
  Applying non_nullable_field.0003_author_homepage... OK

The next step is taking care of the homepage field. Each time we are creating an author we want to ensure the homepage is set. This applies to all new items.

However, for existing rows we still need to take care of NULL values when displaying the value:

from django.db import models
from django.utils import html, safestring

class Author(models.Model):
    name = models.CharField(max_length=50)
    homepage = models.URLField(null=True)

    @classmethod
    def create(cls, name, homepage):
        return cls.objects.create(name=name, homepage=homepage)

    @property
    def homepage_tag(self):
        if self.homepage:
            return html.format_html('<a href="{u}">{u}</a>', u=self.homepage)
        return safestring.mark_safe('<i>No homepage</i>')

Before we can eventually drop the so called “NULL constraint”, we need to populate records with NULL for homepage in the database with values. We do that in another migration. Go ahead and create an empty one

$ python manage.py makemigrations non_nullable_field --empty --name populate_data
Migrations for 'non_nullable_field':
  0004_populate_data.py:

It doesn’t really matter where you get the values for existing rows from. As already said, this can be a CSV file, an API or anything else. The important thing is that you ensure that in the end every row has a value. No row must be NULL:

from django.db import migrations

LOOKUP_DATA = {
    'Author 1': 'http://example.com',
    'Author 2': 'http://other.org',
}

def forwards(apps, schema_editor):
    Author = apps.get_model('non_nullable_field', 'Author')
    for author in Author.objects.filter(homepage__isnull=True):
        author.homepage = LOOKUP_DATA[author.name]
        author.save(update_fields=['homepage'])

class Migration(migrations.Migration):
    dependencies = [
        ('non_nullable_field', '0003_homepage'),
    ]
    operations = [
        migrations.RunPython(forwards, migrations.RunPython.noop),
    ]

Go ahead and apply that migration. Everything past this comparably easy

$ python manage.py migrate non_nullable_field
Operations to perform:
  Apply all migrations: non_nullable_field
Running migrations:
  Rendering model states... DONE
  Applying non_nullable_field.0004_populate_data... OK

Let’s start by dropping the “null=True” from the URLField:

from django.db import models
from django.utils import html, safestring

class Author(models.Model):
    name = models.CharField(max_length=50)
    homepage = models.URLField()

    @classmethod
    def create(cls, name, homepage):
        return cls.objects.create(name=name, homepage=homepage)

    @property
    def homepage_tag(self):
        if self.homepage:
            return html.format_html('<a href="{u}">{u}</a>', u=self.homepage)
        return safestring.mark_safe('<i>No homepage</i>')

When you now run “makemigrations” Django is asking you how to handle the change. Django doesn’t know that we took care of all NULL values. Hence we need to tell it: Select option 2

$ python manage.py makemigrations --name not_null_constraint

You are trying to change the nullable field to non-nullable without a default ...
Please select a fix:
 1) Provide a one-off value ...
 2) Ignore for now ...
 3) Quit ...
Select an option: 2
Migrations for 'non_nullable_field':
  0005_not_null_constraint.py:
    - Alter field homepage on author

This is the resulting migration. As you can see in the AlterField operation, the URLField doesn’t have a null=True anymore and will therefore add a NOT NULL constraint to the database:

from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = [('non_nullable_field', '0004_populate')]
    operations = [
        migrations.AlterField(
            model_name='author',
            name='homepage',
            field=models.URLField(),
        ),
    ]

Go ahead and apply that migration

$ python manage.py migrate non_nullable_field
Operations to perform:
  Apply all migrations: non_nullable_field
Running migrations:
  Rendering model states... DONE
  Applying non_nullable_field.0005_not_null_constraint... OK

Lastly, you should remove the code that handles NULL values from your code base:

from django.db import models
from django.utils import html

class Author(models.Model):
    name = models.CharField(max_length=50)
    homepage = models.URLField()

    @classmethod
    def create(cls, name, homepage):
        return cls.objects.create(name=name, homepage=homepage)

    @property
    def homepage_tag(self):
        return html.format_html('<a href="{u}">{u}</a>', u=self.homepage)

Resources

Django Migrations Recipe #3

2016-04-04T00:00:00+02:00

This is the third migration recipe I publish for Django’s migration framework. I published it as part of my talk at DjangoCon Europe 2016 in Budapest. This recipe will show how to rename an app. For simplicity the app must not have any dependencies from other apps!

Take an app “rename_app” with the following 2 models, an “Author” and a “Book”.

# rename_app/models.py

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=50)

class Book(models.Model):
    title = models.CharField(max_length=50)
    author = models.ForeignKey('rename_app.Author')

Similar to the previous recipe we create the initial migration and a data migration and apply them:

$ python manage.py makemigrations
$ python manage.py makemigrations rename_app --empty --name initial_data

# Edit rename_app/migrations/0002_initial_data.py

$ python manage.py migrate rename_app

The first step in order to rename the app from “rename_app” to “new_app_name” is forcing Django to a particular table name by adding the db_table attribute on the models’ meta classes:

# rename_app/models.py

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=50)

    class Meta:
        db_table = 'rename_app_author'

class Book(models.Model):
    title = models.CharField(max_length=50)
    author = models.ForeignKey('rename_app.Author')

    class Meta:
        db_table = 'rename_app_book'

Also create the respective migration for this change and apply it:

$ python manage.py makemigrations --name pin_db_tables

# rename_app/migrations/0003_pin_db_tables.py

from django.db import migrations, models
class Migration(migrations.Migration):
    dependencies = [
        ('rename_app', '0002_initial_data'),
    ]
    operations = [
        migrations.AlterModelTable(name='author', table='rename_app_author'),
        migrations.AlterModelTable(name='book', table='rename_app_book'),
    ]

$ python manage.py migrate

The next step is letting Django think you’ve never applied these migrations whilst not changing the database. You do that by passing the string “zero” as a migration name together with the —fake flag to the migrate management command:

$ python manage.py migrate rename_app zero --fake
Operations to perform:
  Unapply all migrations: rename_app
Running migrations:
  Rendering model states... DONE
  Unapplying rename_app.0003_pin_db_tables... FAKED
  Unapplying rename_app.0002_initial_data... FAKED
  Unapplying rename_app.0001_initial... FAKED

After that you need to do the actual rename of the files and references from “rename_app” to “new_app_name” in every file, including your settings, models and migration files.

EXCEPT for the “db_table” attribute! DO NOT CHANGE THIS NOW!

# in settings.py
INSTALLED_APPS = [
    # ...
    'new_app_name.apps.NewAppNameConfig',
]


# in new_app_name/models.py
class Author(models.Model):
    author = models.ForeignKey('new_app_name.Author')

    class Meta:
        db_table = 'rename_app_book'  # Keep as is for now!


# in new_app_name/migrations/0003_pin_db_tables.py and others
dependencies = [
    ('new_app_name', '0002_initial_data'),
]

When you’re done with that and you didn’t make a mistake along the way, let Django know about the new migrations. Again, you need to pass the —fake flag in order to only record the migration as applied and not do any database operations:

$ python manage.py migrate new_app_name --fake
Operations to perform:
  Apply all migrations: new_app_name
Running migrations:
  Rendering model states... DONE
  Applying new_app_name.0001_initial... FAKED
  Applying new_app_name.0002_initial_data... FAKED
  Applying new_app_name.0003_pin_db_tables... FAKED

Last but not least, you can optionally drop the db_table attribute to rename the tables from for example “rename_app_author” to “new_app_name_author”. When you remove the db_table attribute you have to run makemigration again:

# new_app_name/models.py

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=50)

class Book(models.Model):
    title = models.CharField(max_length=50)
    author = models.ForeignKey('new_app_name.Author')

When you remove the db_table attribute you have to run makemigration again:

What you can see here is, that the table name is reset to None. Django will therefore automatically derive the name from the app name and model name.

$ python manage.py makemigrations --name rename_tables

# new_app_name/migrations/0004_rename_tables.py

from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = [
        ('new_app_name', '0003_pin_db_tables'),
    ]
    operations = [
        migrations.AlterModelTable(name='author', table=None),
        migrations.AlterModelTable(name='book', table=None),
    ]

$ python manage.py migrate

Resources

Don’t be afraid of writing migrations

2016-04-04T00:00:00+02:00

Introduction

Django’s migration framework evolved a lot since it first release with 1.7. Heaps of bugs have been fixed and numerous features added.

However, there are still lots of things that can be improved to make migrations faster, some tasks more convenient to use, and some tasks even possible. One thing in particular is the makemigrations management command that automatically outputs the required migrations when you changed a model. But there are things makemigrations just can’t do. In those cases you need to write migrations by hand.

Writing migrations may sound scary, but in this blog post and the related talk I want to explain why you don’t need to be afraid of writing migrations. I provide 3 recipes, an easy, and intermediate and a more complex one, which you can use for to get a feeling for how migrations work and how to solve problems you encounter in your own projects. Be advised that some of the recipes may only work with PostgreSQL. SQLite doesn’t enforce ForeignKey integrity and on MySQL you’re screwed when migrations fail. I didn’t try Oracle.

General Layout

Before I go into any details or internals, let me explain the general layout of a migration:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = []
    operations = []

You start off with a class called Migration inheriting from django.db.migrations.Migration.

You then have an attribute dependencies which is a list of 2-tuples pointing to other migrations which this migration depends on. As an example, when you have a “Profile” model with a ForeignKey to a “User” model, the user database table needs to exist before the profile table can point to the user table.

The “operations” attribute holds a list of migration operations, such as CreateModel, AddField, or AlterField. These operations will be run in order when you apply a migration file.

Recipes

Resources

Syntax highlighting for Django’s SQL query logging

2016-01-30T00:00:00+01:00

Let’s start at the beginning: Python’s logging module is hard to understand. And it’s not much easier to configure it the way that you want it to behave. There are filters, formatters, handlers, loggers, to only name a few components.

filters: Provide a fine-grained way to suppress or allow a log record to be send to handlers.
formatters: Given a formatting string and a log record, they generate the string that is actually logged.
handlers: Define how a log record is handled. This could be an output to the console or sending an email.
loggers: Tell the origin of a message on a higher level (stack, file, line number, etc is derived automatically).

This is a simple logging example:

import logging

logger = logging.getLogger('primes.finder')
logger.debug('Next prime after %s is %s', 97, 101)

The logger effectively has the name 'primes.finder' and will be referred to by that name in Django’s LOGGING configuration. Potentially defined filters on a logger or handler would be consulted before a message is passed to a handler from the logger. Such a filter could e.g. inspect the two primes and see if the second has more digits than the first.

A handler takes care of outputting a log record. It could e.g. output the log message on a command line or send it via email. In the former case a formatter would be used to define a general format of the log messages, e.g. prepend a timestamp and log level.

In order to provide syntax highlighting for the SQL queries Django runs we will need to write our own formatter. To do that, we need to look at the way Django logs the queries (django.db.backends.utils.CursorDebugWrapper.execute()):

logger.debug(
    '(%.3f) %s; args=%s' % (duration, sql, params),
    extra={
        'duration': duration,
        'sql': sql,
        'params': params,
    }
)

What we can see here is the record object that our the formatter will receive has the attributes duration, sql and params, derived from the extra keyword argument.

class SQLFormatter(logging.Formatter):
    def format(self, record):
        # Check if Pygments is available for coloring
        try:
            import pygments
            from pygments.lexers import SqlLexer
            from pygments.formatters import TerminalTrueColorFormatter
        except ImportError:
            pygments = None

        # Check if sqlparse is available for indentation
        try:
            import sqlparse
        except ImportError:
            sqlparse = None

        # Remove leading and trailing whitespaces
        sql = record.sql.strip()

        if sqlparse:
            # Indent the SQL query
            sql = sqlparse.format(sql, reindent=True)

        if pygments:
            # Highlight the SQL query
            sql = pygments.highlight(
                sql,
                SqlLexer(),
                TerminalTrueColorFormatter(style='monokai')
            )

        # Set the record's statement to the formatted query
        record.statement = sql
        return super(SQLFormatter, self).format(record)

Update your LOGGING configuration to include the sql formatter, sql handler and django.db.backends logger:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'sql': {
            '()': 'path.to.your.SQLFormatter',
            'format': '[%(duration).3f] %(statement)s',
        }
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        },
        'sql': {
            'class': 'logging.StreamHandler',
            'formatter': 'sql',
            'level': 'DEBUG',
        },
    },
    'loggers': {
        'django.db.backends': {
            'handlers': ['sql'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'django.db.backends.schema': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': False,
        },
    }
}

Update: In case you’re working with a 256 color terminal, you should use the Terminal256Formatter instead of the TerminalTrueColorFormatter in the SQLFormatter.format() method. (Thanks Felix Hummel).

Update: The Necessity of Configuration and System Management Tools

2015-08-01T00:00:00+02:00

Over the last months I wrote my Master’s Thesis in Computer Science about configuration and system management tools at the Technical University of Berlin, Germany, supervised by Prof. Dr. Odej Kao. I was also supported by Flying Circus Internet Operations GmbH.

The official title of my thesis is “Evaluating Methods to Maintain System Stability and Security When Reversing Changes Made by Configuration and System Management Tools in UNIX Environments“. In essence it’s points out what you as a system administrator should care about and take into account when using configuration and system management tools, such as Ansible, Chef, and Puppet. Using these tools is easy, and takes a lot of your plate when dealing with larger IT environments, but without considering certain things you likely break your environment at some point.

Consider the following example:

A hosting provider offers virtual machines that have an IP address that is assigned from a pool of free addresses when the machine is first set up. At some point a customer requests the server to be shut down and removed. Once the server is removed, its IP address is freed and goes back into the pool. However, a customer is able to request a restore of a server from a backup within a given time frame after its removal. This works perfectly fine as long as the original IP address is not in use. However, configuring the network setup will fail if the IP address has been recycled to a different machine. [ch. 1, p. 1]

My master’s thesis deals with this and many other cases where problems might be hard to spot.

In order for you to inspect your IT environment and find those problems I came up with a taxonomy of IT resources that helps you classify all resources [ch. 5.1, p. 23]:

For the example given above, the IP addresses of VMs are identifying resources.

After classifying your resources you can follow the rules I derived from the taxonomy ([ch. 5.2, p. 26]) to narrow down potential conflicts. In the above case “Rule 6: Use environment-wide unique identifiers” is violated. The hosting provider should keep a record of IP addresses that are still in use inside backups before actually recycling the addresses.

In chapter 5.3 I provide a variety of use cases that you might encounter in your IT environment. Chapter 6 takes up those cases and outlines possibilities of how to solve potential issues.

License

You are allowed to use all resources that are part of this article free of charge as long as you include my name and a reference to this article in a place the end-user of your product sees.

You may not copy, distribute, or publish the thesis as a whole unless explicitly granted.

Download

Master’s Thesis
Presentation from PyCon Australia 2015
Video recording from PyCon Australia 2015 — I have to admit I was quite nervous

DjangoCon Europe 2015 — The Boy Band Conference

2015-06-15T00:00:00+02:00

The Announcement

The announcement for DjangoCon Europe 2015 — as the conference itself — was vast. A six day conference, including an open day, open and free to the public, three days of “conference talks”, two days of code sprints, workshops and code clinics. Furthermore, yet another DjangoGirls event. The organizers also took up the cause of making the most diverse, accessible and social responsible DjangoCon ever. And, from my perspective, they succeeded! The team around Daniele Procida organized a memorable conference that around 350 people attended. Thank You!

My Trip to Cardiff

My trip to Cardiff wasn’t my first time in the UK, but it was the longest. All the times I have be to UK before, were shorter. And all these other trips were way rainier that this time.

My plane left Berlin Tegel Airport (TXL) on May 30th at 6am, quite early in the morning. On the plus side, I already met Erik Romijn, another Django committer, at Amsterdam Schiphol Airport (AMS) for our flight to Cardiff Airport (CWL). Interestingly, our airline, KLM, overbooked the flight and was offering 250 EUR to the first person that would take a later flight.

On both, the flight from TXL to AMS and after the conference from CWL to AMS, KLM served sandwiches. And these sandwiches perfectly aligned with one of the policies of the conference: social responsibility.

Cardiff’s Architecture

The conference and all related events took place at Cardiff University (a huge thank you for being involved!) and Cardiff City Hall.

DjangoGirls

Especially Cardiff University conveys a sense of good old British university life. The stunning building, with its many small corridors and corners, served well for the DjangoGirls workshop, putting everybody back into (or forward to) their university life.

Conference Dinner

Right next to Cardiff City Hall, the National Museum Cardiff was host for the conference dinner. I can’t really remember what the meal was called, but I know, it was really good and tasty.

Cardiff Castle

Cardiff’s landmark is its castle (Welsh: Castell Caerdydd). It “was built in the late 11th century by Norman invaders on top of a 3rd-century Roman fort.” [Wiki]

The Conference

Talks

The conference itself was great. The organizers encouraged people that hadn’t been on stage to give a presentation in front of hundreds of people. And it worked quite good. I’ve seen many first time speakers and I was one myself, talking about “Forms are static — No, they aren’t” and my first Django third-party app django-dynamic-forms

The opening keynote by Baptiste Mispelon was a story of “how he tackled burnout by traveling around Europe, eating Welsh cakes, pierogi and stroopwafel” [DCE15]. The entire story made out of Emoji, his favorite language. Nothing even close to programming or code, but an insightful and honest statement of a common problem in IT.

This topic, to listen to yourself and to your friends and colleagues around you, was also supported by the wellbeing service of Cardiff University that offered free sessions for everybody who wanted to talk to somebody and get some recommendations or ideas.but

But even more surprising — at least to me and likely many others in the audience — was Russell Keith-Magee’s lightning about his illness. Thank you for being so open, trusting us with your story and get well soon :)

Another talk I specifically want to mention here, is the keynote from the second day by Ola Sendecka: Into the rabbit hole, about a girl Alice in Djangoland. But the most impressive about this talk was not the content (that was great and correct), but the hand painted images.

There were plenty of other talks I enjoyed a lot, because they explained something difficult in a simple way (Ana Balica: Demystifying mixins with Django), because they showed something outside Django’s normal use case (Dafydd Evans: CAMEL, the Cardiff Maths e-learning project), or because they showed once more that computer science, and computer security in particular, is hard (James Bennett: The net is dark and full of terrors).

Accessibility

As already announced, the organizers wanted everybody to feel welcome. And an important support for people with hearing disabilities were the live transcripts by Sheryll Holley and Hilary Maclean, that are now public for everybody to read.

One Direction

The most important part of the entire DjangoCon, though, were the two One Direction concerts on Friday and Saturday evening after the conference. About 110 000 fans and visitors were expected. That is a third of the population of Cardiff:

License

The photos in this blog post are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The article image (group photo) is licensed under the terms of the Creative Commons Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0) by Patryk Zawadzki.

[Wiki]

https://en.wikipedia.org/wiki/Cardiff_Castle

[DCE15]

http://2015.djangocon.eu/talks/

A lot has happended!

2015-04-11T00:00:00+02:00

It has been 5 months that I wrote something on my blog. I still remember the time writing the article. On my way to Django: Under The Hood 2014, my favorite conference in 2014 as an attendee. And shortly after that my time to write something here vanished. For various reasons.

December 2014

While getting 9 commits into Django’s code base throughout December, we (the server team at ubuntuusers.de) finally upgraded our infrastructure to new hardware. A few days before the holidays I decided to finally fill out the form and enrolled for my master thesis.

January 2015

2015 began quietly. Not many things happened. Until January 9th! Carl Meyer queried me on IRC:

We’d like to invite you to be part of the Django core team, if you’re interested.

I was super happy and excited (and still am). And according to the official announcement I was the most active non-core-team code contributor in 2014. Interesting, I wasn’t aware of that and haven’t even though about it :)

Later that month I got the confirmation for the topic of my Master Thesis. I won’t tell it here until I finished and got my grade for it. It’s about configuration and system management tools. But I’m working on my thesis ever since which is, together with my work as Django core developer, the reason why I don’t really have time to do anything else.

On 25th I was a coach at DjangoGirls Berlin. That’s the second time I’ve been coaching (first time was after Django: Under The Hood) and the second time DjangoGirls came to Berlin (first time was at EuroPython 2014).

February 2015

There are not many things worth mentioning for February. I gave a lightning talk at the Python Users Berlin meetup about how to integrate Sphinx and Elasticsearch.

A week later I gave a talk at the Django User Group Berlin meetup about Combining Django & Elasticsearch (Slides).

March 2015

Compared to February, March was really busy in terms of travelling. I went to the Django Sprint in Amsterdam. Thanks again to the organizers. It was a pleasure to be back in Amsterdam.

One week later a couple of friends and I finally went on a skiing vacation again. We had amazing weather. The view from the top of the mountains was breathtaking. I took a few photos! The photos are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC-BY-NC-SA).

April 2015

On April 1st we released Django 1.8 as our next LTS (Long Term Support) version with a hole bunch of new features and improvements:

Native support for multiple template engines.
Support for complex SQL expressions via the ORM.
A formalized API for Model._meta.
New PostgreSQL specific functionality in contrib.postgres.

The features and changes I’m particularly happy about are the serialization of model managers in migrations (#23822) which I have worked on for quite a while and of course the performance improvements in migrations. It was some hard work but thanks to the amazing work Claude Paroz did on #23745 and Marten Kenbeek did on #24366 the migrations in 1.8 are noticeable faster. Of course thanks to all contributors who made the release the way it is :)

Flashing Android 5 “Lollipop”: Lessons Learned

2014-11-13T00:00:00+01:00

Yesterday evening Google finally published the Android 5 “Lollipop” factory images. Eagerly waiting for the update I couldn’t hold myself and flashed my Nexus 5. This was the first time on the Nexus 5, the previous KitKat updates came OTA. Though I’m not really familiar with the whole flashing thing (last time I flashed my HTC Desire HD with some CyanogenMod fork back in summer 2013), the entire process went smoothly. At least kind of.

This post is more about the things I messed up because they changed or were different to flashing on the HTC Desire HD and thus is a note to myself what I should remember in the future.

The ArchLinux [community] repository has a android-tools package that provides adb and fastboot. No problems at that point.
Everything that is stored under /sdcard/ isn’t on an actual SD card, but internal storage! Guess what, I didn’t realize that until I ran fastboot oem unlock and eventually booted the device later. The device warned me that all user data will be wiped - fair enough - but who thought wiping data in /sdcard/ is a good idea? Anyway, the data couldn’t have been important, I had no backup :D
Be patient during the first boot after flashing the image! The process takes time. It took about 20 minutes (guessing, didn’t measure it).
Remember your two factor authentication setup. Make sure to have the backup keys for your Google account or you might not be able to login, because you just wiped the second part of the 2FA setup. If you’re still logged in to your Google account on a second device, you can temporarily turn off 2FA, setup your smartphone again and activate 2FA again. Make sure you make a copy of the new backup codes. The old ones won’t work anymore. Same goes for e.g. GitHub: you can use an active login to re-enable two factor authentication
If you backup some Chrome data (bookmarks, open tabs, extensions, autofill, …), make sure you have the password to get your data back on the smartphone. Open about:chrome in Chrome and check the value for “Explicit Passphrase”. If it’s true, go and find your password. If it’s false, the data is encrypted with your Google account password, you only need that one.
If you use Google Play Music or Spotify or similar apps with loads of offline data, make sure you have enough time to sync them or you might find yourself on a train without Internet connectivity and without Music :(

Festival of Lights 2014

2014-10-20T00:00:00+02:00

From October 10 to 19, 2014 the 10. Festival of Lights happened in Berlin after its foundation in 2005. This year’s theme was “Colours of Joy”. During those 10 nights many monuments and important places and buildings are turned into artworks by national and international artists.

Unfortunately I didn’t manage to visit the illuminated attractions earlier that week, so I only had few hours on Sunday evening. Sadly I didn’t have enough time to all places, but only a few. But the whole event was a blast and the productions really impressive. Too bad I didn’t hear about it before this year.

I took the photos with my Nexus 5, so don’t expect them to be DSLR quality. I additionally played around with some of Google’s graphic effects filters, in particular the black-white filter.

Brandenburg Gate

Potsdam Square

Humboldt University and Hotel de Rome

License

The photos in this blog post are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License..

Berliner Schulen in Zahlen

2014-09-30T00:00:00+02:00

Berlin. 2014. 1093 Bildungseinrichtungen. 835 Zeilen Javascript Code. 573 Zeilen HTML Code. 347 Zeilen Python Code. 31 Schularten. 29 Schulzweige. 19 Ausstattungskriterien. 15 Sprachen. 12 Stadtteile. 2 Möglichkeiten das selbe zu sagen. 1 Seminar: Ini 2.0.

Wie soll man nun bei den ganzen Zahlen den Überblick bewahren? Das war die Fragestellung mit wir uns im Seminar “Data Science — Hacking Society” der Ini 2.0 im Sommersemester 2014 an der TU Berlin auseinandergesetzt haben. Aber zunächst zurück zum Anfang des Projektes.

Im Wintersemester 13/14 fragten wir bei der Stadt Berlin Daten zu den Bildungseinrichtungen im Stadtgebiet an. Dass die Stadt Berlin sehr detaillierte Daten zu allen Bildungseinrichtungen erfasst, konnten wir anhand der schon existierenden Recherche Webseite erahnen. Die Usability dieser Webseite ist allerdings nicht die allerbeste, außerdem wollten wir auch die zu Grunde liegenden Daten auswerten.

Bis die Anfrage nach den Daten befriedigend beantwortet wurde verging leider recht viel Zeit, daher war im Wintersemester 2013/2014 nur noch Zeit zum ersten Sichten und Aufräumen der Datensätze vorhanden. Uns wurde eine Reihe von Excel-Dokumenten zur Verfügung gestellt, welche glücklicherweise sehr ordentlich waren. Einen ersten Prototyp einer besseren Rechercheanwendung konnten wir zwar beginnen, leider aber nicht fertig stellen.

Das gab den nachfolgenden Studenten im Sommersemester 2014 aber die Möglichkeit mit einem schon vorhandenen Datensatz sehr schnell arbeiten zu können. Das Ziel waren weitere Auswertungen der Daten, erneute Überprüfung der Datenqualität und natürlich das Erstellen einer Webseite, welche die Recherche im Datensatz sehr angenehm macht.

Datenqualität

Trotz recht aktuellen Daten waren diese an vielen Stellen nicht 100% korrekt. Das mitunter größte Fehlerpotential boten Links zu den Webseiten der Bildungseinrichtungen. Etwa 10% aller Links waren fehlerhaft und verwiesen auf nicht mehr existierende Seiten.

Obwohl die Daten zentral gesammelt werden, scheint die Erfassung nicht auf elektronischem Wege, sondern nach wie vor mit Papierfragebogen und Stift abzulaufen. Anders lassen sich inkonsistente Daten wie “Computerraum” und “Computerräume” oder “Werkstatt” und “Werkstätten” nicht wirklich erklären.

Webanwendung

Bilder sagen mehr als tausend Worte, daher hier ein paar Eindrücke der Webseite:

Bis jetzt sind wir ohne technische Details ausgekommen, aber so ganz kommen wir in diesem Abschnitt nicht drum herum. Die Webseite ist als sogenannte Single-Page-App umgesetzt, so dass bei Benutzung des Filters oder bei der Navigation immer nur Teile der Seite neu geladen werden. Designtechnisch haben wir uns auf Zurb’s Foundation verlassen und das Standardtheme praktisch nicht angepasst, daher sieht das auch so gut aus ;)

Für die dynamische Programmierung haben wir AngularJS verwendet, eine Javascript Bibliothek welche wir anhand dieses Projekts kennen gelernt haben. Die Daten der App lagern am Ende in einer Elasticsearch Instanz und können über AJAX Zugriffe abgefragt werden.

Wer sich für den Code interessiert, diesen findet man in einem öffentlichen Github Projekt.

Ein Artikel zu den technischen Details wurde von Alex und Markus veröffentlicht.

Ausblick

Wir haben viel darüber gelernt wie aufwendig es ist solche Daten bei der öffentlichen Verwaltung anzufragen. Ein Mitglied aus unserem Team ist auch motiviert weiter zu machen, allerdings ist in Baden-Württemberg, trotz Bestandteils des Koalitionsvertrags, noch kein Informationsfreiheitsgesetz in Aussicht. Erwartungsgemäß ist die erste Anfrage nach einem ähnlich umfangreichen Datensatz auch nicht zufriedenstellend beantwortet worden: einen zentralen Datensatz gibt es nicht. Noch dazu soll für eine Auflistung aller Adressen über 100 EUR gezahlt werden. Der nächste Schritt ist dort also mal auf entsprechende Landtagsabgeordnete zuzugehen und sie an gewisse Inhalte ihres Koalitionsvertrags zu erinnern.

Django 1.7: Database Migrations done right

2014-09-17T00:00:00+02:00

About two weeks ago Django 1.7 was released with a lot of new and shiny features, such the App Loading Refactoring, Custom Lookups, the Checks Framework the buildin Migration support and many more. The release notes are the longest in Django’s history and span approx. 1,800 lines.

This article is inspired by the questions and problems I saw on Django’s IRC support channel #django on Freenode regarding the new migration framework django.db.migrations. I’ll show different problems developers ran into, explain why their particular way of doing something doesn’t work (anymore) and how to do this now.

Why not integrate South into Django?

A handful of people asked why South hasn’t just been integrated into Django, so all the old migrations would just work. There are many reasons why Andrew Godwin, the author of South, a Django core developer and the author of django.db.migrations did it differently. But one of the main reasons is probably the way how South stores the state the database has over time, the so called frozen ORM. When you look at a regular sized migration file you often have about 50 lines of forwards and backwards code and a often continuously growing frozen ORM below.

This is clearly not the best solution. Which is why another way of detecting the changes between the current implementation of your app / project and the database state created by the previous migrations had to be found.

Django 1.7 behaves differently. Whenever you run python manage.py makemigrations myapp, Django first applies all migration changes internally by constructing the models and fields with their properties on-the-fly. In the second step Django compares the state generated from all migration files to the state presented in your code. All differences will be considered changes that will be written in new migration files.

Hint

One thing you might notice at some point, the models created by Django on-the-fly belong to the Python module __fake__. This is the reason why you won’t be able to access custom function on the model and why e.g. an overwritten save() will not be called.

To work around circular dependencies, Django splits a single migration file into multiple within the same app. The resulting dependencies are declared in a list at the top of a Migration class:

class Migration(migrations.Migration):

    dependencies = [
        ('thisapp', '0001_initial')
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
        ('otherapp', '0002_auto_20140915_1402')
    ]

    operations = [
        # some operations
    ]

How do dependencies between migrations work?

Let’s assume you have two Django apps author and book with these models:

# author/models.py
from django.db import models

class Author(models.Model):
    name = models.CharField('Author', max_length=255)
    birthday = models.DateField('Birthday')


# book/models.py
from django.db import models

class Book(models.Model):
    title = models.CharField('Title', max_length=255)
    pages = models.PositiveSmallIntegerField()
    author = models.ForeignKey('author.Author')

What happens when you call python manage.py makemigrations? First of all, since no apps are given, Django reads the migrations from all apps listed in INSTALLED_APPS. In our case, this is ('author', 'book',). In the next step Django replays all migrations it can find for those apps internally, as explained above. Since there are no migrations yet, Django is going to start from scratch:

$ python manage.py makemigrations
Migrations for 'book':
  0001_initial.py:
    - Create model Book
Migrations for 'author':
  0001_initial.py:
    - Create model Author

As a result, you will end up with two migrations:

# author/migrations/0001_initial.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='Author',
            fields=[
                ('id', models.AutoField(auto_created=True, verbose_name='ID', primary_key=True, serialize=False)),
                ('name', models.CharField(verbose_name='Author', max_length=255)),
                ('birthday', models.DateField(verbose_name='Birthday')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
    ]


# book/migrations/0001_initial.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('author', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='Book',
            fields=[
                ('id', models.AutoField(primary_key=True, serialize=False, verbose_name='ID', auto_created=True)),
                ('title', models.CharField(max_length=255, verbose_name='Title')),
                ('pages', models.PositiveSmallIntegerField()),
                ('author', models.ForeignKey(to='author.Author')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
    ]

When you now run python manage.py migrate Django looks at all migrations that haven’t been applied yet and will perform all their operations in-order, starting with the migrations that have no dependencies to migrations that haven’t been applied. In the case above, this is author.0001_initial. The dependencies list is empty. Afterwards book.0001_initial can be applied, because all dependencies are resolved.

Hint

Django will try to run the migrations ordered by the name of the app they belong to. But obviously only if there are no dependencies that have to be resolved first.

You can see all migrations in all installed apps, as well as their status (applied or not) by running python manage.py migrate --list:

$ python manage.py migrate --list
author
 [ ] 0001_initial
book
 [ ] 0001_initial

What is the “(‘myapp’, ‘first’)” dependency?

Let’s say, author is a third party app that doesn’t ship with Django migrations. Remove the migrations folder author/migrations as well as book/migrations/0001_initial.py to start over. When you now run makemigrations` you will end up with a single migration file for book:

$ python manage.py makemigrations
Migrations for 'book':
  0001_initial.py:
    - Create model Book

#book/migrations/0001_initial.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('author', '__first__'),
    ]

    operations = [
        migrations.CreateModel(
            name='Book',
            fields=[
                ('id', models.AutoField(verbose_name='ID', primary_key=True, serialize=False, auto_created=True)),
                ('title', models.CharField(verbose_name='Title', max_length=255)),
                ('pages', models.PositiveSmallIntegerField()),
                ('author', models.ForeignKey(to='author.Author')),
            ],
            options={
            },
            bases=(models.Model,),
        ),
    ]

The dependency ('author', '__first__'), tells Django to apply a migration at some point after the first migration in the referenced app, independent of its name.

How do I add a data migration?

If you used South you might know about the datamigration command that simply creates a new migration file in the given app and optionally adds some models to the frozen ORM state.

In Django 1.7 it is way simpler: run python manage.py makemigrations --empty myapp You can rename the generated file if you like as long as it ends with .py and doesn’t contain a .:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('myapp', '0001_initial'),
    ]

    operations = [
    ]

Within this migration you can now add the operations you want to perform. For data migrations you can use migrations.RunSQL or migrations.RunPython.

Important

If you add an empty migration file to an app and want to run operations that require another app to be migrated to a specific state, you have to add the required dependencies explicitly!

Running native SQL commands during migrations

If you are able to express your data changes in SQL, please do so, this will be faster than through the ORM. But keep in mind, that this might not be the solution if you have to fight multiple database back-ends.

RunSQL accepts 1 to 3 arguments: sql, reverse_sql and state_operations. sql is required and expects a string (that may consist of multiple statements).

migrations.RunSQL("UPDATE myapp_mymodel SET col1 = col2 + col3;"
                  "UPDATE myapp_mymodel SET col2 = col3 * col3;")

If you don’t specify the reverse_sql argument, you won’t be able to roll-back beyond this migration. The default is None, using "SELECT 1;" is fine for a roll-back.

With the state_operations attribute you are able to modify the model state Django internally constructs while running the migrations. I haven’t seen a usecase for that yet.

Warning

As of time of writing, if you want to use % as a wildecard in e.g. the WHERE- clause, you need to escape it with another % character (Django issue #23426):

migrations.RunSQL(
    "UPDATE myapp_mymodel SET col1 = 'a' WHERE col2 LIKE '%%val%%';"
)

Run custom Python code during migrations

Apart from the RunSQL operation Django 1.7 comes with a RunPython operation. This allows you to run custom Python function during a forwards or a backwards migration.

RunPython accepts 1 to 3 arguments: code, reverse_code and atomic. code is require and accepts any callable with two arguments, so does reverse_code which is optional, though. atomic defaults to True.

Please keep in mind that a reverse_code of None (the default) prevents the migration from being rolled back. If you want to be able to roll-back, because your Python code in code computes some initial data for every row in a newly added column, add something like lambda x, y: None as reverse_code.

For more details on the RunPython operation please see the docs.

The callable for e.g. upload_to cannot be found

There are a few model fields out their that take callables as arguments to do further processing. One of those fields is the FileField that has an upload_to argument which accepts a string as well as a function to dynamically derive the storage location. To make migrations work automatically, this function has to be directly importable from a package or module.

The same goes for classes for custom fields: The way Python works doesn’t allow importing inner classes. Move the class to the module level and you’ll be fine.

See the chapter about serializing values in the docs.

Backwards migrations roll-back too many operations

The way Django handles the order of migrations and the fact that Django strictly enforces dependencies between migration to be present during migration, is different compared to South. While the forwards migration plans won’t really differ from South’s, Django behaves completely different when it comes to backwards migrations (at least in 1.7, follow Django issue #23474 for updates).

By design Django will roll back the database to the state it would have if you roll forward and stop after a given migration. To make this more clear, let’s take the following scenario from the Django tests:

app_a:  0001 <-- 0002 <--- 0003 <-- 0004
                         /
app_b:  0001 <-- 0002 <-/

If you run python manage.py migrate you will end up with:

[X] app_a.0001
[X] app_a.0002 ... (depends on app_a.0001)
[X] app_b.0001
[X] app_b.0002 ... (depends on app_b.0001)
[X] app_a.0003 ... (depends on app_a.0002, app_b.0002)
[X] app_a.0004 ... (depends on app_a.0003)

If you run python manage.py migrate app_a 0003 from this state, you will end up with:

[X] app_a.0001
[X] app_a.0002 ... (depends on app_a.0001)
[X] app_b.0001
[X] app_b.0002 ... (depends on app_b.0001)
[X] app_a.0003 ... (depends on app_a.0002, app_b.0002)
[ ] app_a.0004 ... (depends on app_a.0003)

being applied.

The difference happens when you roll-back past a dependency.

If you run python manage.py migrate app_a 0002 from the initial state, you will end up with:

[X] app_a.0001
[X] app_a.0002 ... (depends on app_a.0001)
[X] app_b.0001
[X] app_b.0002 ... (depends on app_b.0001)
[ ] app_a.0003 ... (depends on app_a.0002, app_b.0002)
[ ] app_a.0004 ... (depends on app_a.0003)

being applied.

But if you run python manage.py migrate app_b 0002, from the initial state, you will end up with:

[X] app_a.0001
[X] app_a.0002 ... (depends on app_a.0001)
[X] app_b.0001
[X] app_b.0002 ... (depends on app_b.0001)
[ ] app_a.0003 ... (depends on app_a.0002, app_b.0002)
[ ] app_a.0004 ... (depends on app_a.0003)

being applied.

Do you recognize the missing app_a.0003 here.

Django file and memory mail back-end for tests

2014-09-15T00:00:00+02:00

Whenever you write unit or integration tests for your Django application that involves sending mails, you end up with the LocMem mail back-end (django.core.mail.backends.console .EmailBackend) as a default set by Django during test setup. However, if you want to have a look at all the mails that have been sent, you are probably going to choose the File mail back-end (django.core.mail.backends.filebased.EmailBackend) [docs].

The former give you the possibility to check for the amount of mails that have been sent and inspect them further through django.core.mail.outbox, the latter stores all mails in a text file for later inspection.

So, if you are using the LocMem back-end and want to inspect all mails after an entire test run, you are first trying to change the back-end with @override_settings [docs]. This works as long as you don’t use django.core.mail.outbox, because this will always be None or []. How are you going to make that work?

When this question came up on the Django IRC channel I’ll came up with

import django
from django.core import mail
from django.core.mail.backends.filebased import EmailBackend as FileEmailBackend


class EmailBackend(FileEmailBackend):
    def __init__(self, *args, **kwargs):
        super(EmailBackend, self).__init__(*args, **kwargs)
        if not hasattr(mail, 'outbox'):
            mail.outbox = []

    def write_message(self, message):
        if django.VERSION[:2] > (1, 5):
            super(EmailBackend, self).write_message(message)
            mail.outbox.append(message)

    def send_messages(self, messages):
        ret = super(EmailBackend, self).send_messages(messages)
        if django.VERSION[:2] < (1, 6):
            mail.outbox.extend(messages)
        return ret

I didn’t test the code myself, but the user asking for support seemed to find it a working solution.

Logitech mouse M705 double click problem

2014-09-09T00:00:00+02:00

A few years back, it must be around 2011, I bought a Logitech M705 mouse. A few days ago this mouse behaved wired. Almost every single left click turned into a double click. Looking through the Internet it seemed to be a common problem.

I replaced the mouse with an even older one laying around, thinking I’d probably need to replace it with a new one sooner than later. Today though I searched for a way to fix the double click problem. I ended up finding a YouTube video showing how to dismantle the mouse, fix a part and put the components back together.

Problem solved. The mouse is as good as new. At least for now.

Using Elasticsearch as Relational Data Storage

2014-08-10T00:00:00+02:00

This blog post is about a university project Alex, a fellow student at my university, and I have been developing over the last months as part of our Master studies. We are both quite experienced software developers focusing on application and web development. As part of a university project we had absolutely no requirements on a coding language or styling. As the only requirement was being a web site and we both knew absolutely nothing about on how to create single page applications, we looked around and eventually chose AngularJS.

We didn’t start from scratch, though. There was a prototype from last semester from another team we used for inspiration. The only problem with the project was being written as a JavaEE application. The setup, even to make it run locally, would be too complicated for the use case. Furthermore we both didn’t like the idea coding in Java; Alex already using Java at work and wanting to expand his knowledge, and Markus trying to avoid Java for various reasons.

About the Project

The web site should list information about all kinds of schools in Berlin. A user should be able to filter and search for schools by different criteria. The intention of the university course and the project is to learn about OpenData and Data Journalism. The city of Berlin offers a similar service already, although the user experience is just “meh”.

We took the data set the group acquired in the last semester and normalized it even more. We also performed some additional cleaning. For example if a URL was given for a school we checked whether that URL was reachable or not and accordingly removed those URLs that couldn’t be reached.

The Underlying Data Storage

Looking at the problem from a system architects point of view clearly shows relations between many parts. While a school has a single address it might have multiple types, like elementary school and high school. A school probably teaches multiple languages, not only German but English, French, Dutch or Chinese too. Thus a relational data store / database like PostgreSQL or MySQL would be one’s first choices.

Contrary to what a system architect normally would do we chose a technology not intended for the given use case: a vanilla Elasticsearch instance. We chose this technology for several reasons:

We both professionally develop software using SQL databases. Therefore it was unlikely that we would learn anything new while implementing such a easy application.
The data set contains a lot of optional data. So there was a requirement for a “dynamic” data model for the schools.
But most important: we wanted to learn something new that we could potentially use in our further as developers
And last but not least we wanted to figure out if one can only use Elasticsearch, as this is way outside its common use cases.

Import of the Data

The original data source consists of several Excel documents. These were provided by the city of Berlin and have been normalized by our fellow students in the semester before. For easier handling we transformed those files to the CSV format. Later these CSV files are used to import the data into the Elasticsearch instance.

Before doing this we created a data model bringing some of the features of a relational database into Elasticsearch. Additionally this step is used to set up and configure how Elasticsearch handles the data, mainly (not) analyzing and (not) indexing.

{
    "school": {
        "properties": {
            "bsn": {
                "index": "not_analyzed", "type": "string"
            },
            "name": {
                "index": "analyzed", "type": "string"
            },
            "phonenumber": {
                "index": "not_analyzed", "type": "string"
            },
            "wwwaddress": {
                "index": "no", "type": "string"
            },
            "address": {
                "properties": {
                    "plz": {
                        "type": "integer"
                    },
                    "location": {
                        "lat_lon": true, "type": "geo_point"
                    },
                    "district": {
                        "type": "string", "index": "not_analyzed"
                    },
                    "street": {
                        "index": "analyzed", "type": "string"
                    },
                    "name": {
                        "type": "string", "index": "analyzed"
                    }
                },
                "type": "nested"
            }
        }
    }
}

As you can see from the excerpt above we are using Elasticsearch’s Nested Mapping Type. We decided to use this type over inner objects or Parent/Child Types for two reasons:

[First,] each nested doc remains independent, and [one] can perform a query like [address.district=Kreutzberg AND address.plz = 10999] without a problem. [Second], reading is faster than the parent/child because the nested document is stored in the same Lucene block as the main document. [ES13]

Although writing may require re-indexing the entire document, this is no problem for our use case as the data is imported once.

The actual import was done by a small Python script using Click that connects to the Elasticsearch instance creates the necessary document structure and later imports the different data types (base data, address data, school profiles, etc.).

The Front-End

We are no designers. But we wanted to create a web-page that is both faster and more functional than what the original web page of the city of Berlin offers. We did not target mobile browsers in particular but if we could support them in a reasonable way, then we would do that. Markus is a fan of Zurb Foundation, therefore we decided to use this CSS framework as a starting point for our layout.

Since our data-store is an Elasticsearch instance we had the possibility to retrieve data directly from the browser via JSON requests. Therefore we decided to create a single page application and eventually chose AngularJS. We also delved a little bit into the world of Website front-end development tooling by using the current state of the art Bower, Grunt and Compass tooling.

Our app has three important components:

The filter form
A map view that shows the schools matching the current filter
A detail page of every school

The Filter

The filter data is dynamically retrieved from Elasticsearch upon page load using a HTTP POST query to the search URL with a body like:

{
    "size": 0,
    "aggs": {
        "nested": {
            "aggs": {
                "districts": {
                    "terms": {
                        "field": "address.district",
                        "order": {
                            "_term": "asc"
                        },
                        "size": 0
                    }
                }
            },
            "nested": {
                "path": "address"
            }
        },
        "branches": {
            "terms": {
                "field": "branches",
                "order": {
                    "_term": "asc"
                },
                "size": 0
            }
        }
    }
}

The idea is to aggregate all distinct values available for various fields. The nested block does that for the district (as this is a nested object), the branches block exemplary shows how it is done for direct attributes.

The "size": 0 definition in the outer block tells Elasticsearch to not return any results entries. Inside the aggregation definition it makes Elasticsearch return all distinct values.

The Map

The map component is responsible to display the result of a filter operation. The map is shown using the Openlayers JavaScript library.

The Detail Page

The detail page is quite straightforward. One requirement we defined for the detail page was, that we should be able to provide a deep linking option. This was actually quite easy to implement using the ngRoute module of AngularJS.

The URL is defined to look like this: “domain.de/#/schools/BSN”. The BSN is a unique identifier for each school in Berlin (we assume it stands for Berlin School Number). The ngRoute module allows to specify parameter captures in the route definition, so it’s very easy to access parts of the current URL in the JavaScript code. The route definition for the school detail page is:

mod.config(['$routeProvider', function ($routeProvider) {
    $routeProvider
    // ...
    .when('/schools/:schoolId', {
        templateUrl: 'views/school.html',
        controller: 'SchoolCtrl'
    })
    // ...
}]);

Using the schools identifier we make a simple lookup in Elasticsearch and get the document for the school. Since Elasticsearch returns data in JSON format we can just set the returned value in the scope of the detail page controller, the layout will then be automatically updated by AngularJS.

Deployment

As already stated above, we chose a way for the implementation that lets us circumvent the usage of an application server (as it would be needed for Java or Python). Instead the page only requires a web server and Elasticsearch to run.

Nginx Setup

An exemplary Nginx server config can look like this:

server {
    listen        [::]:80;
    server_name   example.com;

    gzip          on;
    gzip_types    *;

    # Config location
    location /config.json {
        alias   /var/www/config.json;
    }

    # Permit GET and POST to Elasticsearch on a certain index ...
    location ~* /_es/school/([^/]+)/_search {
        limit_except GET POST {
            deny   all;
        }

        rewrite            /_es/(.+) /$1 break;
        proxy_pass         http://127.0.0.1:9200;
        proxy_set_header   Host            $host;
        proxy_set_header   X-Real-IP       $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # ... and deny everything else
    location /_es {
        deny all;
    }

    # Public part of the website
    location / {
        root    /var/www/htdocs/public/;
        index   index.html;
    }
}

By only allowing GET and POST and restricting those queries to a limited URL pattern we can make sure nobody can remove or add some data or even drop the index.

Elasticsearch Setup

network.host: 127.0.0.1
path:
    conf: /etc/elasticsearch
    data: /var/lib/elasticsearch
    logs: /var/log/elasticsearch
    work: /tmp/elasticsearch
script.disable_dynamic: true

Apart from the protections of Elasticsearch mentioned above, it is highly recommended to disable dynamic scripting as this would potentially expose the entire server to the outside world. Setting the Elasticsearch network host to 127.0.0.1 is also required. Otherwise people could connect directly do Elasticsearch and any of the Nginx protections wouldn’t matter.

Website Config

{
    "elasticsearch": {
        "index": "school",
        "host": "http://example.com/_es"
    },
    "heatmap" : {
        "data": "heatmap.json"
    },
    "map" : {
        "feature_bubble": "/views/inc/map_feature_bubble.html"
    }
}

Sources and Additional Reading

[ES13]

Zachary Tong. Managing Relations inside Elasticsearch. February 20, 2013 http://www.elasticsearch.org/blog/managing-relations-inside-elasticsearch/

The conference island — DjangoCon Europe 2014

2014-06-05T00:00:00+02:00

It has been a while since I had the time to write another blog post. As you might have noticed, I’m an organizer of 2014’s EuroPython in Berlin. My job as the web team leader requires some responsibility I’m eager to fulfill, especially for a conference where we expect up to 1.200 attendees. Furthermore, this job is a sideline to my regular job, being a computer science student.

About a month ago, however, I took the time to attend the DjangoCon Europe. After an incredible circus edition 2013 in Warsaw, this year’s organizers stepped up to the plate to host another amazing conference. While everybody knew DjangoCon Europe 2014 was going to be in France from the beginning, it took some months until the announcement was made that the event would happen somewhere on the Côte d’Azur. The community, though, was kept in the dark about the exact location for a few more months. Eventually the Île des Embiez, a small island between Toulon and Marseille, was announced and djangoisland born.

Getting there

Getting to this little island on Monday before official conference start was kind of a challenge for me. Although I’m living in Berlin, there are not many (fast) ways to get to a bigger city in southern France. I took a flight via Brussels in the early morning at about 7am and arrived in Marseille at about 12pm plus some waiting time at baggage claim. That’s actually not that bad. But getting from Marseille to this little island in the middle of nowhere was hard:

From the Marseille Provence Airport (MRS) to Marseille Saint-Charles train station by bus. Unfortunately the shuttle bus we could have taken if the plane hasn’t been delayed was already gone.

From Saint-Charles we took a one hour train ride to Toulon train station. Sadly we missed the connection bus there which would have taken us to the harbor in La Brusc by a few minutes. But having about 2 hours free time in Toulon wasn’t that bad. We walked through the city to the harbor and grabbed some lunch.

Back at the central bus station we met quite a lot other DjangoCon attendees travelling with us to the port in La Brusc where the ferry brought us to the island.

At about 6pm we arrived at the hotel. After I checked into my room I had an amazing view from the balkony.

The Island

[The Île des Embiez] has a surface area of 0.9 square kilometres (0.35 sq mi), with a coastline of 6 kilometres (3.7 mi). The island is 57 metres (187 ft) high at its highest point. [WIKI]

The sunny and warm weather animated me and many others to explore the island. Here are some photos that capture some nice views.

The Conference

There were many talks spanning all kinds of topics, database related topics next to front-end recommendations. Explaining where to look for performance optimizations and why Django is not necessarily a monolithic web-framework. The complexity covered by those talks varied quite a log. There were talks for beginners giving hints on how to work with generic classed based views and talks for experts on what one might experience when ripping out some parts of Django.

Advanced Web Typography by Idan Gazit
Django minus Django by Jacob Burch & Jacob Kaplan Moss
Frequently missed performance gains by Frank Wiles
From __icontains to search by Honza Kral
Good schema design - and why it matters! by Andrew Godwin
Introduction to docker by Amjith Ramanujam
Really, Really Fast Django by Christophe Pettus
The future of PostgreSQL in Django by Marc Tamlyn
Where the wild things are by Aymeric Augustin

Bottom Line

It was an amazing conference and I’m glad I was able to attend. I met many people from the community. Having interesting discussions on all kind of topics, both Django and not Django related.

As highlight at the end of the conference, my pull-request for #21905 got merged!

License

The photos in this blog post are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License..

The DjangoCon 2014 logo is from http://2014.djangocon.eu/

[Wiki]

http://en.wikipedia.org/wiki/Embiez

EuroPython website sprint a success - many new features implemented

2014-03-30T00:00:00+01:00

Last weekend the web team met in Berlin (and was supported by one member remotely) in the rooms of Veit Schiele Communications GmbH in order to bring the EuroPython 2014 website software forward. The team under the leadership of Markus Holtermann implemented the following new features:

A short while ago we made some heavy changes to the main menu. Unfortunately the user experience wasn’t the way we expected it to be. Thus we got some comments on the EuroPython mailing list. It took us some time to decide how to change the way the menu should work. Eventually, a click on a main menu item only toggles the menu as of now (GitHub issue #103)
We want to generate the badges (and other stuff for the conference) directly from the user data you provide on the website. This presented us with some kind of challenge as you will see in the next bullet point, too. We decided that the singe point of truth for the user data is the user profile. We therefore added a field where you can enter your interests. (GitHub issue #105)
As said before, the single point of truth for the user data is and will be the user profile. Unfortunately, during the purchase process of a ticket, we only have information about the buyer, not any of the intended ticket users. Although we didn’t change this part in the update (and we won’t change it in the future), you are now able to assign a ticket to another user. This is incredible useful for those buyers ordering multiple tickets, eg. for their colleagues or friends. To assign the ticket to somebody else, ask him/her for the user name and assign the ticket to him/her from your purchase view. If the intended user doesn’t have a account yet, ask him to create one. All tickets that are not assigned to somebody will use the first and last name given during purchase. (GitHub issue #101)
Since many attendees are not from Germany, we looked for a way to hand out SIM cards. Due to legal restrictions this is not that simple in Germany (you normally need to present a photo ID). Fortunately we found a reseller where we become some kind of reseller ourselves (it’s a bit more complicated than that ;) ). Hence we will offer SIM cards within the next days that you can buy beforehand and you can pick up during check-in (more details soon). (GitHub issue #100)
When you modify your profile and try to upload a new avatar, but there are errors in some other fields, the avatar gets lost. We solve this by adding front-end validation to this and many other forms. (GitHub issue #47)
If you are looking for a job (in your real live, not the one on the internet :D), or if you think about getting a new job for whatever reason, you can now opt-in (and later opt-out) of job offers by our sponsors. We will not hand over any of your data to the sponsors. The sponsors have to give us their offer and we will send it only to those users interested in job offers. (GitHub issue #78)
The list of your purchases now shows canceled purchases too. (GitHub issue #102)

Apart from those changes interesting to you as an attendee, loads of other changes made it into production. If you are interested, have a look at our repository on GitHub: https://github.com/EuroPython/djep

All new features went into production today with an update of the portal software.

Thanks a lot to the web team and all other sprinters for their dedicated work on the EuroPython 2014 web software.

Source: blog.europython.eu, Mar 30, 2014, 20:30pm CEST

EuroPython 2014 Conference in Berlin at the bcc Berlin Congress Center

2013-12-24T00:00:00+01:00

The next EuroPython conference will be held from July 21 - 27, 2014 at the bcc Berlin Congress Center in the heart of Berlin. The EuroPython conference is the second largest global conference for the popular programming language Python, after the PyCon US conference in North America. The bcc is located at the central Alexander Square. It offers ideal facilities for a developer conference on 3,000 m2. The organizers are expecting up to 1,200 participants, continuing the growth of the EuroPython conference series. The very successful EuroPython this year attracted almost 900 Python enthusiasts. It was hosted in Florence, Italy for the third time running.

Berlin offers ideal conditions for a large international developer conference. The city with its about 2,500 young businesses has a large and fast-growing start-up scene. 300 universities, polytechnics, research institutes and technology parks underline Berlin’s importance as one of Europe’s leading science and research locations. Thanks to its rich cultural landscape with around 180 museums, renowned international art collections and precious works of art as well as a lively art and music scene, Berlin is one of the most vibrant and vividly creative metropolitan centers in Europe. Its attractions and excellent national and international transport links have already put Berlin among the top three in the international conference and congress region rankings.

The conference will be organized locally by the Python Software Verband e.V., the Python Users Berlin, the Django User Group Berlin and the PyLadies Berlin, in cooperation with the EuroPython Society for the Python community. With five lecture tracks and two tutorial tracks running in parallel, the EuroPython conference offers an attractive schedule for all attendees.

The EuroPython 2014 logo expresses what is on the organizer’s minds, which is getting people from very different areas of expertise together to communicate. Python is used as a programming language in industrial and service enterprises as well as in research and education. The EuroPython brings together developers who work together on fascinating tasks and as a result have found unique solutions to their problems.

About Python

Python is an interpreted high-level language. In the TIOBE index Python has featured among the top 10 of favorite programming languages for years. Python is ideal for development. It is a lean language that is easy to learn for both developers and users. The language was created in the late 1980s by Guido van Rossum. Today, the language is developed by a large open source community and backed by the Python Software Foundation (PSF), which owns the copyright to Python.

About the Python Software Verband e.V.

The Python Software Verband e.V. represents the interests of the German-speaking Python community. It is a German non-profit organization that focuses on promoting the Python programming language and supports activities of different Python interests groups. Its members are involved in a wide range of Python activities including Python core development and many Open Source projects.

About the EuroPython Society

The EuroPython Society (EPS) is a Swedish non-profit organization which holds the rights to the EuroPython conference series and trademarks. It coordinates with local organizers to run EuroPython conferences. The EPS was formed in 2004 with the mission to turn EuroPython into a successful Python conference series for the years to come.

About the Python Users Berlin

Founded in 2002, Python Users Berlin holds monthly meetings for presentations and talks about all facets of the Python programming language. Python Users Berlin currently has around 330 members.

About the Django User Group Berlin

The Django User Group Berlin was founded in 2009. Its approximately 130 members meet monthly for talks related to the Python web framework Django.

About the PyLadies Berlin

PyLadies is an international movement that motivates and supports women to engage with and take an active role within the Python community as developers. PyLadies Berlin has around 100 members.

Source: ep2014.europython.eu, Dec 12, 2013, 11:30pm CET

EuroPython 2014 im bcc Berlin Congress Center

2013-12-24T00:00:00+01:00

Die nächste EuroPython wird vom 21. bis 27. Juli 2014 im bcc Berlin Congress Center mitten in Berlin stattfinden. Die EuroPython ist nach der PyCon US weltweit die zweitgrößte Konferenz für die beliebte Programmiersprache. Das bcc am Alexanderplatz bietet mit seinen 3000 Quadratmetern ideale Voraussetzungen für eine Entwicklerkonferenz. Die Veranstalter rechnen mit bis zu 1200 Teilnehmern, entsprechend der Wachstumsrate dieser Konferenzreihe. Die bereits sehr erfolgreiche EuroPython hat dieses Jahr 900 Python Enthusiasten zusammengeführt. Sie fand zum dritten Mal in Florenz, Italien statt.

Berlin bietet ideale Bedingungen für eine große internationale Entwicklerkonferenz. Die Stadt hat mit rund 2500 jungen Unternehmen eine große und schnell wachsende Startup-Szene. Rund 300 Universitäten, Fachhochschulen, Forschungsinstitute und Technologieparks machen Berlin zu einem der wichtigsten Wissenschafts- und Forschungsstandorte in Europa. Dank seiner reichhaltigen Kulturlandschaft mit rund 180 Museen, die bedeutende internationale Sammlungen und Kunstschätze beherbergen, sowie seiner lebendigen Kunst- und Musikszene gilt Berlin als eine der pulsierendsten und kreativsten Metropolen in Europa. Die Attraktivtät der Stadt und ihre hervorragende nationale und internationale Verkehrsanbindung hat Berlin auf Platz 3 der internationalen Tagungs- und Kongressmetropolen katapultiert.

Die Konferenz wird vor Ort vom Python Software Verband e.V., von den Python Users Berlin, der Django User Group Berlin und den PyLadies Berlin organisiert. Mit fünf Vortrags- und zwei parallelen Tutorial-Tracks bietet die EuroPython ihren Besuchern ein prall gefülltes Programm.

Das Signet der EuroPython 2014 zeigt, worum es den Veranstaltern geht. Sie wollen Menschen aus ganz unterschiedlichen Bereichen miteinander ins Gespräch bringen. Python wird als Programmiersprache in Industrie- und Dienstleistungsunternehmen sowie in der Forschung und Lehre eingesetzt. Auf der EuroPython kommen Entwickler zusammen, die an faszinierenden Aufgabenstellungen arbeiten und dabei einzigartige Lösungen gefunden haben.

Über Python

Python ist eine interpretierte höhere Programmiersprache. Sie rangiert im TIOBE-Index seit Jahren unter den zehn beliebtesten Programmiersprachen weltweit. Python eignet sich ideal für agile Entwicklung. Es handelt sich um eine schlanke, leicht zu lernende Programmiersprache, die die Grenzen zwischen Anwendern und Entwicklern verwischt. Die Sprache wurde in den späten achtzigern von Guido van Rossum entwickelt. Heute wird die Sprache von einer großen Open-Source-Community weiterentwickelt und von der Python Software Foundation (PSF) abgesichert, die die Namensrechte an der Marke Python hält.

Über den Python Software Verband e.V.

Der Python Software Verband e.V. vertritt die Interessen der deutschsprachigen Python Community. Er ist eine Non-Profit Organisation nach deutschem Recht, deren Fokus auf die Förderung der Programmiersprache Python und die Unterstützung der Aktivitäten verschiedenster Interessen-Gruppen rund um Python ausgerichtet ist. Seine Mitglieder sind an einer breiten Palette von Aktivitäten inklusive Python Kernentwicklung und vielen Open-Source-Projekten beteiligt.

Über die EuroPython Society

Die EuroPython Society (EPS) ist eine schwedische gemeinnützige Organisation, welche die Rechte an der Europython Konferenz Serie und der Marke hält. Sie koordiniert gemeinsam mit der lokalen Organisation die Ausführung der Europython Konferenzen. Die EuroPython Society wurde 2004 gegründet, um in den darauf folgenden Jahren eine erfolgreiche Python Konferenz zu gewährleisten.

Über die Python Users Berlin

Die Python Users Berlin gründeten sich 2002. Die Gruppe trifft sich monatlich zu Vorträgen und Diskussionen zu Themen rund um die Programmiersprache Python. Zurzeit haben die Python Users Berlin rund 330 Mitglieder.

Über die Django User Group Berlin

Die Django User Group Berlin wurde 2009 gegründet. Die rund 130 Mitgliedertreffen sich im Monatsrhythmus zu Vorträgen rund um das Python-Web-Framework Django.

Über die PyLadies Berlin

Die PyLadies bilden eine internationale Bewegung, die Frauen motiviert und unterstützt, innerhalb der Python-Community als Entwicklerinnen in führenden Positionen aktiv zu werden. Die PyLadies Berlin haben zurzeit rund 100 Mitglieder.

Quelle: ep2014.europython.eu, 12. Dez 2013, 23:30 CET

bakery — Where cookies are made!

2013-10-01T00:00:00+02:00

It has been some time since my last blog post. So here’s an update. This is about a great event I participated in last weekend, the djangodash 2013.

The djangodash is a coding contest where teams have 48 hours time to start and sprint a project. As this dash is called djangodash, the Django must be used. Furthermore the teams are limited to at most 3 members and a couple of other rules.

So, what project were my friend and I working on? It’s called bakery. You can find it on github as well as a live version at djangodash2013.webshox.org. But what does bakery actually do and what is it? In simplified terms bakery is an index for cookiecutter templates. In longer terms, bakery offers an API to create, find, modify, update and share cookiecutter templates at a central point to clear up the mess with manually updated readme files providing an incomplete list of templates. Additionally bakery offers some neat features such as forking a template to your own github account or voting a template up. We are working on many more planned features, e.g. automatically updating the index, directly editing the cookiecutter.json file on the index.

So, what are these cookiecutter templates and what is cookiecutter? cookiecutter itself is a tool that, given a project template, bootstraps a project with its directory structure and files (including their content) based on a set of context variables. To see what these templates (aka cookiecutters) look like, just have a look at the list of available cookiecutters.

Showing BVG Departures In Your Office

2013-06-14T00:00:00+02:00

Yesterday evening I gave a lightning talk at the Python User Group Berlin called “Showing BVG Departures In Your Office“. You can download the slides here.

The whole idea to the project came up when our student organization (Freitagsrunde) had to relocate to a new building at the end of 2012.

Since our new student organization meeting room is quite near to the next bus stop (bee-line), but the actual distance is way longer and we have to cross several doors, waiting in the room and heading to the bus stop when the bus arrives is not going to work.

As computer scientists we had to come up with a solution to somehow grab the departures of the bus stop and show them in the room.

At first the information retrieval was hard-coded and it was only possible to get the departures for a single bus stop. After a short time we got the feature request to also display the departures of nearby subways and other public transport. We ended up rewriting the entire application and created a library we then called bvg-grabber.

The quite simple API allows us to use both information provided by the BVG: the actual departures which, in that days, only returned departures about busses, and the scheduled departures as they are written on the time tables:

A class inheriting from QueryApi must implement a call() function that returns a Response object.

class QueryApi(object):
    """Performs the requests to the data source"""

    def call(self):
        """Needs to return a Response"""
        raise NotImplementedError("The inheriting class needs to "
                                  "implement the call() method!")

A Response object must contain a state (True/False for success/failed request) and, for a successful request, the departing station and a list of Departures, or, for a failed request, the error that occurred.

class Response(object):
    """Returned by a QueryApi, contains list of Departures"""

    def __init__(self, state, station=None, departures=None,
                 error=None):
        self._state = state
        self._departures = [(station, departures)]
        self._error = error

    @property
    def departures(self):
        return self._departures

Finally, a Departure defines when a certain line leaves at a certain start station and at which station the ride will end.

@total_ordering
class Departure(object):
    """Start and end station, next departure time"""

    def __init__(self, start, end, when, line, since=None,
                 no_add_day=False):
        # Some magic happens here

As we are using the actual and the scheduled information from the BVG as you can see above, here are 2 short examples how to use them:

In [1]: from bvggrabber.api.actualdeparture import ActualDepartureQueryApi

In [2]: resp = ActualDepartureQueryApi("Ernst-Reuter-Platz").call()

In [3]: resp.departures
Out[3]:
[('Ernst-Reuter-Platz',
  [Start: Ernst-Reuter-Platz, End: S+U Zoologischer Garten, when: 13:56, now: 13:54, line: Bus 245,
   Start: Ernst-Reuter-Platz, End: S+U Zoologischer Garten, when: 14:01, now: 13:54, line: Bus X9,
   Start: Ernst-Reuter-Platz, End: S+U Zoologischer Garten, when: 14:05, now: 13:54, line: Bus M45,
   Start: Ernst-Reuter-Platz, End: Johannesstift, when: 13:54, now: 13:54, line: Bus M45,
   Start: Ernst-Reuter-Platz, End: Flughafen Tegel, when: 13:55, now: 13:54, line: Bus X9,
   Start: Ernst-Reuter-Platz, End: Johannesstift, when: 14:01, now: 13:54, line: Bus M45])]

In [4]: from bvggrabber.api.scheduleddeparture import ScheduledDepartureQueryApi

In [5]: resp = ScheduledDepartureQueryApi("Ernst-Reuter-Platz").call()

In [6]: resp.departures
Out[6]:
[('Ernst-Reuter-Platz',
  [Start: Ernst-Reuter-Platz, End: Johannesstift (Berlin), when: 13:54, now: 13:54, line: Bus  M45,
   Start: Ernst-Reuter-Platz, End: Hertzallee (Berlin), when: 13:56, now: 13:54, line: Bus  245,
   Start: Ernst-Reuter-Platz, End: S+U Pankow (Berlin), when: 13:56, now: 13:54, line: U2,
   Start: Ernst-Reuter-Platz, End: Hertzallee (Berlin), when: 13:57, now: 13:54, line: Bus  M45,
   Start: Ernst-Reuter-Platz, End: U Theodor-Heuss-Platz (Berlin), when: 13:58, now: 13:54, line: U2])]

I’m looking forward to your ideas and feature requests.

Markus

Mirroring gitolite

2013-02-18T00:00:00+01:00

A long time ago I wrote an article about how to install gitolite. Yesterday an article popped up in my feed reader that shows a short script on how to mirroring gitolite servers.

Since I wrote my custom script once I had set up gitolite, I was a bit curious about the differences to my script today. So I checked the referred script and noticed some problems that probably occur over the time:

New branches on the remote are not fetched
New tags on the remote are not fetched
Using working copies results in way larger mirrors

As one can see, the script lacks some major features.

#!/bin/bash
# Copyright (c) 2011-2013 Markus Holtermann
# MIT License: http://download.markusholtermann.eu/MIT.txt

BACKUP_DIR=/data/git-mirror/
REMOTE=git@git.example.com

if [ ! -d "${BACKUP_DIR}" ] ; then
    mkdir -p "${BACKUP_DIR}"
fi

cd "${BACKUP_DIR}"

for repo in $(ssh ${REMOTE} 2> /dev/null | grep -e "^\s\+R" | sed -e 's/\r//g' | awk '{print $3}')
do
    if [[ -d ${repo}.git ]] ; then
        cd ${repo}.git
        echo "Updating ${REMOTE}:${repo} ..."
        git fetch --all
        echo " ... done"
        cd -
    else
        echo "Cloning ${REMOTE}:${repo} ..."
        git clone --mirror ${REMOTE}:${repo}.git
        echo " ... done"
    fi
done

The magic to prevent working directories lies in the --mirror parameter while cloning a repository. To get notified about new branches and tags on the remote side, I use git fetch --all. The man (1) page to git-fetch describes it as

Fetches named heads or tags from one or more other repositories, along with the objects necessary to complete them. […]

However, the script above does not remove branches/tags that don’t exist on the remote any more. Since I use the script only for backup purposes, I don’t need that feature. But I would just remove all branches (local and remote) from the local mirror and would run git-fetch afterwards.

Enjoy the script.

Pictures from NYC

2013-01-15T00:00:00+01:00

Yesterday a friend of mine asked around if someone has ever been in New York City. I replied and promised him to show some of my favorite pictures from my last trip to NYC in summer 2009. Looking through the more than 1000 photos from those days, I picked the following 8.

Enjoy them!

These images are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

KDE 4.10 RC1 is out

2012-12-21T00:00:00+01:00

Two days ago the KDE team released the first release candidate of the Plasma Workspaces towards version 4.10. The biggest changes that directly catches ones eyes are the graphical modifications and improvements. For me they look much more seamless. I also like the new wallpaper. Looking under to hood, there are many changes. I just want to point out my most favorite ones:

A new QtQuick based screen locker.
Scriptable effects – One can easily write desktop effects without knowledge of C/C++
The notification applet is now based on QML and feels much smoother
The Nepomuk Indexer has been rewritten and is much faster as far as I can see.

Of course there are many, many more changes.

LVM on secondary LUKS device

2012-11-25T00:00:00+01:00

Today I ran into a problem when I set up my new SSD. I have two fully encrypted disks (one SSD and one HDD) and two volume groups (VGs) on top. Each disk has its own VG. I thought about the following partition scheme. sda is the SSD and sdb the HDD:

/dev/sda2            → LUKS → /dev/mapper/luksmain
/dev/sdb1            → LUKS → /dev/mapper/luksdata
/dev/mapper/luksmain → LVM  → /dev/mapper/main
/dev/mapper/luksdata → LVM  → /dev/mapper/data

And from these block devices set up the following file systems:

/dev/mapper/main-root     /
/dev/sda1                 /boot
/dev/mapper/main-home     /home
/dev/mapper/data-var      /var
/dev/mapper/data-music    /home/Music
/dev/mapper/data-videos   /home/Videos
/dev/mapper/main-mysql    /var/lib/mysql

As one can see, the logical volume (LV) data-var is required by the system. It contains e.g. the logs or the package cache.

To use it during startup, enable the lvm-on-luks systemd service:

systemctl enable lvm-on-luks.service

[CloudRAID] 6. Conclusion and Outlook

2012-11-24T00:00:00+01:00

This post is the last post of the blog series about the student research paper CloudRAID.

This is the last part of the student research paper about CloudRAID. Florian summarizes the pros and cons of our implementation. Furthermore, he shows some ideas that could make CloudRAID more convenient, e.g. adding a search possibility.

[CloudRAID] 5. Benchmarks

2012-11-21T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

5. Benchmarks

Since the performance is an integral part and a key aspect of the software and even more one of the most important piece, the following chapter will show some performance tests and benchmark results.

5.1. Environments

In order to make the benchmarks reproducible this chapter will set up some benchmark environments. They define the hardware as well as the software that is used for the benchmark tests.

Environment 1

CPU	Intel® Core™ 2 Duo CPU P8600 @ 2.40 GHz
RAM	4 GiB
Hard disk	Samsung HM320II, 16 MiB Cache
File system	ext4, Block size: 4096 Byte
Disk Encryption	LUKS, cipher: aes-xts-plain, 512 bits
Disk Management	LVM: 2.02.88(2), Library: 1.02.67 Driver: 4.22.0
Operating System	Arch Linux 64bit

Environment 2

CPU	Intel® Core™ 2 Duo CPU E6600 @ 2.40 GHz
RAM	2 GiB
Hard disk	2x Samsung SpinPoint HD501LJ, 16 MiB Cache
File system	ext4, Block size: 4096 Byte
Disk Encryption	LUKS, cipher: aes-xts-plain, 512 bits
Disk Management	LVM: 2.02.88(2), Library: 1.02.67 Driver: 4.22.0
RAID	Software RAID 1, mdadm 3.2.3
Operating System	Arch Linux 64bit

Environment 3

CPU	AMD Phenom™ II X6 1090T
RAM	16 GiB
Hard disk	Seagate ST31000524AS, 32 MiB Cache
File system	ext4, Block size: 4096 Byte
Operating System	Arch Linux 64bit

Environment 4

CPU	Intel® Core™ i7 920 @ 2.67 GHz
RAM	8 GiB
Hard disk	Kingston SH100S3 120G (SSD)
File system	ext4, Block size: 4096 Byte
Operating System	Arch Linux 64bit

5.2. Comparison of Java, Python, and C

One of the key functions of CloudRAID is to ensure data-security and data-safety. Therefore CloudRAID uses the previously explained ARC4 encryption standard andthe RAID level 5 implementation. Since this time critical operation is used during each CloudRAID cloud interaction, a run-time evaluation during the prototype state was interesting.

To make this key component fast, there have been prototypes in Java, Python 2 and C. All three implementations were working with the same algorithm but of course had a big differences when comparing their run-time.

The following table shows the run-time for the “split” process on a 15 MiB file using bitwise RAID level 5 without encryption and without hash calculation.

Language	Runtime[sec]
C (Clang)	1.526
C (Clang -O3)	1.123
C (GCC)	1.975
C (GCC -O3)	1.490
Java	155.000
Python2.6	122.300
Python2.7	121.909
PyPy 1.6	20.967

Table 12: Prototype benchmarks for Java, Python and C

As one can see from the previous table the run-time for Java is more than 2.5 times as long as for CPython2, more than 7 times longer than for PyPy and 77 times longer than the native C implementation. The speed of CPython2 and PyPy is most likely due to the small overhead between the Python interpreter and the underlying native code, while the Java implementation must leave the Java Runtime Environment (JRE) for each read and write operation, since they are also natively implemented and a JNI call is relatively expensive. The speed advantage of the PyPy based execution is probably based on the just-in-time compiler that can increase the speed many times.

The results from this prototype gave the crucial factor to implement the RAID feature in native C – independent of the programming language that would be used for the remaining part of the software. To get better benchmarking results for C there was a test with a 1.9 GB file including compiler optimization like the -O3 flag:

Language	Runtime[sec]
C (Clang -O3)	1.123
C (GCC -O3)	1.490

Table 13: Large file benchmarks for GCC and Clang

This table clearly shows, that Clang is doing a better optimization than GCC does. But nevertheless, using the JNI, the performance of both implementations changed and GCC was slightly faster than Clang.

5.3. RAID Level 5 Benchmarks

To get the performance of the final implementation, some benchmarks on the environments, that have been defined in 6.1 on page 54, have been performed. Therefore files of the sizes 10 kib, 100 kiB, 1 MiB, 20 MiB, 50 MiB, 100 MiB, 250 MiB, 500 MiB and 1 GiB have been created, split and merged. Every complete process (creation - split - merge) has been run eight times for each size on each benchmark environment. On the third environment additional tests for the sizes 2 GiB, 3 GiB, 4 GiB and 5 GiB have been performed to find and show anomalies for large files.

5.3.1. Split

Figure 1 shows the average run-time for the split process in the given benchmark environments. One can see that for the files 100 kiB to 100 MiB the run-time increases mostly linear to the file size for every environment. For the range from 10 kiB to 100 kiB the run-time even slightly decreases, relative to the file size, probably caused by the overhead opening the five files.

Figure 1: Runtime for split in various benchmark environments

Using the T400 notebook with above specifications, the run-time increases for the files larger than 100 MiB. There is no proved explanation for this phenomenon yet, but it has been appeared during all benchmark runs and is not just a single outlier distorting the statistics.

Environment 2 keeps a linear run-time up to files of size 500 MiB and is much slower for files of size 1 GiB.

For the third and forth environments the run-time is linear for all files, even for the huge files. As one can see in figure 2, the relative run-time increases for files of size 2 GiB and larger as well.

Figure 2: Split statistics for Tower 2 for huge files

5.3.2. Merge

After a file has been split into its three device files and regarding information has been stored in the meta data file, the benchmark tool combines these device files to the original file. Again one can see from figure 3 that the overall process is mostly linear, except for the range of very small and very large files. Similar to the split process the hard drive accesses to open the files is probably the main reason for the decreasing relative run-time between 10 kiB and 100 kiB.

Similar to the split process, environment 3 shows an increasing relative run-time for files greater or equal to 2 GiB as one can determine from figure 4 on page 60.

Figure 3: Runtime for merge in various benchmark environments

Further comparisons of the benchmark environments show differences in the hard drive caches. While the hard drives in the environments 1 and 2 have a cache size of 16 MiB the hard disks from environment 3 provide 32 MiB. Comparing the ages of the used hard disks lead to similar assumptions. While the hard disks for the first and the second environment are about three, respectively five, years old, the age of hard disks used in environments 3 and 4 are about one year.

Besides that, the storage devices for the primary two environments are encrypted while the other two hard drives are plain. But due to the implementation of the underlying encryption as part of the Device Mapper of the Linux Kernel, the hardware access is fully transparent and does not do any further hits or accesses on caches. Thus this has probably no or at least not a high impact on the hard drive throughput and only affects the Central Processing Unit (CPU) usage.

[The] Device-mapper uses block devices, such as hard disks and flash-storage devices, and represents them as other block devices by adding various features. The LVM for example provides flexible partitioning of the block device, while dm-crypt adds a transparent symmetric encryption layer to the device by using the Linux in-kernel Crypto API. [Hol12]

Finally, the slight difference in the throughput between the SSD and the third environment is probably caused by the amount of available memory. While the former benchmark environment can use up to 8 GB minus the memory used by the OS itself, the latter has twice that memory available (minus those used by the OS). Thus the files are being cached in in-memory and are not even written to disk. But unfortunately there is no way to check whether the data has been written to disk. Another influence leading to the performance disadvantages of in the forth environment might be caused by the SATA controller.

Figure 4: Merge statistics for third environment for huge files

5.3.3. Comparison

While the figures above only show the compared speed for either split or merge for all four benchmark environments, figure 5 illustrates the relative speed between the split and merge processes for the benchmark environments. One can see, that the split process takes much more time than the merge process for the same file size. This is most likely caused by the number of bytes that are written to the hard disk. While a call to split_file() with an input file of size n writes 1.5 × n + 518 bytes and reads n bytes, the regarding merge done by merge_file() only writes n bytes but reads 1.5 × n + 518 bytes. Since reading is generally much faster than writing this explains the speed of merge processes.

Figure 5a - 5d: Split and merge comparisons for environments one to four

Sources

[Hol12]

Markus Holtermann. Bachelorthesis: Testing Approach for an in-kernel Crypto Functionality – For Linux on System z, June 4, 2012. (unpublished)

[CloudRAID] 4. Implementation (Continuation 2)

2012-11-18T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

After the implementation of the CloudRAID backend has been introduced and the specification of the REST API endpoints is published, Florian is describing the compression API and the client application on his blog in a new post.

[CloudRAID] 4. Implementation (Continuation)

2012-11-15T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

4. Implementation

4.2. RESTful API Endpoint Specifications

4.2.1. URL Mapping for API Endpoints

The default CloudRAID package contains a RESTful HTTP API. This is realized using Java Servlets. A Java web and application server will be used to deploy the servlet. To activate the servlet this is bound to a base path that will be required to be prefixed to the Uniform Resource Locator (URL) path. The actual functionality provided by this servlet will then be bound to sub paths as described in chapters 4.2.2 to 4.2.4 (see below). The basic concept of how the mapping between the sub paths and URL parameters is done, has been taken from the Django URL dispatcher.

The complete pattern and mapping implementation is done in the class RestApiUrlMapping within the package de.dhbw_mannheim.cloudraid.api.impl and is part of the CloudRAID-REST bundle. The RestApiUrlMapping constructor takes two to four parameter of various type as one can see in listing 9 below:

The first parameter must be the pattern that needs to be matched. This can either be a java.lang.String or a regular expression pattern of type java.util.regex.Pattern. If a String is passed, this is converted to a Pattern and hence must be a correct regular expression. Otherwise a IllegalArgumentException will be thrown.
The second, optional, parameter can be used to restrict a mapping to a specific HTTP request method. This allows a developer to have different callback functions for the same URL path, depending on the HTTP method.
The remaining parameters define the callback function which is called if the URL matches a HTTP request. This can either be a single parameter of type java.lang.reflect.Method or two parameters whereof the first must be of type Class<?> and the second of type java.lang.String. In the latter case, the static function findFunction() will be used to resolve both parameters to a Method. If the (resolved) method is null, this will result in a IllegalArgumentException. If the method needs to be resolved, any failure will raise as NoSuchMethodException or SecurityException.

package de.dhbw_mannheim.cloudraid.api.impl;

public class RestApiUrlMapping {
  public static Method findFunction(Class<?> klass,
    String function);

  public RestApiUrlMapping(Pattern pattern, Class<?> klass,
    String function);
  public RestApiUrlMapping(Pattern pattern, Method function);
  public RestApiUrlMapping(Pattern pattern, String method,
    Class<?> klass, String function);
  public RestApiUrlMapping(Pattern pattern, String method,
    Method function);
  public RestApiUrlMapping(String pattern, Class<?> klass,
    String function);
  public RestApiUrlMapping(String pattern, Method function);
  public RestApiUrlMapping(String pattern, String method,
    Class<?> klass, String function);
  public RestApiUrlMapping(String pattern, String method,
    Method function);
}

Listing 9: The URL mapping API

The actual mapping will be initialized in the constructor of the RestApiServlet as shown in the listing 10. The shipped REST API uses only the RestApiUrlMapping constructor defined in lines 17 and 18 of listing 9 because of its simplicity. But one can use this class to extend the API with own functions. Hence the other constructors are still in the code and will not be removed.

package de.dhbw_mannheim.cloudraid.api.impl;

public class RestApiServlet extends HttpServlet {
  private static ArrayList<RestApiUrlMapping> mappings =
    new ArrayList<RestApiUrlMapping>();

  public RestApiServlet() {
    RestApiServlet.mappings.add(new RestApiUrlMapping(
      "^/api/info/$", "GET", RestApiServlet.class,
      "apiInfo"));
    RestApiServlet.mappings.add(new RestApiUrlMapping(
      "^/file/([^/]+)/$", "DELETE", RestApiServlet.class,
      "fileDelete"));
    // Similar for other URL sub paths
  }
}

Listing 10: The URL mapping initialization

On each request to the RestApiServlet the list of available mappings will be checked for a matching mapping with the path information of the current request and its HTTP method. If a valid mapping exists, the regarding callback function will be executed. This callback function must accept exactly three arguments:

javax.servlet.http.HttpServletRequest req: The current request. This variable will contain session information, the HTTP headers and allows to access the request body.
de.dhbw_mannheim.cloudraid.api.responses.IRestApiResponse resp: The response object that will be returned to the client. It will contain a HTTP response code as defined by the specifications in the following chapters and a payload, depending on the request.
java.util.ArrayList<String> args: The URL patterns may contain matching groups, enclosed by ( and ). The content of each matching group will be added to this parameter and is therefore available to the callback function. The Create, Read, Update, Delete (CRUD) endpoints as defined in chapter 4.2.3 – File Related Function make use of a single matching group that contains the file name of the uploaded file.[/olist]

4.2.2. Version Information

The RESTful API already supports many features. But since the API might change someday, some kind of versioning is needed to tell the client side, which features are supported. This information, along with the version information of the Core-, Metadata- and Configuration-Services and the currently running RAID version and the versions of the activated storage connectors is provided via this URL.

Method:	`GET`
Endpoint:	`/api/info/`
Description:	Displays the version and author/vendor information about various service components, such as the Core service, the cloud storage connectors, the configuration, etc.
Usage:	Used on client site to verify that the client is able to handle the provided API.
HTTP Response:	200 – Success – No special meaning in this response

RESTful API: API and service information

HTTP/1.1 200 OK
X-Powered-By: CloudRAID/0.2
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Server: Jetty(6.1.x)

Core-Service:CloudRAID-Core v0.0.1.prealpha by cloudraid
RAID-Version:CloudRAID-RAID5 v0.0.2prealpha by cloudraid
Metadata-Service:CloudRAID-Metadata v0.0.1.prealpha by cloudraid
Configuration-Service:CloudRAID-Config v0.0.1.prealpha by cloudraid
API-Service:CloudRAID-RESTful v0.0.1.prealpha by cloudraid
Storage-Connector-0:CloudRAID-Dropbox v0.0.1.prealpha by cloudraid
Storage-Connector-1:CloudRAID-AmazonS3 v0.0.1.prealpha by cloudraid
Storage-Connector-2:CloudRAID-UbuntuOne v0.0.1.prealpha by cloudraid
API-Version:0.2

Listing 11: Example HTTP response: header elements and body

4.2.3. File Related Function

All CRUD functions are provided by the following four URLs. The endpoint for all of them is /file/([^/]+)/. Based on the HTTP method, a file is either being created, downloaded, removed or updated.

The Create and Update endpoints have to handle a huge amount of data. Therefore they use the POST and PUT methods respectively. In both cases the file will be trans- mitted synchronously to the CloudRAID service. All ongoing steps are going to be asynchronous. They do not influence the HTTP response to the upload.

On the other hand, the Delete and Download functions are completely synchronous. They use the DELETE and GET HTTP methods. The cloud storage connectors will trigger the deletion or get request to the cloud storage providers and are blocking up to their response or until the regarding network protocol throws a timeout error. From that follows, that a failure to the “outbound” connections from the CloudRAID service may block a client application, or at least the thread the connection runs in, for some time.

Method:	`DELETE`
Endpoint:	`/file/([^/]+)/`
Description:	Calls on this endpoint will trigger a synchronous deletion of the given file from the cloud storage connectors.
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	200 – File has been deleted or does not even exist 401 – The requesting user is not logged in 404 – File cannot be found in the meta data manager 405 – The session identifier has not been transferred via cookie 500 – An error occurred during deleting the file 503 – The session given by the identifier does not exist

RESTful API: file deletion

Method:	`GET`
Endpoint:	`/file/([^/]+)/`
Description:	Calls on this endpoint will trigger a synchronous download of the requested file from the cloud storage, a merge to the original file and download to the client.
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	200 – File has been downloaded 401 – The requesting user is not logged in 404 – File cannot be found in the meta data manager 405 – The session identifier has not been transferred via cookie 500 – An error occurred during downloading the file 503 – The session given by the identifier does not exist

RESTful API: file download

Method:	`POST`
Endpoint:	`/file/([^/]+)/`
Description:	Calls on this endpoint will trigger a synchronous upload of the new file. A successful upload of the file will split and push it to the cloud storage providers asynchronously.
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	201 – File has been uploaded 401 – The requesting user is not logged in 405 – The session identifier has not been transferred via cookie 409 – File already exists 411 – No content length specified 500 – An error occurred while storing the new file 503 – The session given by the identifier does not exist

RESTful API: file upload of a new file

Method:	`PUT`
Endpoint:	`/file/([^/]+)/`
Description:	Calls on this endpoint will trigger a synchronous upload of the existing file. A successful upload of the file will split and push it to the cloud storage providers asynchronously.
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	201 – File has been uploaded 401 – The requesting user is not logged in 404 – File cannot be found 405 – The session identifier has not been transferred via cookie 411 – No content length specified 500 – An error occurred while storing the new file 503 – The session given by the identifier does not exist

RESTful API: file upload of an existing file

While using a client application, the user needs to know, which files exist and can be downloaded and, if the system is under heavy usage, what the states of the uploads are. The following URL shows all files a user has access to and displays those information.

Method:	`GET`
Endpoint:	`/list/`
Description:	Calls on this endpoint will return a list of all files the current user has access to.
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	200 – Success – No special meaning in this response 401 – The requesting user is not logged in 405 – The session identifier has not been transferred via cookie 500 – An error occurred while storing the new file 503 – The session given by the identifier does not exist

RESTful API: list all accessible file

4.2.4. User Related Functions

All connections, except the API version request, need some kind of authorization or provide and initiate it. The authentication is done on user base. The creation, authentication and de-authentication (logout) is performed on the /user/ endpoint namespace. Besides that, a user is able to change the login password, while user deletion requests are not allowed. This is basically due to the fact, that a deletion should trigger a deletion of all files belonging to the user from the cloud storage providers. But since this step is non-reversible and a full data loss will be the consequence, the decision to not implement this function has been made.

Method:	`POST`
Endpoint:	`/user/add/`
Description:	Calling this endpoint will add a new user to the system.
Parameter:	X-Username: `USERNAME` X-Password: `PASSWORD` X-Confirm: `CONFIRMATION`
HTTP Response:	200 – Success – No special meaning in this response 400 – The user name is invalid or missing or the password and its confirmation are missing or do not match 406 – The user is already logged in 500 – An error occurred while retrieving the list

RESTful API: add new user

Method:	`POST`
Endpoint:	`/user/auth/`
Description:	This endpoint will be used to authenticate the requesting user.
Parameter:	X-Username: `USERNAME` X-Password: `PASSWORD`
HTTP Response:	202 – Success – User has been authenticated 400 – The user name is invalid or missing or the password is missing or invalid 406 – The user is already logged in 503 – The session could not be created

RESTful API: authenticate a user

Method:	`POST`
Endpoint:	`/user/chgpw/`
Description:	This endpoint must be used to change a user’s password.
Parameter:	Cookie: `JSESSIONID=VALUE` X-Username: `USERNAME` X-Password: `PASSWORD` X-Confirm: `CONFIRMATION`
HTTP Response:	200 – Success – Password has been changed 400 – The user name is invalid or missing or the password and its confirmation are missing or do not match 401 – The requesting user is not logged in 405 – The session identifier has not been transferred via cookie 500 – An error occurred while updating the user record 503 – The session given by the identifier does not exist

RESTful API: change a user password

Method:	`GET`
Endpoint:	`/user/auth/logout/`
Description:	This endpoint must be used to log-off
Parameter:	Cookie: `JSESSIONID=VALUE`
HTTP Response:	200 – Success – User has been logged out and session has been invalidated 401 – The requesting user is not logged in 405 – The session identifier has not been transferred via cookie 503 – The session given by the identifier does not exist

RESTful API: user logout

[CloudRAID] 4. Implementation

2012-11-13T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

After Florian’s publications about the architecture the concrete implementation design will follow.

4. Implementation

4.1. JNI API

The supplied RAID 5 implementation uses SHA-256 for integrity and validity checks. The hash implementation has been taken from the Apache Portable Runtime Project (APR) that has been published by the Apache Software Foundation under the terms of the Apache License version 2.0.

4.1.1. Pre-compiler Flags and Constants

The native RAID 5 level implementation uses multiple pre-compiler flags and constants to control the split and merge processes:

RAID5BLOCKSIZE specifies how many bytes the library will write at once to each device file (default 1024 bytes). Hence RAID5BLOCKSIZE × 2 bytes are read at once from the input file.
Normally all data will be encrypted before it is being split into the device files. This can be changed by setting ENCRYPT_DATA to 0.
The constant ENCRYPTION_SALT_BYTES specifies the size of the salt in bytes. The default value is 256.
For debugging purposes, the DEBUG constant can be set. There are three debug levels possible, where 1 prints minimum debug information and 3 all. A value of 0 disables entirely disables the debug output.
_NAME_, _VENDOR_ and _VERSION_ contain meta data for this library as exemplarily shown in listing 1. They information will be used by the native functions getName(), getVendor() and getVersion() as explained in chapter 4.1.3 - Function Signatures.

#define RAID5BLOCKSIZE 1024
#define ENCRYPT_DATA 1
#define ENCRYPTION_SALT_BYTES 256
#define DEBUG 0
#define _NAME_ "CloudRAID-RAID5"
#define _VENDOR_ "CloudRAID Team"
#define _VERSION_ "1.0.0-beta.1"

Listing 1: Pre-compiler flags and constants

4.1.2. Structs

For internal storage purposes, the native RAID implementation needs two C structs that are shown in listing 2:

Lines 1 to 6 show the SHA-256 context that allows to iteratively calculate the hash sum of data larger than an internal boundary size by storing the algorithms state, among other things, within this struct. This struct’s usage, as well as how the Secure Hash Algorithm works, is not explained as part of this research paper.
Lines 7 to 15 represent the information stored in a meta data file:
- First of all, each meta data file will contain a version number. This allows CloudRAID to implement backwards compatibility and makes the handling of new features to the native RAID implementation much more flexible.
- The unsigned char arrays hash_dev0, hash_dev1 and hash_dev2 are used to store the SHA-256 sum of the regarding device files, while hash_in stores the hash sum of the original input file. This makes it possible to verify the integrity of the output file. If any device file has been compromised, include the corresponding hash sum, the integrity check of the output file will fail.
- The salt is used to randomize the encryption key and hence increases the encryption strength.
- Finally the missing element contains information about how many bytes are missing on the secondary device (not the parity device) to fill the complete block. This is necessary due to the way CloudRAID handles files of a size different to the internal RAID5 block size.

struct sha256_ctx {
  uint32_t state[8];
  uint32_t total[2];
  size_t buflen;
  uint32_t buffer[32];
};

typedef struct raid5md {
  unsigned char version;
  unsigned char hash_dev0[65];
  unsigned char hash_dev1[65];
  unsigned char hash_dev2[65];
  unsigned char hash_in[65];
  unsigned char salt[ENCRYPTION_SALT_BYTES];
  unsigned int missing;
} raid5md;

Listing 2: Structs used for internal storage within RAID level 5 imlementation

4.1.3 Function Signatures

The The Java Native Interface (JNI) implements six functions whereof two are directly used for the RAID features and three are used to determine some version and vendor information about the library. The remaining, sixth, function is used to exchange information about the size of the struct raid5md from the library to CloudRAID. The function signatures are shown in listing 3 and will be explained here. All JNI functions take at least two parameters: JNIEnv *env and jclass cls. The former parameter refers to the Java environment that provides functions to for example convert and access strings. The latter is the instance of the current object the JNI function is working in but is not used within the CloudRAID implementation.

mergeInterface() in lines 1 to 4 is the function that is called to merge three device files to the output file. It takes the directory where the device files are stored as first (additional) argument. The file’s hash value is taken as second argument. The function will append the file extensions .0, .1, .2 and .m automatically. The third parameter defines the full path to the output file. The target directory must exist, otherwise an error will be returned while trying to open the output file. The last parameter is the secret part of the key that has been used during encryption. One must keep in mind, that the _tempInputDirPath must contain a trailing folder separator. On Unix systems, this is most likely the forward slash (/) and on Microsoft Windows the backward slash (\).
splitInterface(), as defined in lines 5 to 8, is the function signature for split calls. The function takes four additional parameters: the first one specifies the base input directory for the file that will be split. The second parameter is the path, relative to the first parameter, that directs to the file. In a multi user environment, adding the user name or any other user-unique identifier to this relative path will ensure that every output file exists only once (as long as there are no hash collision for two files with different user identifiers). The file names for the output files will be the SHA-256 sum over the concatenation of _inputBasePath and _inputFilePath appended by .0, .1 and .2 for the device files and .m for the meta data file. The third parameter denotes the output directory. Finally, _key takes the secure part of the encryption key. In the same manner as in the mergeInterface() function, the paths _inputBasePath and _tempOutputDirPath must be terminated by folder separators.
The functions getName(), getVendor() and getVersion() as defined in lines 9 to 14 do not take additional parameters. They can be used to show versioning information of this library that will then exemplarily be shown on the REST API information page (c.f. chapter 4.2.2 - Version Information).
Lines 15 and 16 show the getMetadataByteLength() function that is for CloudRAID core internal use only and must return the size of the meta data file in bytes.

JNIEXPORT jint JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_mergeInterface(JNIEnv *env, jclass cls,
    jstring _tempInputDirPath, jstring _hash, jstring _outputFilePath,jstring _key);

JNIEXPORT jstring JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_splitInterface(JNIEnv *env, jclass cls,
    jstring _inputBasePath, jstring _inputFilePath, jstring _tempOutputDirPath, jstring _key);

JNIEXPORT jstring JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_getName(JNIEnv *env, jclass cls);

JNIEXPORT jstring JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_getVendor(JNIEnv *env, jclass cls);

JNIEXPORT jstring JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_getVersion(JNIEnv *env, jclass cls);

JNIEXPORT jint JNICALL Java_de_dhbw_1mannheim_cloudraid_core_impl_jni_RaidAccessInterface_getMetadataByteLength(JNIEnv *env, jclass cls);

Listing 3: JNI header functions signatures.

The main internal functions used by the primary two JNI functions above are explained in this paragraph. For both processes, split and merge, there are two functions each, split_file() and split_byte_block() (merge_file() and merge_byte_block() respectively) that actually handle the processes (see listing 4 at the end of this post). The JNI functions for split and merge must be seen as wrapper for the split_file() and merge_file() functions. These functions handle the complete processes show by the figures 1 and 2 (introduced and explained in a previous article of this article series). All FILE* parameters must be correctly opened for read or write access. On Unix systems this includes the binary b mode for fopen().

split_file() expects a file pointer (rb mode) to the input file as first parameter. The FILE *devices[] array must contain three elements to files opened in wb mode. These files are the device files. The meta parameter points to the meta data file as must have been opened in wb mode as well. const char *key is the secure key used for file encryption and will be passed to the hmac() function together with its length keylen.

Figure 15: The nativ split process

This function reads up to 2 × RAID5BLOCKSIZE for each iteration shown in figure 1. These bytes are encrypted with the key generated by hmac() if ENCRYPT_DATA is not 0 and then passed to split_byte_block(). The call-by-reference “return” values are written to the device files. After each iteration the parity position is incremented. Starting with device 2 as parity for the first block, the parity position will be 0 for the second block and 1 for the third block. The forth block will restart with position 2.
The split_byte_block() function takes the bytes read and encrypted in split_file() as first parameter and the length of that buffer as second parameter. The function splits the input character array in three parts that will be returned via the unsigned char *out parameter. The output size for each device is returned via the fourth parameter: size_t out_len[]. The alignment for the output buffer is as follows:
- The primary device content will be stored at out[0]. Its length is out_len[0].
- The secondary device content will be stored at out[RAID5BLOCKSIZE]. Its length is out_len[1].
- The tertiary device content, representing the parity device, will be stored at out[2 * RAID5BLOCKSIZE]. Its length is out_len[2].
Depending on the input length in_len the values of out_len[0] to out_len[2] may vary:
merge_file() is the inverse function to split_file(). It basically has the same function signature, but the first parameter specifies the output file unlike in split_file() and hence must be opened in wb mode. Thus the device files and meta data file, parameters two and three, are input files and therefore they must be opened in rb mode. The fourth parameter, const char *key is the secure key used for file decryption and will be passed to the hmac() function together with its length keylen in the same way as in split_file().

Figure 16: The native merge process

At first, the integrity of the device files is validated as shown in figure 2 by checking the hash sums for the device files. If more than two check sums are wrong the function terminates with an error. Otherwise the function will read up to RAID5BLOCKSIZE from each device file and passes those buffers with their respective lengths to merge_byte_block(). The call-by-reference “return” value from each block merge is passed to a SHA-256 context to validate the correctness of the output file and then decrypted.
The actual merge occurs in the merge_byte_block() function. Again the content for all three buffers is stored in a single character array as first parameter: const unsigned char *in. The second parameter, const size_t in_len[] represents the lengths for the three read buffers from the device files. Both parameters together align the input buffers in the same way as in split_byte_block():
- The primary device content will be stored at in[0].Its length is in_len[0].
- The secondary device content will be stored at in[RAID5BLOCKSIZE]. Its length is in_len[1].
- The tertiary device content, representing the parity device, will be stored at in[2 * RAID5BLOCKSIZE]. Its length is in_len[2].
The third parameter, parity_pos, specifies which device is the parity. This is necessary in combination with the fourth parameter, dead_device, to determine the merge strategy: concatenating the content of the buffers for device files 0 and 1 or using the parity in combination with either of first two device files. missing defines how many bytes are missing on the secondary device. Only the last block of each overall merge process may set this parameter to a value > 0. The penultimate parameter, *out stores the merged data with a total length of *out_len but at most 2 × RAID5BLOCKSIZE.
create_salt(): the complete security of the whole encryption process relies on the strength of the key given to split_file() and merge_file(). However, to protect the data against attacks using rainbow tables the encryption key is being salted. The salt of length ENCRYPTION_SALT_BYTES is generated by this function. On Microsoft Windows each byte of the salt will be set by a return value of rand() after initializing the pseudo random generator with srand(time(NULL)). On all other Operating Systems (OSs), that can be expect to be Unix, the functions reads ENCRYPTION_SALT_BYTES from /dev/urandom which can be seen to be random.
The salted key used during encryption and decryption is created by the function hmac(). This function is an implementation of the Federal Information Processing Standards Publication (FIPS PUB) 198 [NIS02] HMAC specification and takes five parameters: the first two parameters (*key and keylen) define the secure key given by the JNI function split_file() and merge_file(). Parameters three and four link to the *salt (optimally generated by the create_salt() function) and its length that is ENCRYPTION_SALT_BYTES by default. The last parameter, *hash, is a call-by-reference parameter and “returns” the HMAC for the given *key and *salt.

The general mode of operation can be expressed by the following two expressions:

The implementation of the HMAC [NIS02] standard has been validated by computing the digest values for the test cases 1, 2, 3, 4, 6 and 7 defined in RFC 4231 [Nys05]. Test case 5 has not been tested since the truncation is not needed in CloudRAID and was therefore not implemented.

int split_file(FILE *in, FILE *devices[], FILE *meta,
  const char *key, const int keylen);

void split_byte_block(const unsigned char *in, const size_t in_len,
  unsigned char *out, size_t out_len[]);

int merge_file(FILE *out, FILE *devices[], FILE *meta,
  const char *key, const int keylen);

void merge_byte_block(const unsigned char *in, const size_t in_len[],
  const unsigned int parity_pos, const unsigned int dead_device,
  const unsigned int missing, unsigned char *out, size_t *out_len);

int create_salt(unsigned char *salt);

unsigned long hmac(const unsigned char *key,
  const unsigned int keylen, const unsigned char *salt,
  const unsigned int saltlen, unsigned char *hash);

Listing 4: Internal native header function signatures

Sources

[NIS02]

NIST Computer Security Division (CSD). The Keyed-Hash Message Authentication Code (HMAC). Federal Information Processing Standards Publication FIPS PUB 198, National Institute of Standards and Technology, March 6, 2002. http://csrc.nist.gov/publications/fips/fips198/fips-198a.pdf.

[Nys05]

Magnus Nystrom. Identifiers and Test Vectors for HMAC-SHA-224, HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512, December 2005. http://tools.ietf.org/html/rfc4231

[CloudRAID] 3. Concept (Continuation)

2012-11-09T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

With some delay I can announce the closing chapter for the architectural backgrounds of CloudRAID. Florian published the rest of chapter “3.3. Server Architecture” and the internals of the “3.4. Client Application”.

[CloudRAID] 3. Concept

2012-11-07T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

I’m happy to announce the next part of our student research paper. Today Florian presents the beginning of the third chapter that is all about the concept behind CloudRAID. His post covers the “Requirements” that need to be fulfilled, the “General Architecture” of the server application and also gives some insight on the “Server Architecture”.

[CloudRAID] 2. Basics (Continuation)

2012-11-03T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

2. Basics

2.3. Background on RAID Technology

In order to provide data-safety on a common server a RAID can be used. First introduced in 1988 under the title Redundant Arrays of Inexpensive Disks [DK88], the usage of hard drives in an array collection is state of the art nowadays. The paper by Petterson, Gibson and Katz introduces the RAID levels 1 to 5 as follows.

Source: [Cbu06a]

RAID Level 1 provides a high data-safety with a complete fall-back to a secondary device. This “mirroring” has a space efficiency of 50% of the total disk capacity [DK88] (ch. 7, p. 112) for two disks. In general the space efficiency is at 1/n. The fault tolerance is n-1 since all disks contain the same data and all but the last can fail without any data-loss.

Source: [kna09]

RAID Level 2 defines a bit-level striping with an Error Correction Code (ECC) for recovery [DK88] (ch. 8, p. 112) that uses the Hamming Code [Wik12a] which is stored on multiple check disks. Nowadays RAID systems of level 2 are not used any more. They are too prohibitive and do not provide more fault tolerance than RAID level 3, 4 or even 5. RAID level 2 can recover from one drive failure and has a space efficiency of [Wik12b]:

Source: [Cbu06b]

RAID Level 3 provides byte-level striping with a single parity per byte [DK88] (ch. 9, pp. 112f). Taking at least three disks, two for content and one dedicated for the parity, RAID level 3 provides a space efficiency of 1 - 1/n and a fault tolerance of one broken disk [Wik12b]. These statistics for minimum number of disks, space efficiency and fault tolerance are the same for RAID level 4 and 5.

Source: [Cbu06c]

RAID Level 4, similar to level 2 and level 3, uses striping to spread the data over multiple disks. But RAID level 4 splits the data block-wise and not bit- or byte-wise [DK88] (ch. 10, pp. 113f). This improves Input / Output (I/O) performance, but can result in a bottleneck, since the parity is stored on a single device.

Source: [Cbu06d]

RAID Level 5 introduces a distribution of the check disk in order to resolve the bottleneck that exists in RAID level 4. This does not change any minimum requirements or fault tolerance, but increases the I/O [DK88] (ch. 11, p. 114), [Wik12b] and therefore is one of the most used RAID levels nowadays.

2.4. Encryption Standards and Hash Algorithms

In modern computer science cryptography is an important topic while planning and designing software. It is mostly used when data is transferred over an insecure channel or is stored at a place that cannot be guaranteed to protect sensitive or confidential data. Encryption algorithms like the Advanced Encryption Standard (AES) and RC4 (also known as ARC4) are widely used and provide strong encryption. The former algorithm is a block cipher which means that a certain number of bytes is encrypted simultaneously with the given key. In contrast, the RC4 algorithm is a stream cipher, meaning that each byte is encrypted on its own with respect to the previously encrypted bytes and the secure key.

Both named algorithms use the same key for encryption and decryption and therefore belong to the group of symmetric encryption algorithms. The RSA algorithm (see U.S. Patent 4,405,829) on the other hand belongs to the group of asymmetric encryption algorithms or public-key algorithms because it uses a private key for decryption and a public key for encryption. One cannot construct one of these keys form the other without knowing an additional private information.

Even if all stated encryption algorithms ensure that the data that has been encrypted is secure, they cannot provide data integrity and consistency. If an encrypted message or file is changed, the decryption will return data that differs from the original data. To eliminate this issue, so called hash algorithm are used. Given some input data, a hash function will return a hash sum (also referred as checksum or only hash) to this data. Since hash functions are one-way functions, one cannot get the original data from the hash sum. A strong hash function will always return a distinct hash sum for a specific input and does not (or at least tries to minimize) the likelihood of hash collisions. Hash collisions occur when a hash function returns the same hash sum for two different input values. This can easily be shown by the following formula, where x and y define the input values and H defines the hash function:

Based on these requirements, the NIST and the German “Bundesnetzagentur” recommend and claim the use of algorithms of the Secure Hash Algorithm (SHA)-2 family [Eck12] with 224, 256, 384 or 512 bits rather than MD5 (128 bits) or SHA-1 (160 bits)

Sources

[Cbu06a]

Cburnett. RAID 1. Wikimedia Commons, GNU Free Documentation License, December 31, 2006. http://en.wikipedia.org/wiki/File:RAID_1.svg

[Cbu06b]

Cburnett. RAID 3. Wikimedia Commons, GNU Free Documentation License, December 31, 2006. http://en.wikipedia.org/wiki/File:RAID_3.svg

[Cbu06c]

Cburnett. RAID 4. Wikimedia Commons, GNU Free Documentation License, December 31, 2006. http://en.wikipedia.org/wiki/File:RAID_4.svg

[Cbu06d]

Cburnett. RAID 5. Wikimedia Commons, GNU Free Documentation License, December 31, 2006. http://en.wikipedia.org/wiki/File:RAID_5.svg

[DK88]

(1, 2, 3, 4, 5, 6) Garth Gibson David A. Patterson and Randy H. Katz. A Case for Redundant Arrays of Inexpensive Disks (RAID). Technical report, University of California Berkeley, 1988.

[Eck12]

Claudia Eckert. IT-Sicherheit – Konzepte – Verfahren – Protokolle. Oldenbourg Verlag München, 7 edition, 2012.

[kna09]

knakts. RAID 2. Wikimedia Commons, GNU Free Documentation License, January 18, 2009. http://en.wikipedia.org/wiki/File:RAID2_arch.svg

[Wik12a]

Wikipedia. Hamming code — Wikipedia, The Free Encyclopedia, January 22, 2012. http://en.wikipedia.org/w/index.php?title=Hamming_code&oldid=472688059

[Wik12b]

(1, 2, 3) Wikipedia. RAID — Wikipedia, The Free Encyclopedia, January 25, 2012 http://en.wikipedia.org/w/index.php?title=RAID&oldid=473130999

[CloudRAID] 2. Basics

2012-10-31T00:00:00+01:00

This post is a continuation of the blog series about the student research paper CloudRAID.

Today Florian Bausch published the next part of our student seminar paper. Chapters 2.1. and 2.2. describe the technical backgrounds of the “OSGi Framework” and explain the various kinds of “The Cloud” like private cloud, public cloud and hybrid cloud.

[CloudRAID] 1. Introduction

2012-10-28T00:00:00+02:00

During last year a fellow student of mine, Florian Bausch, and I wrote a student research paper about how to provide availability, redundancy and security of data in the overall existing and “well known” cloud. In this context we additionally developed a prototype that we call CloudRAID. The software is licensed under the terms of the Apache 2 License and published on GitHub.

During the next weeks we are going to publish our paper as a series of posts on our blogs. This post makes the start of this series and is going to introduce the general idea behind the topic. The next part of the student research paper will be published by Florian on his blog and covers some basic topics like the “OSGi Framework” and “Cloud”. I will announce the publication here. So, stay tuned.

0. Table of Contents

1. Introduction

In a world that gets more and more mobile, everybody wants to have access to his data all the time. The knowledge that one needs an important document and cannot access it because it is on the computer at home, makes one use cloud storage providers. But relying on the availability of those cloud providers is not a good idea as is shown later on. To work around the availability concerns, one might use multiple cloud storage provides and ends up with a lot of work to synchronize the storages.

Figure 1: Structural layout idea

This student research paper by Florian Bausch and Markus Holtermann finds a solution to this problem. It describes and evaluates the idea, concept and example implementation of a cloud-based Redundant Array of Independent Disks (RAID) storage system. The research goal is a file storage system, that combines multiple cloud services as back-end storages and represents them as a single front-end storage to the user. Providing several features like encryption, authorization, availability and redundancy, the cloud-based RAID can be used as a reliable backup storage even for sensitive personal information.

After introducing the basic knowledge and requirements of the research project, mainly Open Services Gateway initiative framework (OSGi), Cloud Computing, RAID and cryptography, there will be the concept of how to implement the features and building a cloud-based RAID. This elaboration is then followed by detailed implementation notes. The program will be titled CloudRAID to combine the terms “Cloud” and “RAID” as they are the foundational backgrounds for the project. To show the validity of CloudRAID, there are some benchmarks that point out the advantages and disadvantages as well as the problems that might occur.

One must keep in mind that CloudRAID will not be a RAID system in the strict sense, but uses the ideas and techniques of those systems.

PyCon DE 2012 in Leipzig

2012-10-19T00:00:00+02:00

At the end of October 2012 another PyCon takes place. This time the venue is the Kubus in Leipzig, Germany. As this is my first participation at a Python conference I’m a bit excited about this event.

First of all, I like the wide range of various topics that are offered. There are presentations about Machine Learning (Oct 30 @ 1:30pm in Hall 2) and Efficient Data Analysis (Oct 30 @ 2pm in Hall 1B). Other talks address the topic of computer graphics (SimpleCV: Fun with Computer Vision, Oct 31 @ 1:30pm in Hall 2; Tiffany - Reading and Writing Multipage Tiff Files w/ and w/o PIL, Nov 1 @ 11am in Hall 1A). And of course, there are multiple presentations covering different parts of the Django web framework: LFS - A Django based webshop (Oct 31 @ 2pm in Hall 1C), Overview over the Django admin area (Oct 31 @ 11am in Hall 1C) Django Class Based Generic Views (Oct 31 @ 1pm in Hall 1C) and many, many more talks.

Second, while looking through all the talks I just picked some I think I’m going to attend:

Tuesday, October 30th
Wednesday, October 31st
- Fast loops with Cython
- Designing and Testing PyZMQ Applications
Thursday, November 1st

During the barcamp and the sprints on Nov 2nd and 3rd I will work on Inyoka and fix a bunch of tickets (the list of open issues is way too long).

So, if you are at the PyCon DE 2012 too, leave a comment and we can meet.

Markus

[Update] My own IPv6 tunnel broker.

2012-03-20T00:00:00+01:00

Some days ago I needed to test a web application on its IPv6 support. But since my Internet service provider only gives me an IPv4 address, I was not able to test the application. So, I thought about registering at SixXS to get an IPv6 address. But then I thought about setting up my own broker. After some time I found a nearly perfect tutorial that uses OpenVPN.

I took the scripts from that tutorial and stored them on my server and client, set everything up as explained and … didn’t got it working :(. I must say, that I had limited time to completely understand what was going wrong, so I stop trying.

Last weekend I had some hours to work on the IPv6 tunnel broker again and finally got it working. So, here are my scripts and a short explanation.

Update: I did some improvements and enhancements on the scripts and pushed them to a Github repository. Feel free to fork and report issues.

Server

You need a dedicated OpenVPN account if your OpenVPN does not run as root. You must give this user full password-less sudo access to /sbin/ip: openvpn ALL=(ALL) NOPASSWD: /sbin/ip
You must enable package forwarding for IPv6. Append net.ipv6.conf.all.forwarding = 1 to /etc/sysctl.conf.
Warning

If you automatically receive your IPv6 routes, you must set your v6 routes manually:
```
$ ip -6 route add ::/0 via <IPv6-Gateway> dev <eth0>
```
You may need to activate Neighbor Discovery Proxy: Add net.ipv6.conf.all.proxy_ndp = 1 to your /etc/sysctl.conf
Copy the files client-connect-ipv6-broker.sh, client-disconnect-ipv6-broker.sh and ipv6-broker.sh to your server into the directory /etc/openvpn/ and make sure they are executable.
Append the lines from the server-ipv6.conf file to your OpenVPN server configuration.

Client

You need a dedicated OpenVPN account if your OpenVPN does not run as root. You must give this user full password-less sudo access to /sbin/ip: openvpn ALL=(ALL) NOPASSWD: /sbin/ip
Copy the files up-ipv6-broker.sh, down-ipv6-broker.sh and ipv6-broker.sh to your client into the directory /etc/openvpn/ and make sure they are executable.
Append the lines from the client-ipv6.conf file to your OpenVPN client configuration.

The scripts

OpenVPN IPv6 Tunnel Broker

Copyright (c) 2012 Markus Holtermann

Copyright (c) 2011 Lyndsay Roger - https://www.zagbot.com/

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program. If not, see http://www.gnu.org/licenses/.

ipv6-broker.sh

#!/bin/bash

# Your /64 Network prefix
export BASERANGE="1234:5678:90ab:cdef"
# The IPv4 address of your vpn server
export VPN_HOST="10.8.0.1"

export LOG_TAG="ipv6-broker"
export DEBUG=1

if [ -e "/etc/openvpn/client.conf" ] ; then
    export CLIENT=1
else
    export CLIENT=0
fi

if [ $CLIENT -eq 1 ] ; then
    export SITID="sit1"
    export V6NET="${BASERANGE}::$(echo ${ifconfig_local} | awk -F. '{printf "%02x%02x:%02x", $2, $3, $4}')"
else
    export SITID="sit$(echo ${ifconfig_pool_remote_ip} | awk -F. '{print $2"-"$3"-"$4}')"
    export V6NET="${BASERANGE}::$(echo ${ifconfig_pool_remote_ip} | awk -F. '{printf "%02x%02x:%02x", $2, $3, $4}')"
fi

client-connect-ipv6-broker.sh

#!/bin/bash

source /etc/openvpn/ipv6-broker.sh

# This is a script that is run each time a remote client connects
# to this openvpn server.
# it will setup the ipv6 tunnel depending on the ip address that was
# given to the client

# setup the sit between the local and remote openvpn addresses
test $DEBUG -eq 1 && echo "/sbin/ip tunnel add ${SITID} mode sit ttl 255 remote ${ifconfig_pool_remote_ip} local ${ifconfig_local}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip tunnel add ${SITID} mode sit ttl 255 remote ${ifconfig_pool_remote_ip} local ${ifconfig_local}

# activate the tunnel device
test $DEBUG -eq 1 && echo "/sbin/ip link set dev ${SITID} up" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip link set dev ${SITID} up

# config routing for the new network
test $DEBUG -eq 1 && echo "/sbin/ip -6 addr add ${V6NET}01/124 dev ${SITID}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 addr add ${V6NET}01/124 dev ${SITID}

# add the route for the network
test $DEBUG -eq 1 && echo "/sbin/ip -6 route add ${V6NET}00/124 via ${V6NET}02 dev ${SITID} metric 1" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 route add ${V6NET}00/124 via ${V6NET}02 dev ${SITID} metric 1

# add neighbor discovering proxy for this interface
test $DEBUG -eq 1 && echo "/sbin/ip -6 neigh add proxy ${V6NET}02 dev eth0" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 neigh add proxy ${V6NET}02 dev eth0

# log to syslog
test $DEBUG -eq 1 && echo "${script_type} client_ip:${trusted_ip} common_name:${common_name} local_ip:${ifconfig_local} remote_ip:${ifconfig_pool_remote_ip} sit:${SITID} ipv6net:${V6NET}" | /usr/bin/logger -t $LOG_TAG
test $DEBUG -eq 1 && sudo /sbin/ip addr show | /usr/bin/logger -t $LOG_TAG
test $DEBUG -eq 1 && sudo /sbin/ip -6 route show | /usr/bin/logger -t $LOG_TAG

# needed for connection
exit 0

client-disconnect-ipv6-broker.sh

#!/bin/bash

source /etc/openvpn/ipv6-broker.sh

# remove neighbor discovering proxy again
test $DEBUG -eq 1 && echo "/sbin/ip -6 neigh del proxy ${V6NET}02 dev eth0" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 neigh del proxy ${V6NET}02 dev eth0

# remove the route
test $DEBUG -eq 1 && echo "/sbin/ip -6 route del ${V6NET}00/124 via ${V6NET}02 dev ${SITID} metric 1" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 route del ${V6NET}00/124 via ${V6NET}02 dev ${SITID} metric 1

# unset the ipv6 address
test $DEBUG -eq 1 && echo "/sbin/ip -6 addr del ${V6NET}01/124 dev ${SITID}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 addr del ${V6NET}01/124 dev ${SITID}

# deactivate the tunnel
test $DEBUG -eq 1 && echo "/sbin/ip link set dev ${SITID} down" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip link set dev ${SITID} down

# remove the tunnel interface
test $DEBUG -eq 1 && echo "/sbin/ip tunnel del ${SITID} mode sit ttl 255 remote ${ifconfig_pool_remote_ip} local ${ifconfig_local}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip tunnel del ${SITID} mode sit ttl 255 remote ${ifconfig_pool_remote_ip} local ${ifconfig_local}

test $DEBUG -eq 1 && sudo /sbin/ip addr show | /usr/bin/logger -t $LOG_TAG
test $DEBUG -eq 1 && sudo /sbin/ip -6 route show | /usr/bin/logger -t $LOG_TAG

exit 0

up-ipv6-broker.sh

#!/bin/bash

source /etc/openvpn/ipv6-broker.sh

# script that is run on the client when it creates a tunnel to the remote OpenVPN server

test $DEBUG -eq 1 && echo "/sbin/ip tunnel add ${SITID} mode sit ttl 255 remote ${VPN_HOST} local ${ifconfig_local}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip tunnel add ${SITID} mode sit ttl 255 remote ${VPN_HOST} local ${ifconfig_local}

test $DEBUG -eq 1 && echo "/sbin/ip link set dev ${SITID} up" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip link set dev ${SITID} up

test $DEBUG -eq 1 && echo "/sbin/ip -6 addr add ${V6NET}02/124 dev ${SITID}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 addr add ${V6NET}02/124 dev ${SITID}

test $DEBUG -eq 1 && echo "/sbin/ip route add ::/0 via ${V6NET}01" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip route add ::/0 via ${V6NET}01

test $DEBUG -eq 1 && sudo /sbin/ip addr show | /usr/bin/logger -t $LOG_TAG
test $DEBUG -eq 1 && sudo /sbin/ip -6 route show | /usr/bin/logger -t $LOG_TAG

exit 0

down-ipv6-broker.sh

#!/bin/bash

source /etc/openvpn/ipv6-broker.sh

test $DEBUG -eq 1 && echo "/sbin/ip route del ::/0 via ${V6NET}01" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip route del ::/0 via ${V6NET}01

test $DEBUG -eq 1 && echo "/sbin/ip -6 addr del ${V6NET}02/124 dev ${SITID}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip -6 addr del ${V6NET}02/124 dev ${SITID}

test $DEBUG -eq 1 && echo "/sbin/ip link set dev ${SITID} down" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip link set dev ${SITID} down

test $DEBUG -eq 1 && echo "/sbin/ip tunnel del ${SITID} mode sit ttl 255 remote ${VPN_HOST} local ${ifconfig_local}" | /usr/bin/logger -t $LOG_TAG
sudo /sbin/ip tunnel del ${SITID} mode sit ttl 255 remote ${VPN_HOST} local ${ifconfig_local}

test $DEBUG -eq 1 && sudo /sbin/ip addr show | /usr/bin/logger -t $LOG_TAG
test $DEBUG -eq 1 && sudo /sbin/ip -6 route show | /usr/bin/logger -t $LOG_TAG

exit 0

Configuration

You finally need to add a few lines to your /etc/openvpn/server.conf

###########################################
# IPv6 tunnel broker
###########################################
script-security 2
client-connect /etc/openvpn/client-connect-ipv6-broker.sh
client-disconnect /etc/openvpn/client-disconnect-ipv6-broker.sh

And a few lines to your /etc/openvpn/client.conf:

#############################################
# IPv6 tunnel broker
#############################################
# need this so when the client disconnects it
# tells the server so the server can remove
# the ipv6 tunnel the client was using
explicit-exit-notify
script-security 2

# create the ipv6 tunnel
up /etc/openvpn/up-ipv6-broker.sh
down /etc/openvpn/down-ipv6-broker.sh

Mandelbrot in Python

2011-11-17T00:00:00+01:00

I just found a nice Python script that draws the Mandelbrot set.

_                                      =   (
                                        255,
                                      lambda
                               V       ,B,c
                             :c   and Y(V*V+B,B,  c
                               -1)if(abs(V)&lt;6)else
               (              2+c-4*abs(V)**-0.4)/i
                 )  ;v,      x=1500,1000;C=range(v*x
                  );import  struct;P=struct.pack;M,\
            j  ='&lt;QIIHHHH',open('M.bmp','wb').write
for X in j('BM'+P(M,v*x*3+26,26,12,v,x,1,24))or C:
            i  ,Y=_;j(P('BBB',*(lambda T:(T*80+T**9
                  *i-950*T  **99,T*70-880*T**18+701*
                 T  **9     ,T*i**(1-T**45*2)))(sum(
               [              Y(0,(A%3/3.+X%v+(X/v+
                               A/3/3.-x/2)/1j)*2.5
                             /x   -2.7,i)**2 for  \
                               A       in C
                                      [:9]])
                                        /9)
                                       )   )

You can change the output dimension in line 8. Just make sure that the width is dividable by 4.

I ran the script with a resolution of 3000x2000 px on my T400 with an Intel Core2Duo P8600 @ 2.40GHz and 4GB RAM. I used Python 2.7.2-2 on Arch Linux 64bit and it took took me half an our. And as stated in the article as well, PyPy 1.6-2 should be much faster. And I can confirm that. It only runs half the time of the regular Python implementation:

$ time python2 mandelbrot.py
real    29m16.779s
user    29m12.519s
sys     0m0.977s

$ time pypy mandelbrot.py
real    16m21.750s
user    16m18.200s
sys     0m1.320s

And here the download link. I converted the file M.bmp, which had a size of 18MB to PNG with a size of 1.7MB:

Dominator 1.0 released

2011-11-14T00:00:00+01:00

As a result of my fourth semester at the university, my team and I created the DOMINATOR (de) program, an OpenGL 2.1 based 3D domino simulation. You can find a demo video at YouTube.

In round about 13,000 lines of C++ code we solved this exercise. As a UI framework we decided to make use of the Qt framework which has perfect OpenGL integration. The lib3ds and the Newton Game Dynamics are used to import 3DS models and calculate the physical environment. Several 3rd-party libraries finalize the set of external software that we use for our project.

DOMINATOR 1.0 officially supports Microsoft Windows 7 and Ubuntu 10.04 (Lucid Lynx) and 10.10 (Maverick Meerkat). I tested it on Arch Linux as well and even developed it on my Arch Linux systems. But there is no install script for Arch right now, sorry for that, but you can easily compile it from source. See the links below and have a brief look at the development installation docs. In theory Mac OSX should be supported as well, but we couldn’t get Newton Game Dynamics compiled.

And here’s the code on Github.

Markus

My own Git server - installing gitolite

2011-09-16T00:00:00+02:00

Yesterday I wanted to share some code with some friends, but I didn’t want to create a repository on GitHub, so I decided to install my own git server. After some minutes googleing I found gitosis and gitolite. Both programs seem to be popular, but the last commit to gitosis was two yeas ago. So I chose gitolite.

I had two options on how to install gitolite: either through the package manager (sudo apt-get install gitolite) or by directly cloning the repository from Github. I used the latter one to be up to date.

I wanted gitolite to run under its own user account, lets say gitolite. So I had to add this user:

markus@server:~$ sudo adduser --system \
--shell /bin/bash \
--group \
--disabled-password \
--home /home/gitolite \
gitolite

The installation was quite easy. For initial configuration I needed my SSH-key. So I copied my local ~/.ssh/id_rsa.pub to /tmp/markus.pub on the server. Then I switched to the newly created user and cloned the code from Github:

markus@server:~$ sudo su gitolite -
gitolite@server:/home/markus$ cd
gitolite@server:~$ git clone https://github.com/sitaramc/gitolite.git

After these steps I could install gitolite. Therefor I ran those commands:

gitolite@server:~$ cd gitolite
gitolite@server:~$ src/gl-system-install
gitolite@server:~$ gl-setup /tmp/markus.pub

Now I was able to clone the repository to my local machine:

git clone gitolite@server:gitolite-admin

A new folder gitolite-admin was created that contains a configuration file and an directory for SSH-keys. To add my other computers and my friends to the repositories, I changed the configuration file as follows:

@markus = markus markus2 markus3
@user = user1 user2
@otheruser = otheruser1 otheruser2 otheruser3

repo gitolite-admin
     RW = @markus

repo project1
     RW = @markus @user otheruser2

repo project2
     RW = @markus @user
     R  = @otheruser

repo project3
     RW = @markus
     R  = @all

The first three lines declare some groups, so if one gets a new client, I can add that to the group and he will directly have access to all repositories that his other clients can access as well.

The repository gitolite-admin is only readable (R) and writabel (W) by myself. The project1 has read/write permissions for all my clients, all clients of user but only the second client of otheruser. In the second project repository, (project2) all clients of otheruser can only read, but not write. Finally, repository project3 is only writable for all my clients but readable for all users.

The next step was to add the changes and all new SSH-keys to the repository:

$ git add conf/gitolite.conf keydir/markus2.pub keydir/markus3.pub \
keydir/user1.pub keydir/user2.pub \
keydir/otheruser1.pub keydir/otheruser2.pub keydir/otheruser3.pub
$ git commit -m 'add first projects and users'
$ git push

Ready.

If you are now trying to connect via SSH and user gitolite to the server, you will get a nice information:

$ ssh git@server
hello markus, this is gitolite running on git
the gitolite config gives you the following access:
     R   W      gitolite-admin
     R   W      project1
     R   W      project2
     R   W      project3
Connection to server closed.

Thanks for reading :)

Markus

pyChallenge 1.0 veröffentlicht

2011-07-22T00:00:00+02:00

Wie in meinem Artikel vom 26.05.2011 angekündigt, habe ich mit einigen Kommilitonen in der Datenbankvorlesung das Projekt pyChallenge entwickelt mit dem die Spielstärken von Schach- oder Tennisspielern berechnet werden können. Dies geschieht anhand der Algorithmen ELO und Glicko.

Weiterhin lassen sich mit pyChallenge auch die Besten- und Schlechtesten- Listen eines Spiels ausgeben, und die bestmöglichen Paarungen für einen jeden Spieler errechnen.

Kern der gesamten Entwicklung lag aber auf dem ORM, welches wir für unsere Zwecke selber entwickelt haben. Das ORM nimmt uns nun folgende Funktionen ab wodurch die Anwendungslogik sehr gut von der Datenrepräsentation abgetrennt wird:

INSERT INTO
SELECT FROM
UPDATE WHERE
DELETE FROM
CREATE TABLE
TRUNCATE TABLE
DROP TABLE

Ein kurzes Beispiel, wie das ORM genutzt wird, ist dort zu sehen.

Dann bleibt uns nur noch Danke für das Testen und Nutzen zu sagen.

Ein Artikel zu dem Projekt Dominator aus der Software-Engineering Vorlesung, folgt demnächst.

Folge uns auf github.com

Neues vom Studium

2011-05-26T00:00:00+02:00

Das neue Semester hat Anfang des Monats begonnen und es stehen wieder Projekte an. Im letzten Semester habe ich mit meinem Team das Programm pxChat entwicket. Dieses Mal müssen jedoch zwei Projekte gestemmt werden.

In der Vorlesung Software Engineering entwickeln wir einen 3D-Domino Simulator. Dabei setzen wir auf C/C++, Qt und OpenGL sowie einige andere Bibliotheken. Eigentlich wollten wir das Programm ausschließlich für Linux programmieren, aber unser Dozent will eine Windows-Version haben :-| . Daher hat sich ein Kommilitone bereits an die Portierung der ersten paar 1000 Zeilen Quelltext gemacht und hat schon Stunden benötigt. Hoffen wir also, dass bald die komplizierten Bibliotheken portiert sind und die Entwicklung dann effizient fortschreiten kann.

Als Projekt der Vorlesung Datenbanken werden wir pyChallenge entwickeln. Dieses in Python geschriebene Programm ist zum “Vorhersagen” von Spielergebnissen. Solche Vorhersagen erfolgen bspw. mittels ELO oder Glicko. Die nötigen statistischen Daten werden wir in einer SQLite Datenbank ablegen. Der Zugriff auf diese Daten soll dann über ein kleines selbst entwickeltes ORM erfolgen.

So, das wäre es dann erstmal.

Markus

Binärtaschenrechner mit Flex und Bison

2010-12-14T00:00:00+01:00

Einen Taschenrechner für Binärzahlen kann man mit Flex und Bison recht schnell erstellen. Zunächst erstellen wir eine Datei calc_binary.l. Diese ist der sogenannte Lexer:

%{
#include "calc_binary.y.h"
%}
%%
\+ { return PLUS;}
-  { return MINUS;}
\* { return MAL;}
0  { return NU;}
1  { return EINS;}
\( { return AUF; }
\) { return ZU; }
\n { return AUS; }
[ ]+ ;
.  { printf("what?");}

Diese Datei übersetzen wir dann mit:

$ flex -o calc_binary.c calc_binary.l

Nun folgt der Parser, ein Bison-File, mit dem Namen calc_binary.y:

%{
#include <stdio.h>
#include <math.h>
void yyerror(char *message);
%}

%start S

%token AUS
%token PLUS MINUS
%token MAL
%token NU EINS
%token AUF ZU

%left PLUS MINUS
%left MAL

%%
S : E AUS {printf("= %d\n ", $1); }
  | S E AUS {printf("=%d\n ", $2);};

E : B {$$ = $1;}
  | E PLUS E {$$ = $1 + $3;}
  | E MINUS E {$$ = $1 - $3;}
  | E MAL E {$$ = $1 * $3;}
  | AUF E ZU {$$ = $2;}
  | MINUS E {$$ = - $2;};

B : NU {$$ = 0;}
  | EINS {$$ = 1;}
  | B NU {$$ = $1 * 2;}
  | B EINS {$$ = $1 * 2 + 1;};

%%

int main(int argc, char **argv) {
        yyparse();
        return 0;
}

void yyerror(char *message) {
        printf("Good bye\n");
}

Auch diesen müssen wir übersetzen:

$ bison -d -b y -o calc_binary.y.c calc_binary.y

Nun können wir mit GCC eine ausführbare Datei erstellen können:

$ gcc calc_binary.c calc_binary.y.c -lfl -lm -o calc_binary

Wenn wir diese mit:

$ ./calc_binary

aufrufen, können wir Binärzahlen addieren, subtrahieren und multiplizieren.

Markus Holtermann

Writing Safe Database Migrations in Django

Migrations! How? When?

When Do We Deploy Migrations?

How Do We Deploy Migrations?

Only Go Forwards & Never Look Back

Adding A Field Is Harmless

But I Want A Default Value!

Adding An Index

Test Your Migrations

Summary

A Pony On The Move: How Migrations Work In Django 🐎

Introduction

ModelStates, ProjectStates, StateApps

Migrations and Operations

Migration Graph

Migration Loader

Migration Executor

Model Field References

Performance Considerations

Unapplying Migrations

Migration Autodetector

Summary

FAQ

Why does changing verbose_name cause a migration?

Why can’t I remove old field classes?

Why doesn’t the SchemaEditor use ModelStates?

All Hands on Deck — Handling Security Issues

A Historic Event

A Present Company

Receiving Security Issue Report

Freeing Up Resources

Fixing The Security Issue

Getting A Fix To Customers

Dealing With The Fallout

What Do We Learn From The Disaster?

(Re)evaluate Requirements

Practice Makes Perfect

Communication Is Crucial

Resources

Logging Rethought

The Current State Of Logging

Our Logging Is Broken

Tracing Events

Tracing Errors

There Is More Out There Than HTTP

Tracing Users And Other Objects

Adding Structure To Our Logs

WYSIWYU – What You See Is What You Understand

A Picture Says More Than A Thousand Words

Structuring My Django Logs

Logging For European Users

Less Obvious Things To Do With Django’s ORM

Our Models

Counting objects

Fetching single objects

Advanced filtering

Following related objects

Sidebar: Inspecting database queries

Following one-to-one and many-to-one relationships

Following one-to-many and many-to-many relationships

Aggregating data

Let’s COUNT again

Let’s SUM it up

Annotating “arbitrary” data

Top-k selects

Top three by author

Top three by genre

Setting up a private & secure IRC channel

Simple TOTP Bash Script

Oh, I Found a Security Issue

Django’s Security Process

Reporting the issue

Assessing the reported issue

Fixing the issue

Confirming the fix

Pre-notification

Release & Announcement

Implementing these steps yourself

Setup reporting channel

Let’s `COUNT` again

Let’s `SUM` it up

`db_index=<IndexClass>`

`Field.default_index_class`

Refactor `index_together` and `db_index`

`GiSTIndex`