Metaprogramming and Django: using decorators

While programming is about, in some way, doing code to transform data, metaprogramming can be seen as the task of doing code to change code. This category is often used to help programmers to enhance the readability and maintainability of the code, help with separation of concerns and respect one of the most important principles of software development, "Don't repeat yourself".Here, I will show how we, at Vinta, often use a bit of this concept with Django using a great Python tool, called Decorator. But first, let's record how a decorator works.<h3 id="a-brief-introduction">A Brief Introduction</h3>In Python, we have this powerful feature called "Decorators". They are just callables that take a function as an argument and extend its behavior in some way, without explicitly modifying it. Check the simple example below.<pre><code>from boltons.funcutils import wraps def decorator(some_function): @wraps(some_function) def wrapper(): print("This is printed before call the function.") some_function() print("This is printed after call the function.") return wrapper def my_function(): print("This is the function!") my_function = decorator(my_function) my_function() </code></pre>The output would be:<pre><code>This is printed before calling the function. This is the function! This is printed after calling the function. </code></pre>In the example, the <code>decorator</code> function is a function that takes another function as an argument, function with this characteristic is called higher-order functions. You can apply a decorator to a function by explicitly passing the function:<code>my_function = decorator(my_function)</code>Or you can use the most pythonic approach by using the '@' operator:<pre><code>@decorator def my_fuction(): print("This is the function!") my_function() </code></pre>Note that we are using the <code>wraps()</code> from boltons library, if you already know something about decorators you might be asking why I didn't use <code>functools.wraps()</code>, right? So, these decorators are used to make your decorator’s wrapper functions reflect the wrapped function’s metadata, it means the name, documentation, module and signature. However <code>functools.wraps()</code> does not preserve the function signature and the <code>wraps()</code> from boltons preserves. If you are using Python 3.5+ functool.wraps() will work ok, but as we're always using publics libraries, that are written in Python 2.7, this can break. If you want to understand more about it, you can read this article: <a href="https://hynek.me/articles/decorators/">https://hynek.me/articles/decorators/</a><h4 id="writing-tests">Writing Tests</h4>In this real example, we have two types of messages - drafts and queued - and the user should be able to edit both of them. So, to test this behavior we would do something like the code below.<pre><code>def test_valid_authenticated_put_returns_200(self): message_types = ['draft', 'queued'] for message_type in message_types: self.message.type = message_type self.message.save() response = self.auth_client.put(reverse(self.view_name, kwargs={'pk': self.message.id}), data=json.dumps(self.params), content_type='application/json') self.assertEqual(response.status_code, status.HTTP_200_OK) self.assertIn('id', response.data) </code></pre>So, it was necessary to iterate over the list <code>message_types</code> to guarantee that the behavior is the same for every message type. Now, we want to test if an unauthenticated post would return HTTP 403, we would do the same iteration for this test and for every test that both types of a message have the same behavior.To avoid this code repetition we can write a decorator to vary the test class attributes.<pre><code>from boltons.funcutils import wraps def vary_test_self_context(**context): def decorator(f): @wraps(f) def wrapper(self, *args, **kwargs): for k, v_list in context.items(): for v in v_list: setattr(self, k, getattr(self, v)) f(self, *args, **kwargs) return wrapper return decorator </code></pre>This decorator will set the self attributes, with the values passed to it, before execute the test. So our new test would be like the code above:<pre><code>@vary_test_self_context(message=['draft_message', 'queued_message']) def test_valid_authenticated_put_returns_200(self): response = self.auth_client.put(reverse(self.view_name, kwargs={'pk': self.message.id}), data=json.dumps(self.params), content_type='application/json') self.assertEqual(response.status_code, status.HTTP_200_OK) self.assertIn('id', response.data) </code></pre>Test with metaprogramming is a controversial topic, sometimes the metaprogramming can add complexity to the test, making it unclear 'cause it will hide from you where or in which loop the error is happening. But the example above, in my opinion, isn't complex and this can be fixed using friendly error messages. If you want to read more about this topic there is a <a href="http://blog.jayfields.com/2008/03/testing-anti-pattern-metaprogrammed.html">good blog post</a> about it written by <a href="https://twitter.com/thejayfields">@thejayfields</a>.<h4 id="implementing-guards">Implementing Guards</h4><h5 id="sending-email">Sending Email</h5>Let's see another real example, I'll follow the first example where we have two types of messages, <code>draft</code> and <code>queued</code>. We want to write a function to send a message by email but we need to assure that it's a queued message and it's been sent to an admin. So the function we'll be like this:<pre><code>def send_queued_message_email(message): if message.is_queued: if message.user.is_admin: {send the email} else: logger.warning("The recipient user_pk={user.pk} is not an admin".format(user=message.user)) else: logger.warning("Message pk={message.id} is a draft.".format(message=message)) </code></pre>The problem with this code is that is hard to separate the logic of the {send the email} part. Another point is that the condition it's on the top of the function while the error is at the bottom. Making more difficult to correlate the condition with the error.So, how to do this with a decorator?<pre><code>def queued_message_required(f): @wraps(f) def wrapper(message, *args, **kwrags): if message.is_queued: return f(message, *args, **kwargs) # return error return wrapper def message_to_admin_required(f): @wraps(f) def wrapper(message, *args, **kwargs): if message.user.is_admin: return f(message, *args, **kwargs) # return error return wrapper </code></pre>And we'll decorate the <code>send_queued_message_email</code> like below:<pre><code>@queued_message_required @message_to_admin_required def send_queued_message_email(message): {send the email} </code></pre>Then when we call the <code>send_to_admin_required</code> it first enters in the <code>queued_message_required</code> decorators function if it passes the check the function is passed to <code>message_to_admin_required</code> decorator function. In the end, we have a reusable guard decorator and a code clearer.<h4 id="doing-payment-charging-users">Doing Payment: Charging Users</h4>Payment operations are always critical, so it's a good practice to write the code in a way that others programmers won't call this operation to the wrong user or in a wrong moment, like the example above. In this example, we had two kinds of users, users that receive his package on Wednesdays and Fridays, and they must be charged three days before the delivery date. So we wrote a custom model manager to the User model and wrote a method to charge every user. Something like this:<pre><code>def charge_wednesday_users(self): """week_day = 4, when filtering queries in Django, means Wednesday""" users = self.get_queryset().filter(delivery_dates__0__week_day=4) for user in users.iterator(): user.charge() def charge_friday_users(self): """week_day = 6, when filtering queries in Django, means Friday""" users = self.get_queryset().filter(delivery_dates__0__week_day=6) for user in users.iterator(): user.charge() </code></pre>Note: Work with <code>week_days</code> in Django may be a bit surprising in the first moment. While in python <code>datetime</code> the weekdays go from Monday(0) to Sunday(6), when filtering in Django queries the number go from Sunday(1) to Saturday(7).So, as this method was being runned in a Celery task, to avoid accidents we did a decorator to prevent this method to be called and charge the user in a wrong weekday.<pre><code>def charge_weekday_guard(charge_week_day=None): def decorator(f): @wraps(f) def wrapper(*args, **kwargs): dt_now = timezone.now() if dt_now.weekday() == charge_week_day: return f(*args, **kwargs) else: logger.info('Tried to charge weekly user in wrong day.') return wrapper return decorator </code></pre>In the end, the function was wrote like above:<pre><code>@charge_weekday_guard(charge_week_day=6) def charge_wednesday_users(self): """week_day = 4, when filtering queries in Django, means Wednesday""" users = self.get_queryset().filter(delivery_dates__0__week_day=4) for user in users.iterator(): user.charge() </code></pre>So, in this example, we didn't remove code from the original function, neither improve the readability but we decrease the chances that something wrong could be done.You can read more interesting stuff about Django and Decorators here:<ul><li><a href="https://docs.djangoproject.com/en/1.10/topics/class-based-views/intro/#decorating-the-class">https://docs.djangoproject.com/en/1.10/topics/class-based-views/intro/#decorating-the-class</a></li><li><a href="https://hynek.me/articles/decorators/">https://hynek.me/articles/decorators/</a></li></ul>