It is DjangoCon time! As avid participants, we are super excited for this year's edition of DjangoCon US. We've been participating in the event since 2017 and sponsoring it since 2018. This community built around the framework we love has a special place in our hearts, and we are excited to be able to sponsor the event again this year.
DjangoCon comes back in a hybrid model. A necessity after the COVID-19 pandemic still discourages many people from attending in-person events. Since the situation has improved from last year, the organizers felt it was finally possible to go back to having talks at a venue. In their own words, the event: "has something for everyone, from the person who develops Django applications for a living to the person who just tinkers in their spare time."
If you still haven't attended, we strongly recommend you check their profiles here and here. This way, you can check talks from past editions and stay in on the news about where the conference will happen next year!
This year, the conference will run for six days. It is distributed, as usual, between talks, tutorials, and sprints. One of our partners, Flávio Juvenal, will discuss how DRY can be a double-edged sword and the lib we built to help circumvent related issues! It will air on October 19th at 2:00 PM PDT. If that schedule doesn't work for you, no worries; a recording will be available on their YouTube channel.
Our talk at the event
Why large Django projects need a data (prefetching) layer
Speaker: Flavio Juvenal
Slides presented: Link to slides
Talk recording: Link to talk
About the talk:
The main building blocks of Django REST Framework projects, i.e., Views, Serializers, Managers, and Querysets, allow developers to implement complex APIs with minimal code repetition while reusing built-ins for essential API features. Developers feel guided by DRF to architect the project in a "Don't Repeat Yourself" way by using inheritance, nesting, annotations, and model / app-based separation of concerns. They can group code in viewsets, inherit from base classes, reuse the same serializer across views, nest serializers into others, compute fields dynamically with ORM annotations, select or prefetch relations for performance, organize custom behavior with managers and querysets, and much more. All this DRYness is excellent because it integrates well with common web API concerns like permissions, pagination, and filters, among others.
Based on our multi-year experience in building and maintaining several large Django projects, while using those built-in concepts yields a DRY code, the overuse also results in a codebase full of complicated bugs and performance issues, especially related to ORM usage. View, serializer, and model methods are often heavily coupled to querysets' annotations and prefetches, but those methods are spread across the codebase. Django's default queryset laziness and heavy usage of inheritance and nesting is the perfect recipe for a codebase where N+1 issues and heavy unnecessary queries can happen in any line of code after some less careful change.
For example, to prevent N+1 issues, if a serializer method field uses a filtered relationship, you must ensure this relationship is prefetched in all querysets related to that serializer. But this serializer can be nested into others, so you must carefully change all queryset references in seemingly unrelated views. Other "change amplification" situations also happen on large DRF codebases with heavy ORM usage. Requiring developers to be careful while navigating many files to perform changes isn't reasonable. Maybe being DRY is leading to the wrong abstraction?
Designing a better architecture optimized for enabling changes and avoiding performance regressions is possible. With a new custom data prefetching layer that keeps compatibility with serializers and views, we can respect DRY while keeping performance and maintainability. That's what we've been doing in our Django projects, and we will share our learnings in this talk. That applies to other maintainers of complex DRF projects.
