Uploading files from the frontend to Amazon S3

<p>A common problem appears when uploading large files to Heroku. Every request made to Heroku must last less than 30 seconds or it will get terminated, when uploading large files, 30 seconds might not be enough. More information can be found <a href="https://devcenter.heroku.com/articles/request-timeout">here</a>.</p><p>One way to deal with this situation is to upload files to Amazon S3 directly from the browser. On this post we will show how to do this using Django.</p><p>There is currently a Django app that provides a complete solution for the problem and can be found <a href="https://github.com/bradleyg/django-s3direct">here</a>. If you need something a bit more flexible you might want to implement your own, if that is your case, continue reading and we will guide you through it.</p><p>The full example can be found on our <a href="https://github.com/vintasoftware/django-upload-files-straight-to-s3">repository</a>.</p><h2 id="configuring-s3">Configuring S3</h2><p>For this guide we will take into consideration that Django is already working with S3. For more information on how to get this setup going, take a look at <a href="http://www.vinta.com.br/blog/2015/how-to-configure-sass-and-bower-with-django-compressor-part-2.html">How to configure Sass and Bower with django-compressor - part 2 (deployment to Heroku and S3)</a>.</p><p>To get your bucket ready go to <a href="https://console.aws.amazon.com/S3/home">S3 bucket list</a>, right click on the bucket, then <em>Properties</em>, after <em>Permissions</em> and on <em>Add/Edit CORS Configuration</em> and fill as below:</p><pre><code class="language-xml"><?xml version="1.0" encoding="UTF-8"?> <CORSConfiguration xmlns="http://S3.amazonaws.com/doc/2006-03-01/"> <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedMethod>POST</AllowedMethod> <AllowedMethod>PUT</AllowedMethod> <AllowedHeader>*</AllowedHeader> </CORSRule> </CORSConfiguration> </code></pre><p>Once you have your domain you can change <code>AllowedOrigin</code> from <code>*</code> to <code>yourdomain.com</code>.</p><h2 id="model">Model</h2><p>For this post we will be using the following model:</p><pre><code class="language-python">from django.db import models import uuid def document_upload_path(instance, filename): return 'documents/{}/{}'.format(uuid.uuid4(), filename) class Document(models.Model): name = models.CharField(max_length=100) doc_file = models.FileField(upload_to=document_upload_path) </code></pre><h2 id="endpoint">Endpoint</h2><p>We are going to need an endpoint for the client to authenticate with S3. For that endpoint we use <a href="https://github.com/boto/boto">boto</a> to create the arguments for the POST request that will upload the file to S3.<br>The code for the endpoint is pretty generic and you should only need to change path where to upload the file. You can see the endpoint <a>here</a>.</p><h2 id="form">Form</h2><p>The form used to upload the document will look like:</p><pre><code class="language-python">from django import forms from django.forms import ValidationError from crispy_forms.helper import FormHelper from crispy_forms.layout import Submit from example.models import Document class DocumentForm(forms.ModelForm): # since we are not going to send the file via POST we need a field to save # the already uploaded path of the uploaded file. file_name = forms.CharField(required=False) class Meta: model = Document fields = ('id', 'name', 'doc_file', 'file_name') def __init__(self, *args, **kwargs): # crispy forms self.helper = FormHelper() self.helper.form_class = 's3upload-form' self.helper.add_input(Submit('submit', 'Submit')) super(DocumentForm, self).__init__(*args, **kwargs) # doc_file is not required if file_name is set self.fields['doc_file'].required = False self.fields['file_name'].widget.attrs['readonly'] = True def clean(self): file_name = self.cleaned_data.get('file_name') doc_file = self.cleaned_data.get('doc_file') # if doc_file and file_name is not set we are missing the file if not doc_file and not file_name: self.add_error( 'doc_file', ValidationError(self.fields['doc_file'] .error_messages['required'], code='required')) # if file_name is set with the path of the file it was uploaded by # the frontend elif not doc_file and file_name: self.cleaned_data['doc_file'] = file_name return super(DocumentForm, self).clean() </code></pre><h2 id="frontend">Frontend</h2><p>For the frontend we need to make a request to authenticate before we can POST to S3. The full code for the frontend can be found <a href="https://github.com/vintasoftware/django-upload-files-straight-to-s3/blob/master/example/static/js/upload.js">here</a>. The main part of the code will look similar to this:</p><pre><code class="language-javascript"> // first we need to get signature for authorization $.ajax('/example/documents/s3auth/?' + 'file_name=' + filename).done(function (data) { // now we can construct the payload with the signature var fd = new FormData(); for (var key in data.form_args.fields) { if (data.form_args.fields.hasOwnProperty(key)) { console.log(key, data.form_args.fields[key]); fd.append(key, data.form_args.fields[key]); } } fd.append('file', evt.target.files[0]); $.ajax({method: "POST", url: data.form_args.action, data: fd, processData: false, contentType: false, success: function(){ // set the hidden field to the uploaded file's path file_name.val(data.file_path); // clear the input_file so we don't send it when submitting the form input_file.val(''); }, error: function(){ // clear the field so the user can try again. input_file.val(''); file_name.val(''); } }); </code></pre><h2 id="extra">Extra</h2><p>You might want to make sure it works on localhost without S3. A nice way to achieve this is by creating a context_processor:</p><pre><code class="language-python"># app.context_processors.py from django.conf import settings def use_s3(context): # add flag to check on template if we should upload to S3 # on local host return {'USE_S3': settings.USE_S3} </code></pre><p>Append the context processor to your <code>settings.py</code>:</p><pre><code class="language-python"># Django 1.8+ TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'app.context_processors.use_s3', # add this! ], }, }, ] # Django <1.8 TEMPLATE_CONTEXT_PROCESSORS = ( "django.contrib.auth.context_processors.auth", "django.core.context_processors.debug", "django.core.context_processors.i18n", "django.core.context_processors.media", "django.core.context_processors.static", "django.core.context_processors.tz", "django.contrib.messages.context_processors.messages", 'app.context_processors.use_s3', # add this! ) # and set USE_S3 USE_S3 = False </code></pre><p>Now on your template you can check for USE_S3:</p><pre><code class="language-html"> {% if USE_S3 %} <script> $( document ).ready(function() { // call your frontend code! }); </script> {% endif %} </code></pre><p>Leave your comments down bellow!</p>