Showing posts with label django. Show all posts
Showing posts with label django. Show all posts

Easily Integrate React.js to a Django Website for dev and production builds

Assuming you already have a Django website up and running, this will get you started with compiled React.js on the front-end with:

  • Django and Webpack integrated nicely together during development and deployment
  • Support for React.js code in your website
  • Packaged and obfuscated Javascript code
  • SCSS support
  • Proper sourcemaps for dev builds
  • Hot reloading of (S)CSS changes during development, automatic page reloading for JS changes

Throughout this tutorial, it treats the client (browser) and django (server) projects as separate codebases. I'll try to be as clear as possible to which one we're currently working on.

This post was meant to be written much earlier, but things happened...

Zelda: Breath of the Wild took over my free time

      Setup

      • Grab the latest Node.js (6.10.0)
      • Make sure npm is up to date. Comes with Node if you upgraded it (currently using 3.10.8)
      • Install yarn globally (currently using 0.21.3)

      npm install -g yarn

      • Install create-react-app globally (currently 0.9.4 on github, but install says 1.3.0? so confusing)

      npm install -g create-react-app

      Note: don't use 0.9.3 if you want hot-reloading without some extra work.

      Prepping Django for integration with Webpack

      This side is much simpler, so we'll get it out of the way first.

      • In your Django project folder, install "django-webpack-loader" (v0.3.3 at time of writing):

      pip install django-webpack-loader

      • Add "webpack_loader" to INSTALLED_APPS in settings.py
      • Now to add WEBPACK_LOADER in settings.py. Take note that webpack loader will load different files depending on the value of settings.DEBUG.

      # Webpack loader
      WEBPACK_LOADER = {
          'DEFAULT': {
              'BUNDLE_DIR_NAME': 'bundles/',
              'STATS_FILE': os.path.join(BASE_DIR, 'webpack-stats-dev.json' if DEBUG else 'webpack-stats-live.json'),
          }
      }

      • Using your knowledge of setting up Django views and templates, point "your-react-view" url to this basic template react.html

      from django.shortcuts import render

      def your_react_view(request):

          return render(request, 'react.html')

      • The react.html file should contain:

      <!doctype html>
      < html lang="en">
        <head>
          <meta charset="utf-8" />
          <title>Testing</title>
        </head>

        <body>
          <div id="root"></div>

          {% load render_bundle from webpack_loader %}      
          {% render_bundle 'main' %}
        </body>
      < /html>

      In the template is simply an empty html5 document which imports the "webpacked" files and an empty div with the ID "#root" is used for the mounting point for the React code (see client/src/index.js).

      It also imports the "main" bundle compiled by webpack and executes it.

      Those are the 3 most important lines as they bring in the webpacked content to the page of your liking.

      Creating a front-end React/Webpack project

      Now for the nitty gritty. All the jokes and complaints you've heard about JS compilation and complexity are somewhat true. This guide will probably be out of date by the time you finish reading it!

      But don't worry, this tutorial will get you through the hardest part (the setup) by being as clear as possible, letting you know what you're changing and why.

      In a console, go to your Django project folder and type:

      create-react-app client
      cd client/
      yarn start

      I'm using the name "client" for the browser project, but you can quite easily call it "frontend" or whatever else you fancy.

      After typing "yarn start", it should pop up a browser tab/window while compiling and show you that it's working.

      At the moment it's a completely separate website to your Django website, so we'll have to make some changes to link them together.

      Ejecting from  create-react-app bootstrap

      So far it's very barebones and there are no files in client project that you can modify to change the build process.

      To make any sort of customisations, you will need to eject the project so there are files to configure. This means leaving the safety nest of create-react-app. It's a one way process, but you'll be far better off afterwards.

      • In the console for the client project, type in:

      yarn eject

      • Press Y to confirm (I got an error at the end of it, but the ejection process seemed to work just fine)
      • Now looking through the client project folder, you have lots of new files to modify!
      • Let's test that the client project still works with:

      yarn start

      Customisations

      By the time we're done with this section, your config should:
      • create production build to your-django-project/media/client/(css|js|media)
      • use SCSS in your code (it's great for managing nested CSS)
      • produce a webpack-stats-(dev|prod).json file for django integration
      • have a working dev build integrated with Django (served via websockets, so no output folder)

      Before we get started, there are some downloads needed for the extra features we want.

      • In the client project folder, type in:

      yarn add webpack-bundle-tracker node-sass sass-loader

      Current versions are:

        • webpack-bundle-tracker@0.2.0
        • sass-loader@6.0.2
        • node-sass@4.5.0

      While we're waiting, lets get rid of the annoying a browser tab that opens everytime we run yarn start!

      • Open up client/scripts/start.js and search for any lines of code with openBrowser
      • There should only be two lines using it; the import and call.
      • Terminate with extreme prejudice.

        Changes to client/config/webpack.config.dev.js

        • Under "var paths = require('./paths')", add in:

        var BundleTracker = require('webpack-bundle-tracker');

        • Replace "var publicPath = '/';" with:

        var publicPath = 'http://' + paths.serverHostname + ':3000/assets/bundles/'; // override django's STATIC_URL for webpack bundles

        • Change module.exports > output > path to paths.appBuildDev
        • Under module.exports > resolve, add in root: paths.appSrc,
        • Under module.exports > module > preloaders, under 'include' add in:

        exclude: /node_modules/

        There is no need to lint check modules that we shouldn't be changing.

        • Under module.exports > module > loaders > first entry > exclude, change:

        /\.css$/,

        to

        /\.(css|scss)$/,

        • Under module.exports > module > loaders, add in the following after the .css test:

        // SCSS support
        {
          test: /\.scss$/,
          loaders: ["style", "css", "sass"]
        },

        • Search for HtmlWebpackPlugin and disable/remove it
        • And at the very end of module.exports > module > plugins, add in:

        // For django to know about webpack
        new BundleTracker({ filename: '../webpack-stats-dev.json' })

        Note: the dev specific filename

        • Search for devtool and change it from 'cheap-module-source-map' to 'source-map'.

        We don't want to go cheap while we're debugging! More information is always better.

        Changes to client/config/webpack.config.prod.js

        • Under "var paths = require('./paths')", add in:

        var BundleTracker = require('webpack-bundle-tracker');

        • Under module.exports > module > preloaders, add the follow under 'include':

        exclude: /node_modules/

        There is no need to lint check modules we shouldn't be changing.

        • Under module.exports > resolve, add in root: paths.appSrc,
        • Under module.exports > module > loaders > first entry > exclude, change:

        /\.css$/,

        to

        /\.(css|scss)$/,

        • Under module.exports > module > loaders, and after the .css test, add in:

        // SCSS support
        {
          test: /\.scss$/,
          loader: ExtractTextPlugin.extract('style', 'css?-autoprefixer!postcss!sass')
        },

        • Search for HtmlWebpackPlugin and disable/remove it
        • Same with ManifestPlugin
        • And at the very end of module.exports > module > plugins, add in:

        // For django to know about webpack
        new BundleTracker({ filename: '../webpack-stats-prod.json' })

        Note: the production specific filename

        • I'd strongly suggest disabling sourcemaps by commenting out "devtool" for production so people can't view your plain text source code unless you have something in place for specifically your map files.

        Changes to client/config/paths.js

        • In getPublicUrl(), change the one liner to:

        return envPublicUrl || require(appPackageJson).homepage || '/media/';

        • Scroll to the bottom to find "module.exports"
        • Comment out or remove appHtml, we don't need it anymore.
        • Change appBuild to:

        appBuild: resolveApp('../media/'),

        • And add these to module.exports:
        appBuildDev: resolveApp('build'),   

        serverHostname: 'localhost',

        Changes to client/scripts/build.js

        Replace the following line

        fs.emptyDirSync(paths.appBuild);

        With:

        fs.emptyDirSync(paths.appBuild + '/client/');

        This line empties the output folder each time you run yarn build. The change ensures your existing media folder doesn't get wiped each time.

        Changes to client/scripts/build.js and client/scripts/start.js

        • Remove paths.appHtml from

        if (!checkRequiredFiles([paths.appHtml, paths.appIndexJs])) {

        You can also take the time to remove the files:

        • client\public\favicon.ico
        • client\public\index.html

        Phew, we're mostly done with the configuration!

        Switching to SCSS

        Now let's switch to SCSS. Rename client/src/App.css to client/src/App.scss and update the import in App.js from:

        import './App.css';

        to:

        import './App.scss';

        It's a relatively minor change, but makes a world of difference when writing neatly nested CSS classes. You can also define variables and macros/functions when defining styles.

        Testing your setup

        Dev setup

        • Build your project:

        yarn start

        • And run Django as per usual:

        python manage.py runserver

        Production setup

        • Build your project:
        • In your production settings.py file, make sure you're reading webpack information from webpack-stats-prod.json in the WEBPACK_LOADER setting.
        • Compile your production browser project by typing:

        yarn build

        • Check for output in DjangoPath/media/client/

        Enabling hot loading

        You may have noticed some 404's in the Django log to /sockjs-node/info while testing your setup.

        That's webpack hot-loading looking for the hot-reload websocket signal in the wrong place.

        In webpack.config.dev.js, disable/remove the line:

        require.resolve('react-dev-utils/webpackHotDevClient')

        And replace it with:

        require.resolve('webpack-dev-server/client') + '?http://' + paths.serverHostname + ':3000',
        require.resolve('webpack/hot/dev-server'),

        The default webpackHotDevClient provided by create-react-app is neat, but isn't configurable so makes it's useless for our setup. So we have to switch it out for the one that comes with webpack-dev-server which allows us to set the webpack server location.

        Hot-reload bug with create-react-app 0.9.3

        Please note that there is a bug which prevents webpack-dev-server/client from working properly with create-react-app v0.9.3 (or v1.2.1?). The bug was fixed in 0.9.4.

        If you're on v0.9.3 and want hot reloading, you can just fix this yourself by editing webpack.config.dev.js.

        • Find the following line under module > loaders > first entry > exclude:

        /\.(js|jsx)$/,

        • And replace it with:

        /\.(js|jsx)(\?.*)?$/,

        Reason being Webpack was stripping out the query information after the "?" on the line where webpack-dev-server/client was imported. For further information about the bug, see this pull request.

        Other recommendations

        Use JSX file extension

          Switching your React component code to use the file extension jsx instead of js. Some editors will behave better for React specific code.

          • Rename index.js to index.jsx and App.js to App.jsx
          • Remember to change paths.appIndexJs accordingly.  

          Disabling chunk hash on output filenames

            For production, sometimes you may find it useful to not include hashes in your code filenames.

            In webpack.config.prod.js:

            • output.filename: 'client/js/[name].js'
            • plugins at the end: new ExtractTextPlugin('client/css/[name].css')

            I would definitely keep it in for media files though as they don't change often.

                  Change dev build output filename from "bundle.js"

                  You'll need to change this if you want to support code splitting/chunks.

                    Delete logo.svg

                      It's likely you'll be removing the demo code so why not the demo assets too?

                      That's about it for now. For me it's back to Zelda...

                      Sources

                      Updates

                      22/05/2017

                      • Fixed output folder from static to media
                      • Fixed bug with SCSS output in the wrong order
                      • Removing file hashes from output.
                      • Removed the section about /static/static

                      10/03/2017

                      • Forgot to remove ManifestPlugin from webpack.config.prod.js
                      • Added module.exports > resolve > root

                      Django: "AppRegistryNotReady: Apps aren't loaded yet" error after upgrading from 1.6 (to 1.7 or 1.8)

                      Depending on your setup, this might be a troublesome one to figure out. You won't see any error emails coming through since it doesn't hit Django code yet, just 500 error page when trying to view the website.

                      With the changes to the WSGI code in Django, some of us may have forgotten to update the WSGI dispatcher which is needed to glue our project to the HTTP server.

                      To fix the error, you'll need to open up your WSGI dispatcher file and change the following lines:

                      import django.core.handlers.wsgi
                      application = django.core.handlers.wsgi.WSGIHandler()

                      to

                      from django.core.wsgi import get_wsgi_application
                      application = get_wsgi_application()

                      That should ensure that everything is smooth sailing again.

                      KFWNVz8

                      Source

                      Django: Export comments to Disqus WXR / XML format

                      With the help of the django-disqus module, it's pretty easy to export our comments out to Disqus.

                      First you'll need to install the module django-disqus (v0.4.3 at time of writing).

                      The example provided was used when I switched www.thatawesomeshirt.com over to Disqus. The class used is loosely based off the regular class used in the syndication module.

                      urls.py

                      from tas.feeds import ShirtCommentsWxrFeed

                      urlpatterns += patterns('',
                      ('^export/comments/', ShirtCommentsWxrFeed()),
                      )

                      feeds.py

                      As you can see, this is where all the heavy lifting happens.

                      Once you have the output, you can import it more than once without creating duplicates. I believe duplicate detection is done using the value from comment_id().

                      image

                      In your Disqus admin:

                      • Go to Discussions > Import > Generic (WXR) (using WordPress one will give you strange errors regarding the thread)
                      • Upload the WXR file generated
                      • Wait until it's done

                      I've noticed that files which are less than 10mb are processed rather quickly. Anything closer to the 50mb limit will take almost 24hrs to process.

                      All in all, it's probably one of the best processes for migrating comments that I've used so far.

                      like-a-boss_o_177339

                      Almost as good as this guy.

                      Sources

                      Python: Visualise your class hierarchy in UML

                      Sometimes it's handy to graph out the way your classes are arranged, either for training purposes or simply to help you plan your next move.

                      There's always the option of doing it manually, or you can use some handy tools to do so.

                      To get the ball rolling, grab pylint and graphviz

                      sudo apt-get install graphviz libgraphviz-dev python-dev
                      pip install pylint pygraphviz

                      If you don't have python-dev installed, you'll run into the error below:

                      pygraphviz/graphviz_wrap.c:124:20: fatal error: Python.h: No such file or directory
                      #include <Python.h>

                      Now back in your project folder, type:

                      pyreverse -my -A -o png -p test **/classes.py

                      Replace "classes.py" with "**.py" if you want all python files.

                      Another example is if you wanted to graph all the form classes:

                      pyreverse -my -A -o png -p test **/forms.py

                      Once it's done, you'll now have classes_test.png and packages_test.png in the folder. The packages image allows you to determine if your code modules are properly decoupled. The classes files shows you how twisted your class inheritance may be.

                      classes_test

                      Django models

                      For Django specific projects, you can install a module called "django_extensions". It also uses graphviz to generate some cool charts but is very Django friendly so you don't have to manually remove Meta classes from your models.

                      The syntax for graphing models is:

                      python manage.py graph_models -g -e -l dot -o my_project.png module_a module_b module_c
                      • -g groups the models into their respective apps for easier viewing
                      • -e shows the inheritance arrows
                      • -l uses the "dot" rendering layout by GraphViz (possible values are: twopi, gvcolor, wc, ccomps, tred, sccmap, fdp, circo, neato, acyclic, nop, gvpr, dot, sfdp. I found dot to be the only legible one, and most of them didn't work for me)
                      • "-o my_project.png" is the output file
                      • module_x includes all the modules which you want to draw

                      directory_listing

                      Django: How to read current select_related() values

                      This was a bit of a tricky bit of information to get out. The information for select_related() is stored as a nested map. You'll need some recursive code to fetch that back out.

                      def generate_field_key(map, key, ancestor=None):
                      values = []
                      if ancestor:
                      parent = '%s__%s' % (ancestor, key)
                      else:
                      parent = key
                      values.append(parent)
                      for k, v in map.items():
                      values.extend(generate_field_key(v, k, parent))
                      return values


                      # Select related is a nested map
                      qs = Something.objects.all().select_related('hey', 'hey__there')
                      select_related_data = qs.query.select_related
                      select_related = []

                      # This can be a map or False
                      if select_related_data:
                      for key, items in select_related_data.items():
                      select_related += generate_field_key(items, key, None)

                      print select_related

                      6uudsa0n9dqhs3ebyzvzlpxm
                      All good?

                      Django: Adding images to RSS feed items

                      It was a bit tricky trying to find information on how to do this and StackOverflow seemed to be giving advice of all sorts except for code that actually worked.

                      aXbnBw9_460sa
                      StackOverflow in a nutshell. Sometimes it feels like people are just answering for the upvotes and not to actually answer the question...

                      Below is a working sample which uses Django's syndication framework the way it's meant to be used but also checks out nicely with any well known validator.

                      We create a class called MediaRssFeed which subclasses a built-in feed generator called Rss201rev2Feed.

                      MediaRssFeed allows us to declare the namespaces required for the thumbnail markup to be valid, but also allows us to insert additional elements into the item as it is rendered.

                      In our LatestNewsFeed class, we simply replace feed_type with MediaRssFeed and add a new function item_extra_kwargs() which returns information for the feed generator to use.

                      from django.contrib.syndication.views import Feed
                      from django.utils.feedgenerator import Rss201rev2Feed

                      class MediaRssFeed(Rss201rev2Feed):
                      """
                      Implement thumbnails which adhere to Yahoo Media RSS (mrss) feed.

                      @see http://djangosnippets.org/snippets/1648/
                      """
                      def rss_attributes(self):
                      attrs = super(MediaRssFeed, self).rss_attributes()
                      attrs['xmlns:dc'] = "http://purl.org/dc/elements/1.1/"
                      attrs['xmlns:media'] = 'http://search.yahoo.com/mrss/'
                      return attrs

                      def add_item_elements(self, handler, item):
                      """
                      Callback to add thumbnail element to each item (item/entry) element.
                      """
                      super(MediaRssFeed, self).add_item_elements(handler, item)

                      thumbnail = { 'url': item['thumbnail_url'] }

                      if 'thumbnail_width' in item:
                      thumbnail['width'] = str(item['thumbnail_width'])

                      if 'thumbnail_height' in item:
                      thumbnail['height'] = str(item['thumbnail_height'])

                      handler.addQuickElement(u"media:thumbnail", '', thumbnail)


                      class LatestNewsFeed(Feed):
                      feed_type = MediaRssFeed

                      def item_extra_kwargs(self, article):
                      """
                      Return a dictionary to the feedgenerator for each item to be added to the feed.
                      If the object is a Gallery, uses a random sample image for use as the feed Item
                      """
                      image = article.get_main_image()

                      item = {
                      'thumbnail_url': generate_image_url(image, 300, 150, absolute = True),

                      # Optional
                      'thumbnail_width': 300,
                      'thumbnail_height': 150,
                      }

                      return item


                      # The rest of your feed class here as usual

                      Source

                      Varnish: Setting up Varnish as your ESI cache (with a Django backend example)

                      Varnish is a load balancer, an extra layer of server software which sits in place of your web server (Apache, nginx, etc) listening to port 80 (or whatever you configure) and waits for HTTP requests. It then forwards the request to the server which isn't overloaded.

                      After the response is returned by your web server, Varnish will detect any ESI fragments which need replacing and cache the fragment. Varnish will not request the fragment from the server again until the content expires.

                      The data fragments are returned from Varnish's memory cache, which is pretty damn quick. This saves your server a whole heap of computational and database load.

                      This setup has an additional benefit of making it easy to clear the cache of a specific ESI fragment.

                      Setting up Varnish

                      You configure Varnish by editing the file at /etc/varnish/default.vcl. The VCL script file tells it how to behave when certain information comes through.

                      Since Varnish is running on port 80 as your primary point of contact, you'll have to tell it:

                      • where your web servers are
                      • clear any varying HTTP headers and cookie information when a URL contains "esi" (so caching works for every request)
                      • check response for a custom header called "X-ESI-max-age" or "esi-enabled" and enable ESI parsing
                      • "X-ESI-max-age" is removed before the request is returned to the user
                      # Server/port where your HTTP server runs
                      backend default {
                      .host = "127.0.0.1";
                      .port = "82";
                      }

                      sub vcl_recv {
                      # Assuming that all your cacheable fragments/views contain the pattern /esi/ in the URL.
                      # You can add other patterns too, but for the sake of this tutorial we'll just use this.
                      if (req.url ~ "/esi/") {
                      # Remove variances so it caches for every request
                      unset req.http.Vary;
                      unset req.http.cookie;
                      }
                      }

                      sub vcl_fetch {
                      if (beresp.http.Pragma ~ "nocache") {
                      return(pass);
                      }

                      # This tells Varnish that we want to "varnish-cache" this page
                      # Check for our custom header
                      if (beresp.http.X-ESI-max-age == "1") {
                      # Remove custom header
                      unset beresp.http.X-ESI-max-age;
                      unset beresp.http.set-cookie;
                      esi;
                      }

                      # This tells Varnish that we want ESI processing on this page
                      if (beresp.http.esi-enabled == "1") {
                      esi;
                      }
                      }

                      To test Varnish, run it and type "start". If there are any errors, check your indenting or syntax. Now that's Varnish all set up.

                      Coding!

                      Now for the fun part, tweaking your webpages! Add in some settings to "settings.py" so it's easier to configure.

                      VARNISH_USE_ESI = True # Quick kill-switch
                      VARNISH_SERVER = "localhost" # Your HTTP server
                      VARNISH_PORT = 80 # The port of your HTTP server

                      I've added a module called "varnish" to store the following code. This helper function goes into "varnish/utils.py"

                      from django.utils.cache import patch_cache_control

                      def esi(response, use_esi = False, max_age = 0):
                      """
                      This is a helper function which sets the HTTP response headers
                      required to enable ESI caching and allow for configurable cache lifetimes.
                      """
                      # Use ESI so template fragments are parsed
                      if use_esi:
                      response['esi-enabled'] = "1"

                      # Specify cache time on the ESI views
                      if max_age > 0:
                      response['X-IDG-ESI-max-age'] = "1"
                      patch_cache_control(response, max_age = max_age)

                      return response

                      As you can see, it conditionally sets the custom headers which the VCL script is expecting.

                      A Python decorator simply wraps around a function and can modify the input/output of the given function. The following code is for a decorator varnish() in "varnish/decorators.py", which basically makes it easier to use esi().

                      def varnish(use_esi = False, max_age = 0):
                      """
                      This decorator calls the esi() function to modify
                      the response headers from a view.
                      """
                      def wrap(func):
                      def wrapper(*args, **kwargs):
                      response = func(*args, **kwargs)
                      esi(response, use_esi = use_esi, max_age = max_age)
                      return response

                      return wrapper

                      return wrap

                      In order for Varnish to know that we want a cacheable code fragment view, we need to add into the HTML:

                      <esi:include src="/path/to/your/esi/cached/view/" />

                      It's important that we have "/esi/" in the URL pattern as we've configured that pattern in the VCL script. Varnish will attempt to fill it in automatically from cache, or fetch the include URL from your server if necessary.

                      The following code is for a Django template tag in "varnish/templatetags/varnish.py" which I use to quickly write "esi:include" tags when ESI is enabled, or output the fragment content directly into the template when ESI is disabled.

                      You can find ContextNode here.

                      from django import template
                      from django.conf import settings
                      from django.template import TemplateSyntaxError, resolve_variable
                      from django.core import urlresolvers

                      from twig.utils import ContextNode

                      register = template.Library()

                      @register.tag
                      def esi_cache(parser, tokens):
                      """
                      Usage: Output a HTML fragment, either cache request
                      to Varnish ESI or full HTML output.
                      {% url esi-view-name object.id as esi_url %}
                      {% esi_cache esi_url %}
                      """
                      bits = tokens.split_contents()

                      if len(bits) != 2:
                      raise TemplateSyntaxError('%s expects 2 arguments' % bits[0])

                      def wrap(context):
                      url = resolve_variable(bits[1], context)

                      if settings.VARNISH_USE_ESI:
                      esi_url = "http://%s:%s%s" % (settings.VARNISH_SERVER, settings.VARNISH_PORT, url)
                      return '<esi:include src="%s"/>' % esi_url

                      else:
                      # If we're not using ESI, we can just plug in the view output directly
                      esi_url = url

                      # Otherwise call the view and return the data
                      # @see http://djangosnippets.org/snippets/1568/
                      resolver = urlresolvers.RegexURLResolver(r'^/', settings.ROOT_URLCONF)
                      view, args, kwargs = resolver.resolve(esi_url)

                      if callable(view):
                      return view(context['request'], *args, **kwargs).content
                      else:
                      # This gives a nicer error email in case it ever happens
                      raise TemplateSyntaxError('Error retrieving "%s"' % esi_url)

                      return ContextNode(wrap)

                      Putting it all together

                      For example you have a URL pattern named "esi-product-summary" in your urls.py file.

                      urlpatterns = patterns('',
                      url(r'^product/esi/summary/(?P<id>\d+)/$', 'products.views.esi_summary', name = 'esi-product-summary'),
                      )

                      To use ESI fragments in the template:

                      <div class="reviews">
                      {% for product in products %}
                      {% url esi-product-summary product.id as esi_url %}
                      {% esi_cache esi_url %}
                      {% endfor %}
                      </div>

                      The built-in tag {% url %} generates the URL and stores it into a variable called "esi_url".

                      If VARNISH_USE_ESI is enabled, then {% esi_cache %} outputs the <esi:include src="/product/esi/summary/123"/> element into the skeleton template.

                      This skeleton template is then returned to Varnish, which detects the missing fragments and fills them in (if it's not already cached) by making extra HTTP requests to the server for each fragment.

                      When all the fragments are collected, all the parts are put together and returned to the user. Sounds lengthy but it all happens very quickly, especially when it's already cached.

                      Controlling cache expiry

                      So how long does it take to expire? You can configure that in your view. Just add a simple decorator and it'll "just work".

                      # This tells Varnish to cache it for 1800 seconds (30mins)
                      from varnish.decorators import varnish

                      @varnish(max_age = 1800) def esi_summary(request, id):
                      # The following code has no impact on the ESI stuff.
                      c = {
                      'product': Product.objects.get(id = product_id),
                      }
                      return render(request, 'products/esi/summary.html', c)

                      This code is a bit more verbose than it needs to be, but that's mainly due to the extra option USE_ESI.

                      Now your site is less likely to crumble when a thundering herd of traffic comes your way!

                      0uNuQ
                      Prepare yourselves, reddit/digg/slashdot is only a link away!

                      Enabling ESI on ALL views

                      If you're using an ESI fragment on a base template, then it may be in convenient for you to enable ESI site-wide. You can either do this through the VCL config file or using Django middleware.

                      Here's the middleware if you need it:

                      class VarnishEnableSitewideEsi():
                      """
                      This enables ESI across ALL views on the site which are text/html.
                      """
                      def process_response(self, request, response):
                      mimetype = response['Content-Type']
                      mimetype = mimetype.split(';')[0]

                      if mimetype == "text/html":
                      return esi(response, use_esi = True)

                      return response

                      Sources

                      Python/Django: Dealing with UTF8 in URLs/URIs

                      UTF8 is a nice set of characters to use, but one must remember that the standard for URL encoding has to be in ASCII. You've probably run into it before with the Python urllib and urllib2 libraries when encoding issues are raised. These are valid exceptions!

                      92JJn
                      Yeah that's right bitch, urllib ain't taking your UTF8 shit.

                      "But it just works in my browser!" you may be thinking. Yes, most modern browsers will automatically convert the UTF8 URL into an ASCII request behind the scenes without showing it to you.

                      So as a developer, if you have a URL containing UTF8 characters then you'll have to convert it to ASCII before you can send a request.

                      If you're using Django, then you have some nice helper functions to deal with this. By using django.utils.encoding.iri_to_uri(), you can simply convert the UTF8 portions of the URL into ASCII and keeping everything else unmodified.

                      If you're not using Django... I guess you can try to incorporate their madness from their encoding module source file.

                      Examples

                      First make sure your URLs are properly encoded in Unicode (note the little "u" in front of the string when defining "url").

                      Take for example this URL which contains UTF8 characters. A quick test shows: http://twigstechtips.blogspot.com/seârch/labél/pythön

                      >>> url = 'http://twigstechtips.blogspot.com/seârch/labél/pythön'
                      >>> print url
                      # http://twigstechtips.blogspot.com/se├órch/lab├®l/pyth├Ân


                      >>> url = u'http://twigstechtips.blogspot.com/seârch/labél/pythön'
                      >>> print url
                      # http://twigstechtips.blogspot.com/seârch/labél/pythön

                      But what about GET args? Don't worry, iri_to_url() deals with them too. For example: http://twigstechtips.blogspot.com/seârch/labél/pythön?query=djångõ

                      >>> url = u'http://twigstechtips.blogspot.com/seârch/labél/pythön?query=djångõ'
                      >>> print iri_to_uri(url)
                      # http://twigstechtips.blogspot.com/se%C3%A2rch/lab%C3%A9l/pyth%C3%B6n?query=dj%C3%A5ng%C3%B5

                      And it also works for UTF8 domains too (yes, they exist!). In this instance, we'll use http://camtasia教程网.com:

                      >>> url = u'http://camtasia教程网.com'
                      >>> print url
                      # http://camtasia教程网.com


                      >>> print iri_to_uri(url)
                      # http://camtasia%E6%95%99%E7%A8%8B%E7%BD%91.com

                      I'll be honest, I'm not quite sure what the heck an IRI is but this magic function works as advertised.

                      39gyh1taajden89mwf8v331ug
                      BOOM! It just works.

                      On a final note, if you're after UTF8 friendly versions of urllib.quote() and urllib.quote_plus(), then there are also:

                      The functions django.utils.http.urlquote() and django.utils.http.urlquote_plus() are versions of Python’s standard urllib.quote() and urllib.quote_plus() that work with non-ASCII characters. (The data is converted to UTF-8 prior to encoding.)

                      Source

                      Django: Upload files to a dynamic path

                      For the projects I'm working on, we have a reusable "image file" model which stores information about an image (such as height, width, hash, image type, etc) and automatically stashes it to Amazon S3.

                      Any time you want to use an image in another model, you can just refer to it through a foreign key.

                      class BasicImage(models.Model):
                      caption = models.CharField(max_length=300, blank=True)
                      # ...
                      image = models.ImageField(storage = s3_storage, upload_to = "images", max_length = 500)

                      def __unicode__(self):
                      return "%s - %s" % (self.caption, self.image)

                      And an example of a  model which uses BasicImage.

                      class UserProfile(models.Model):
                      # ...
                      profile_picture = models.ForeignKey(BasicImage)

                      Whenever a user uploads a profile picture, it'll be uploaded to a folder called "images", along with every other image file. That's fine and dandy if you're ok with mixing user profile pictures with other images (such as company logos or user photos).

                      However, most people would like to separate things out a bit. Here's a relatively clean way of preventing a huge binary mess. (Key points are listed below)

                      def _image_upload_path(instance, filename):
                      # To customise the path which the image saves to.
                      return instance.get_upload_path(filename)


                      class BasicImage(models.Model):
                      caption = models.CharField(max_length=300, blank=True)
                      # ...
                      image = models.ImageField(storage = s3_storage, upload_to = _image_upload_path, max_length = 500)

                      def __unicode__(self):
                      return "%s - %s" % (self.caption, self.image)

                      def get_upload_path(self, filename):
                      return "dimg/%s" % filename


                      # Required to add new files to a different path
                      class UserProfileImage(BasicImage):
                      class Meta:
                      proxy = True # So this model doesn't actually need a new table

                      def get_upload_path(self, filename):
                      return "avatars/%s" % filename


                      # Updated user model so any pictures will be uploaded to /avatars folder
                      class UserProfile(models.Model):
                      user = models.OneToOneField(User, primary_key = True)
                      # ...
                      profile_picture = models.ForeignKey(UserProfileImage)

                       

                      Take note of the differences:

                      • Create a helper function called _image_upload_path()
                      • Add BasicImage.get_upload_path(self, filename)
                      • You'll need to use BasicImage as a base class for any other image models
                      • Other image models should be a proxy class unless they contain additional information
                      • Override get_upload_path() in your subclasses if you wish to customise the upload folder

                      abbot-rudder
                      Now you can upload random pictures like this into a separate folder

                      Sources

                      Django: "AssertionError - No exception supplied" on "assert salt and '$' not in salt"

                      After upgrading from Django 1.2.7 to Django 1.4.3, we had a problem where users were unable to log in.

                      This was due to an upgrade in the crypto algorithms used to store passwords.

                      When a user attempts to log in, the authentication process checks the "password" field in the User table.

                      The password field contains 3 components: algorithm$salt$hash

                      The algorithm is used to determine which crypto to use. The salt is a randomly generated salt upon setting of password, which in this case is an empty string "".

                      For now, you can temporarily fix this error by adding a new file "working_unsalted\hasher.py".

                      from django.contrib.auth.hashers import UnsaltedMD5PasswordHasher
                      from django.utils.crypto import constant_time_compare

                      class WorkingUnsaltedMD5PasswordHasher(UnsaltedMD5PasswordHasher):
                      """
                      The default UnsaltedMD5PasswordHasher uses constant_time_compare(), but passes it the wrong values.
                      """
                      algorithm = "working_unsalted_md5"

                      def verify(self, password, encoded):
                      encoded_2 = self.encode(password, '')
                      return constant_time_compare(encoded[22:], encoded_2)

                      In your settings file, be sure to define the new hasher.

                      PASSWORD_HASHERS = (
                      'django.contrib.auth.hashers.BCryptPasswordHasher',
                      'django.contrib.auth.hashers.PBKDF2PasswordHasher',
                      'django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher',
                      'django.contrib.auth.hashers.SHA1PasswordHasher',
                      'django.contrib.auth.hashers.UnsaltedMD5PasswordHasher',
                      'django.contrib.auth.hashers.MD5PasswordHasher',
                      'django.contrib.auth.hashers.CryptPasswordHasher',

                      'working_unsalted.hashers.WorkingUnsaltedMD5PasswordHasher',
                      )

                      Lastly, change the algorithm in the database by changing auth_user.password values from "md5$..." to "working_unsalted_md5$...". When the user tries to log in, the new hasher will take effect.

                      I've made a ticket and pull request for this issue, so hopefully it'll be fixed in Django 1.4.4.

                      o0480085412073412180
                      For now, enjoy your copy-pasta fix.

                      Django: Fix CSRF Token Cookie not generating for non-html pages

                      This was a bit of a tricky one to debug. I was creating an API to log into the site via an app which communicated through JSON.

                      If the user was already logged in, the JSON response contained:

                      • the account details
                      • session ID (automatically set)
                      • the CSRF token from request.META["CSRF_COOKIE"]

                      Otherwise, the user had to use HTTP POST to gain access to functionality. Since there wasn't any CSRF information yet, I decided to unprotect that view using @csrf_exempt.

                      This worked fine, as the JSON response was still giving back the right information:

                      • account details
                      • session ID
                      • CSRF token from request.META["CSRF_COOKIE"]

                      The strange thing was that the CSRF token was being regenerated on each request, making any AJAX type calls fail the CSRF test every time even with "X-CSRFToken" set.

                      After a little investigation by delving into the Django source files, I realised there were 2 reasons why this was happening.

                      Firstly, the view to check/login was using @csrf_exempt. The cookie won't be set if the view is exempt. Removing the exemption case is fine, as the app will always perform an initial GET fetch request to check existing login data.

                      Strangely, this was still not enough to generate the "csrftoken" cookie.

                      A little more investigation by browsing made me realise something, it's generating when I view regular pages but not my special JSON view.

                      The view isn't terribly complicated either. Can you spot the problem causing the CSRF token to not be set?

                      c = {
                      'success': u.is_authenticated(),
                      'username': u.username if u.is_authenticated() else '',
                      'csrf_token': request.META["CSRF_COOKIE"],
                      }

                      return HttpResponse(JSONEncoder().encode(c))

                      No? Don't worry, I didn't either.

                      The cookie won't be set unless the content/mime type of the page is a known HTML type (such as "text/html" or "application/xhtml+xml").

                      So apart from changing the mime-type, what can we do?

                      Easy. Just add in this line just before returning HttpResponse.

                      # This triggers CsrfViewMiddleware to make it set the CSRF token upon first request.
                      request.META["CSRF_COOKIE_USED"] = True

                      That's all it needed. That sneaky little bastard wasn't in the docs when I looked at it, but it should solve this fairly niche issue.

                      81366568
                      Now, off to solve more important problems!

                      django: taggit error "FieldError: Cannot resolve keyword 'tagged_items' into field."

                      Even with an extremely basic model, the django-taggit submodule was not working. I'm not even sure if this was tested before merging.

                      class TwigTest(models.Model):
                      taggit = TaggableManager()

                      TwigTest.objects.filter(taggit__name = 'dell')

                      FieldError: Cannot resolve keyword 'tagged_items' into field. Choices are: id, taggit, twigtest_tagged_items

                      Took a look around a few links gave no certain solutions, so I managed to fix it by commenting out TaggableManager.extra_filters() in manage.py.

                      Take it with a grain of salt, because I have no idea what I may have broken by doing this.

                      229196-254280708008944-785296244-n
                      These instructions will work too!

                      Sources

                      Django: Get name of SQL table from Model

                      It's rare, but sometimes you have to delve deep into the deep edges of Django's ORM to do raw SQL.

                      Something that would definitely make your life easier is knowing how to get the table name from a models.Model object.

                      You can fetch it by grabbing YourModel._meta.db_table.

                      ibvZA3Ux1aA3Nu
                      Grab that Model, Gangnam style!

                      Django: Quick snippet to get user from session ID

                      This is pretty much a drop-and-go snippet.

                      def get_user_from_session(session_key):
                      from django.contrib.sessions.models import Session
                      from django.contrib.auth.models import User

                      session = Session.objects.get(session_key = session_key)
                      uid = session.get_decoded().get('_auth_user_id')
                      return User.objects.get(pk = uid)

                      user = get_user_from_session('6117e4e8882448931b3de1f2440243ff')
                      print user.username, user.get_full_name(), user.email

                      That's it!

                      Lu6DW
                      Now, back to being productive...

                      Source

                      Django: Form.clean() is being called even though required fields are empty

                      So I decided to randomly check some code by submitting an empty form. To my surprise, Form.clean() was being called even though the required fields weren't valid. I was certain it was a core Django bug.

                      69E6e
                      Hello Twig, I'd like to play a game ...

                      It took a bit of thinking but I finally understand now. It's meant to work like that.

                      Wait, hear me out!

                      Most people would do this:

                      def clean(self):
                      data = self.cleaned_data
                      country = data['country']

                      Assuming that the required field "country" is set, but that's not true.

                      The framework is so lenient that it allows you to raise custom error messages even if the required fields aren't filled in, just in case you wanted it to.

                      That's ok, Uncle Twig has a fix for you and it's only 2 lines long.

                      To remedy this, make sure you do this on every custom Form.clean() function!

                      def clean(self):
                      if self.errors:
                      return self.cleaned_data

                      data = self.cleaned_data
                      # ...

                      That's it! At the start, if there's ANY errors at all, simply don't bother. No more invalid key exceptions.

                      Django: Mixing managed transactions, raw SQL with cursors and COMMIT statements

                      This is definitely an issue that can slip under the radar without any explicit warnings. To get an idea of the issue at hand, imagine this scenario.

                      from django.db import transaction, models, connection

                      @transaction.commit_on_success
                      def safe_view(request):
                      # Object 1
                      x, is_new = SomeModel.objects.get_or_create(arg1 = '1', arg2 = '2')
                      x.some_custom_sql()

                      # Object 2
                      y = ImageModel(associated_object = x)
                      y.caption = "This should be created if the whole request is valid"
                      y.save()

                      # Abort
                      raise Exception("STOP! Hammertime.")


                      class SomeModel(models.Model):
                      title = models.TextField()

                      def some_custom_sql(self):
                      sql = "UPDATE %s SET fti = to_tsvector('%s') WHERE %s = %s" % (self._meta.db_table, self.title, self._meta.pk.column, self.pk)

                      cursor = connection.cursor()
                      cursor.execute(sql)
                      cursor.execute("COMMIT;")

                      Because the view safe_view() is wrapped in a transaction, you would presume that any code in there will be rolled back if an exception were to occur, so an instance of SomeModel and ImageModel would not exist.

                      However, due to an explicit call to "COMMIT" in some_custom_sql(), the transaction is finalised by the time it returns back to safe_view().

                      Once the code continues, it will create an instance of ImageModel and save it to the database.

                      Even though the view is wrapped in commit_on_success() and an exception is raised, the changes to the database are already permenant.

                      EBy the time you hit the exception, it would have:

                      • Created an instance of SomeModel
                      • Updated the row for the FTI entry
                      • Created an instance of ImageModel

                      This is not ideal!

                      To prevent this, refrain from using "COMMIT" in raw SQL unless you know exactly what you're doing!

                      Instead, replace cursor.execute("COMMIT;") with transaction.commit_unless_managed(). It'll automatically detect if the database is locked in a transaction before committing the transaction.

                      This problem looks incredibly easy to spot when it's next to each other on the same page, but remember that the "COMMIT" lines could be anywhere in your project. Even in 3rd party modules you've downloaded.

                      They will silently break your transactions and leave little evidence of permanent database changes until somebody notices!

                      Be vigilant and you will spot out all the problems in no time!

                      7151214449524

                      Django: How to save InMemoryUploadedFile (a temporary uploaded file) to disk?

                      When a user uploads a file using forms.FileField, you usually want to save it in some way.

                      To stash the file away on disk, just use the following snippet.

                      from django.core.files.storage import default_storage
                      from django.core.files.base import ContentFile

                      file = request.FILES['your_file_fieldname']
                      path = default_storage.save('heart_of_the_swarm.txt', ContentFile(file.read()))

                      There you have it!

                      i4IaIek6mz5sW

                      Now back to more important things, like eating cheerleaders.

                      Source

                      Django: Programmatically saving image from URL to FileField or ImageField

                      This was a bit of a tricky one to figure out. The reason being we store our files not in the file system but using MogileFS, which required a custom Storage wrapper.

                      So our model looks a little something like this.

                      class Product(models.Model):
                      # other fields
                      image = models.FileField(storage = MogileFSStorage(), upload_to = 'product_images')

                      And that's completely fine if we're using the admin to manually upload image files.

                      However, if we wanted to programmatically save an image using code... Where do we begin?

                      I'm glad to say that it's actually easier than expected.

                      from django.core.files import File
                      from django.core.files.temp import NamedTemporaryFile

                      product = Product()
                      # set all your variables here
                      product.save()

                      # Save image
                      image_url = 'http://whatever.com/image.jpg'
                      img_temp = NamedTemporaryFile(delete = True)
                      img_temp.write(urlopen(image_url).read())
                      img_temp.flush()

                      product.image.save("image_%s" % product.pk, File(img_temp))
                      product.save()

                      And that's it! There's no need to worry about storage stuff because that's handled at the model declaration level.

                      This code should work fine for both FileField and ImageField.

                      Of course, how to access the file is completely up to you and out of the scope of this post.

                      5KbWT 
                      It'll all makes sense once someone puts it into perspective for you!

                      Sources

                      Django: Using the ORM to (automatically) create custom database field types

                      When creating the full text index search layer in Django, I had a little issue when new models were created.

                      Luckily, it's quite easy to create custom database types in your models.

                      class TSearchVectorField(models.fields.TextField):
                      """
                      This lets Django know how to create a field with the "tsvector" type.
                      """
                      description = "Tsearch Vector"

                      def db_type(self, connection):
                      return 'tsvector'


                      class TSearch(models.Model):
                      fti = TSearchVectorField()

                      flock-of-seagals
                      Elegant, like a flock of Seagals.

                      Source

                      Django: Sending out a HTML email... with attachments

                      The usual django mail.send_mail() is sufficient for most cases if you're sending by plain text. It does leave a little to be desired, such as changing it to a HTML format or something.

                      However, if you're after an email with attachments, you'll have to get a little busy.

                      from django.core.mail.message import EmailMessage

                      email = EmailMessage()
                      email.subject = "New shirt submitted"
                      email.body = message
                      email.from_email = "ThatAwesomeShirt! <noreply@thatawesomeshirt.com>"
                      email.to = [ "whoever@whatever.com", ]

                      # Save the file to a temporary file because that's how email attachments work
                      output = StringIO.StringIO()
                      image.save(output, format = "PNG")
                      email.attach(filename = "test.png", mimetype = "image/png", content = output.getvalue())
                      output.close()

                      email.send()

                      Optionally, you can add email.content_subtype = "html" to set the email to be a HTML email.

                      tumblr_lyegh0j09a1qdlh1io1_400

                      No more trying to force people into reading plain text emails!

                      Source

                       
                      Copyright © Twig's Tech Tips
                      Theme by BloggerThemes & TopWPThemes Sponsored by iBlogtoBlog