Category Archives: Django

Make sure you are closing the DB connections after accessing Django ORM in your threads

I was writing a service which import various feeds and then post to another RESTful API, simple enough as below:

from django.db import connections
class Receiver(PolymorphicModel):
    ...

class Database(Receiver):
    name = models.CharField(max_length=255)
    def get_data_source(self):
        return connections[name]

from django.db import connections

class Receiver(PolymorphicModel):

...

class Database(Receiver):

name = models.CharField(max_length=255)

def get_data_source(self):

return connections[name]

Then the feed model:

class Feed(models.Model):
    receiver = models.ForeignKey('Receiver')

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

class Feed(models.Model):

receiver = models.ForeignKey('Receiver')

def get_data_source(self):

return self.receiver.get_real_instance().get_data_source()

Then run this feed in trivial code:

import threading

function transform_listing(feed):
    cursor = feed.get_data_source().cursor()
    ....

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):
    th = threading.Thread(transform_listing, [feed])
    th.start()

import threading

function transform_listing(feed):

cursor = feed.get_data_source().cursor()

....

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):

th = threading.Thread(transform_listing, [feed])

th.start()

And you will quickly notice the threads complaining

Too many connections

1	Too many connections

What it’s happening is Django will create a new connection per thread… Read More »

Laravel for Django developers, part 2

Global functions I am seeing a lot of framework specific global functions but don’t know where they come from, such as “view()”, “redirect()” etc etc. ORM Again, as mentioned in part 1, since we are creating migration files first, then we won’t be able to define FK relationships directly point to another model. According to… Read More »

Laravel for Django developers, part 1

Project Structure After setting up the quickstart project, I feel the project structure is quite different to Django. In Django a lot of code are organised in “app” domain, but it seems to me that multi-app is not something out-of-the-box in Laravel: https://laracasts.com/discuss/channels/general-discussion/laravel-5-multi-app-or-modular-applications Instead, it seems the apps in Django is more similar to the… Read More »

Laravel for Django Developers, prelude.

It appeared to me that “Laravel” is kind of similar to Django in the way of implementation concept, so this is my obvious choice. I did a google search “Laravel for Django developers” and got pretty much nothing useful, sure, who would have move back to PHP from Python? I think this only happens in… Read More »

A simple architecture to consolidate the realestate websites in NZ

Just a business idea, while I am browsing all these real estate websites, they pretty much give the same types of information, it could be reasonable profitable to consolidate the online platform, and even cheaper option for RE companies. I am not a sales person so I don’t know how to sell it, but it’s not… Read More »

A very useful reference website if you use django class based views

A colleague of mine mentioned this site http://ccbv.co.uk/ which provides references and information about django class based views. Classy Class-Based Views. ccbv.co.uk The best way to understand Django’s class-based views is to see it in Classy CBV, so pick your version and jump in at the deep end.

Django should have OneToManyField

A bit of background here about ForeignKey (many to one): If you have Order and OrderItem models, your order FK is in OrderItem model, from OrderItem model point of view, it’s Many OrderItem to One Order, in other words ManyToOne. What this implies is you need to have Order records before OrderItem records. Now, why… Read More »

django-rest-framework: api versioning

Django Rest Framework is a very solid api framework, but it doesn’t provide out-of-box versioning mechanism, here is my attempt to implement version specific APIs. The goal is to achieve something like http://localhost:8000/api/(resource)/ http://localhost:8000/api/v1/(resource)/ plus allowing clients to specify the version in request header (X-Version), here is how we did it. Structure in side the… Read More »

Django CMS Haystack 2.0 Search Index

Currently django-cms-search doesn’t support haystack 2.0 yet, so here is my modified version to work on haystack 2.0 based on

import datetime
from haystack import indexes   
from cms.models.managers import PageManager
from cms.models.pagemodel import Page
from cms.models.pluginmodel import CMSPlugin


class PageIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=False)
    pub_date = indexes.DateTimeField(model_attr='publication_date', null=True)
    login_required = indexes.BooleanField(model_attr='login_required')
    url = indexes.CharField(model_attr='get_absolute_url')
    title = indexes.CharField(model_attr='get_title')

    def prepare(self, obj):    
        self.prepared_data = super(PageIndex, self).prepare(obj)
        plugins = CMSPlugin.objects.filter(placeholder__in=obj.placeholders.all())
        text = ''              
        for plugin in plugins: 
            instance, _ = plugin.get_plugin_instance()
            if hasattr(instance, 'search_fields'):
                text += ''.join(getattr(instance, field) for field in instance.search_fields)
        text += obj.get_meta_description() or u''
        text += obj.get_title() or u''
        text += obj.get_meta_keywords() if hasattr(obj, 'get_meta_keywords') and obj.get_meta_keywords() else u''
        self.prepared_data['text'] = text
        return self.prepared_data       

    def get_model(self):
        return Page

    def index_queryset(self, using=None):
        return Page.objects.published().filter(publisher_is_draft=False).distinct()

import datetime

from haystack import indexes

from cms.models.managers import PageManager

from cms.models.pagemodel import Page

from cms.models.pluginmodel import CMSPlugin

class PageIndex(indexes.SearchIndex, indexes.Indexable):

text = indexes.CharField(document=True, use_template=False)

pub_date = indexes.DateTimeField(model_attr='publication_date', null=True)

login_required = indexes.BooleanField(model_attr='login_required')

url = indexes.CharField(model_attr='get_absolute_url')

title = indexes.CharField(model_attr='get_title')

def prepare(self, obj):

self.prepared_data = super(PageIndex, self).prepare(obj)

plugins = CMSPlugin.objects.filter(placeholder__in=obj.placeholders.all())

text = ''

for plugin in plugins:

instance, _ = plugin.get_plugin_instance()

if hasattr(instance, 'search_fields'):

text += ''.join(getattr(instance, field) for field in instance.search_fields)

text += obj.get_meta_description() or u''

text += obj.get_title() or u''

text += obj.get_meta_keywords() if hasattr(obj, 'get_meta_keywords') and obj.get_meta_keywords() else u''

self.prepared_data['text'] = text

return self.prepared_data

def get_model(self):

return Page

def index_queryset(self, using=None):

return Page.objects.published().filter(publisher_is_draft=False).distinct()

Django admin UnicodeEncodeError

Don’t we all get confused by unicode encoding sometimes? Especially you wouldn’t expect this kind of error to happen in django admin. The problem is actually not in the admin section, the culprit is __unicode__ method in your model. when you have a unicode function like this:

def __unicode__(self):
  return "{0}".format(self.field_one)

1 2	def __unicode__(self): return "{0}".format(self.field_one)

It is actually returning a ASCII string(which… Read More »

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this: