Make sure you are closing the DB connections after accessing Django ORM in your threads

By | April 22, 2016

I was writing a service which import various feeds and then post to another RESTful API, simple enough as below:

from django.db import connections
class Receiver(PolymorphicModel):

class Database(Receiver):
    name = models.CharField(max_length=255)
    def get_data_source(self):
        return connections[name]

Then the feed model:

class Feed(models.Model):
    receiver = models.ForeignKey('Receiver')

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

Then run this feed in trivial code:

import threading

function transform_listing(feed):
    cursor = feed.get_data_source().cursor()

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):
    th = threading.Thread(transform_listing, [feed])

And you will quickly notice the threads complaining

Too many connections

What it’s happening is Django will create a new connection per thread when you access the ORM, and connections are left open even the threads are terminated, in this case the culprit is

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

So in this case, simply refactor the code to close the connection before return:

from django.db import connection

    def get_data_source(self):
        result = self.receiver.get_real_instance().get_data_source()
        return result

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.