concurrency – James Lin

Make sure you are closing the DB connections after accessing Django ORM in your threads

I was writing a service which import various feeds and then post to another RESTful API, simple enough as below:

from django.db import connections
class Receiver(PolymorphicModel):
    ...

class Database(Receiver):
    name = models.CharField(max_length=255)
    def get_data_source(self):
        return connections[name]

from django.db import connections

class Receiver(PolymorphicModel):

...

class Database(Receiver):

name = models.CharField(max_length=255)

def get_data_source(self):

return connections[name]

Then the feed model:

class Feed(models.Model):
    receiver = models.ForeignKey('Receiver')

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

class Feed(models.Model):

receiver = models.ForeignKey('Receiver')

def get_data_source(self):

return self.receiver.get_real_instance().get_data_source()

Then run this feed in trivial code:

import threading

function transform_listing(feed):
    cursor = feed.get_data_source().cursor()
    ....

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):
    th = threading.Thread(transform_listing, [feed])
    th.start()

import threading

function transform_listing(feed):

cursor = feed.get_data_source().cursor()

....

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):

th = threading.Thread(transform_listing, [feed])

th.start()

And you will quickly notice the threads complaining

Too many connections

1	Too many connections

What it’s happening is Django will create a new connection per thread… Read More »

Share this: