Make sure you are closing the DB connections after accessing Django ORM in your threads

I was writing a service which import various feeds and then post to another RESTful API, simple enough as below:

from django.db import connections
class Receiver(PolymorphicModel):
    ...

class Database(Receiver):
    name = models.CharField(max_length=255)
    def get_data_source(self):
        return connections[name]

Then the feed model:

class Feed(models.Model):
    receiver = models.ForeignKey('Receiver')

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

Then run this feed in trivial code:

import threading

function transform_listing(feed):
    cursor = feed.get_data_source().cursor()
    ....

feed = Feed.objects.create(receiver=Database.objects.create(name='legacy'))

for i in range(100):
    th = threading.Thread(transform_listing, [feed])
    th.start()

And you will quickly notice the threads complaining

Too many connections

What it’s happening is Django will create a new connection per thread when you access the ORM, and connections are left open even the threads are terminated, in this case the culprit is

    def get_data_source(self):
        return self.receiver.get_real_instance().get_data_source()

So in this case, simply refactor the code to close the connection before return:

from django.db import connection
...

    def get_data_source(self):
        result = self.receiver.get_real_instance().get_data_source()
        connection.close()
        return result