GenericForeignKeys with fewer queries

When working with generic relations in Django you have to be quite careful not to end up with n+1 queries for a simple fetch of n elements. The reason for this is that internally a generic relation is not really a true foreign key (naturally) but just an id combined with a foreign key to a content-type. But there are some ways around this problem. Among them a quite simple one: Doing the actual content-loading by yourself.

This is a really great tip! Why don't you add the ready-made manager to djangosnippets.com? I Guess some people'd love it.

Cheers.

PS. Again: Great work.

Martin Geber on Aug. 15, 2008 at 10:07 +0200

I'm still guessing on what my password there is ;-)

zerok on Aug. 15, 2008 at 17:22 +0200

http://www.djangosnippets.org/snippets/984/

zerok on Aug. 15, 2008 at 22:27 +0200

Awesome. Great Manager! adding it to delicious

Martin Geber on Aug. 19, 2008 at 11:06 +0200

Martin,

I have been playing with your solution, but it seems that if you use ContentType object as a key in the .setdefault function, each object is unique and therefore resulting model_map has the same number of elements as there are items (=n). The following 'for' cycle is then repeated 'n' times, not 'm'. At least that is how it works for me in Django 0.96.1.

The solution could be to use item.content_type.name as a key for model_map, and translate it later back to ContentType object.

Also, it seems that this procedure will change the ordering of the list, so you basically can not control it. So far I do not know how to solve that.

The last two things, just a corrections, I think you can not use filter() and all() together in ct.model_class().objects.select_related() .filter(id__in=items_.keys()).all(), and in the snippet you posted you probably want to return item_map.items(), not the initial 'qs'.

I might be wrong, so please let me know what you think about that.

Anyway your code gave me some good ideas, if we will be able to fix/clarify above stuff, it will be perfect.

hab on Aug. 24, 2008 at 11:08 +0200

sorry, not Martin, but zerok, of course:)

hab on Aug. 24, 2008 at 11:30 +0200

hab,

  • len(model_map.keys()) == m at least in trunk. Never tried it with 0.96.x since I don't use it anywhere.
  • No, it doesn't change the ordering since the items in the resultset will be changed through the ordering thanks to pass-by-reference in Python
  • Regarding filter+all: At least in post-qsrf all() doesn't really do all that much, so I basically put it everywhere ;-) I might be wrong there, but so far it hasn't bitten me :-)
  • qs vs. items: This is exactly what I meant above: If qs is returned, you still operate on the original order of things ;-)

zerok on Aug. 24, 2008 at 12:51 +0200

Thanks for your reply, I realized I was wrong about 'ordering' and 'qs'. Regarding using 'all()' and 'filter()' together - that returns error in Django 0.96.1 (I do no use the latest SVN version, maybe I should:). The same for 'object as key' issue, again it is probably fixed in newer versions. To sum it up, I have put together a code that I believe works as meant in the release that I use, maybe it could help also some other guys. BTW, it makes me wandering, if it is necessary to use select_related('content_type'), as it could be done using content_type_id as a key and then get the correct ContentType object using that...

items = Item.objects.select_related('content_type').all()
model_map = {}
item_map = {}

for item in items:
  model_map.setdefault(item.content_type.name, {})[item.object_id] = item.id
  item_map[item.id] = item

    for ct_name, items_ in model_map.items():
      for o in ContentType.objects.get(name=ct_name).model_class().objects.select_related().filter(id__in=items_.keys()):
           node_map[nodes_[o.id]].content_object = o

## We do not have to do this, because if
## we modify item in item_map, the change is reflected in 'items' also (OOP:), so I take back my comment about 'ordering' and also about 'return qs'
#     for item in items:
#         item.content_object = item_map[item.id].content_object

return items

Thanks again, keep up good work.

hab on Aug. 24, 2008 at 13:05 +0200

fewer.

Anonymous on Sept. 24, 2008 at 09:24 +0200

Comment on this

You can use Markdown here.

Please answer the question in this field. It should help keeping spambots out ;-)