Wednesday, June 19, 2013

Django Custom Chainable QuerySets

How to reuse custom defined 'models.Manager' QuerySets

Django recommends to define custom query sets on your sublcassed models.Manager, there is one problem however that these are not chainable, that is you can only call your custom queryset once - assuming I have defined an '.active()' queryset the following will not work:

>> PaymentTerm.objects.active().active()
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'PaymentTermManager' object has no attribute 'active'
>>>

Current solutions on the web involve creating a subclassed 'models.Manager' and a custom 'QuerySet' which do work but are to code verbose for my linking. I believe the below approach is simpler, cleaner and more succinct.

class OfferManager(models.Manager):
 
 ...
 
 STATUS_CHOICES = (
  (STATUS_DISABLED, "Disabled"),
  (STATUS_ENABLED, "Enabled"),
  (STATUS_NEGOTIATED, "Negotiated"),
  (STATUS_ARCHIVED, "Archived"),
 )
 QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED}
 QUERYSET_ACTIVE_KWARGS = {'status': STATUS_ENABLED}
  def get_query_set(self):
   """ Assing a modified QuerySet to include our chainable QuerySets """
   class OfferManagerQuerySet(QuerySet): # clone QuerySet
                pass
  # explictly copy our QuerySet methods
  OfferManagerQuerySet.public = self.public
  OfferManagerQuerySet.active = self.active

  ...
 
  return QuerySetDynamicallyCreated(self.model, using=self._db)
 def public(self):
  """ Returns all entries accessible through front end site"""
  return self.all().filter(**OfferManager.QUERYSET_PUBLIC_KWARGS)
 public.chainable = True
 def active(self):
  """ returns offers that are open to negotiation """
  return self.public().filter(**OfferManager.QUERYSET_ACTIVE_KWARGS)

 ...

Django patch for model.Manager chainable QuerySets

Please take a look at Django Ticket #20625 with a github Django branch for new proposed method for defining chainable filters based on this idea which works through a 'chainable == True' attribute as shown below:

class OfferManager(models.Manager):

    ...
    
    def public(self):
        """ Returns all entries accessible through front end site"""
        return self.all().filter(...)
    public.chainable = True     # instructs to dynamically tranplat this method onto
                                # returned QuerySet as .public(...) 
                                # effectively providing chainable custom QuerySets

    def active(self):
        """ Returns offers that are open to negotiation """
        return self.public().filter(**OfferManager.QUERYSET_ACTIVE_KWARGS)
                                    # an example of how to reffer to OfferManager
                                    # constants as 'self' context changes
    active.chainable = True
    ...

As always feel free to comment or follow me on twitter @danielsokolowski

Thursday, June 6, 2013

CommandError: Unable to serialize database: Error encountered checking Geometry returned from GEOS C function "GEOSWKBReader_read_r".

ERROR django.contrib.gis libgeos.py@80: GEOS_ERROR: ParseException: Unknown WKB type 0

These cryptic messages are caused by using the vanilla Django `model.Manager` with your geo enabled models. This caused me a frew hours of grief trying to figure it out, thinking it's the import data, or an issue with the third party app, etc. but the fix is to just extend your custom model manager from the `gis.models.GeoManager` which is super easy to forget:


from django.contrib.gis.db import models

class MyCustomManager(models.GeoManager):
    """
    Additional methods / constants to Base's objects manager - using a GeoManager is fine even for plain models:
    
    ``BaseManager.objects.public()`` - all instances that are asccessible through front end
    """
    # Model (db table) wide constants - we put these and not in model definition to avoid circular imports.
    # one can access these constants through .objects.STATUS_DISABLED or ImageManager.STATUS_DISABLED
    STATUS_DISABLED = 0
    STATUS_ENABLED = 100
    STATUS_ARCHIVED = 500
    STATUS_CHOICES = (
        (STATUS_DISABLED, "Disabled"),
        (STATUS_ENABLED, "Enabled"),
        (STATUS_ARCHIVED, "Archived"),
    )
    # We keep status field and custom queries naming a little different as it is not one-to-one mapping in all situations
    QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED} # Because you can't yet chain custom manager filters ex. 
                                                             #'public().open()' we provide access this way and we can use directly in 'ForeignKey.limit_choices_to'
                                                             # workaround - http://stackoverflow.com/questions/2163151/custom-queryset-and-manager-without-breaking-dry  
    def public(self):
        """ Returns all entries someway accessible through front end site"""
        return self.filter(**self.QUERYSET_PUBLIC_KWARGS)      # def active(self):

If this post hepls you do say thank you by commeting or following me on twitter. .

Wednesday, June 5, 2013

Django import fixtures IntegrityError: Problem installing fixtures:

IntegrityError: Problem installing fixtures: The row in table 'geonames_alternatename' with primary key '1830' has an invalid foreign key: geonames_alternatename.locality_id contains a value '109436' that does not have a corresponding value in geonames_locality.geonameid.

In a situation where a third party app data export works just fine but the fixutre fails due to any IntegrityError is most likley cause by an overiden `objects` manager that returns only a subset. The solution is to explicitly tell django to export using the default model managers using the `--all` argument for example like so:

dumpdata geonames --indent=4 --all --settings=settings_development > geonames_ext/fixtures/initial_data.json

This is clearly a situation where explicit is better then implicit, that is when defining custom model managers do not touch the `get_query_set` method rather instead add an explict method; for post completeness below is my Eclipse PyDev template I utlize when defining new models.

from django.db import models
#from django.contrib.db import models
from django.core.urlresolvers import reverse
from django.core.urlresolvers import NoReverseMatch
from djanfrom django.utils.encoding import iri_to_uri
from django.utils.http import urlquote
from django.core.exceptions import ValidationError
from django.core.validators import MinValueValidator, RegexValidator

class ${model_name}Manager(models.Manager):
    """
    Additional methods / constants to ${model_name}'s objects manager:
    
    ``${model_name}Manager.objects.public()`` - all instances that are asccessible through front end
    """
 ### Model (db table) wide constants - we put these and not in model definition to avoid circular imports.
    ### One can access these constants through .objects.STATUS_DISABLED or ImageManager.STATUS_DISABLED
    STATUS_DISABLED = 0
    STATUS_ENABLED = 100
    STATUS_ARCHIVED = 500
    STATUS_CHOICES = (
        (STATUS_DISABLED, "Disabled"),
        (STATUS_ENABLED, "Enabled"),
        (STATUS_ARCHIVED, "Archived"),
    )
 ### We keep status field and custom queries naming a little different as it is not one-to-one mapping in all situations
 ### Note: workarrund - http://stackoverflow.com/questions/2163151/custom-queryset-and-manager-without-breaking-dry
    QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED} # because you can't yet chain custom manager filters ex.: 
                                                             # 'public().open()' we provide access the QUERYSET_KWARGS
    def public(self):
        """ Returns all entries someway accessible through front end site"""
        return self.filter(**self.QUERYSET_PUBLIC_KWARGS)   # def active(self):
 # """ Returns all active entries, for example entries that can be selected in forms, searched on, etc""
 #def active(self)
 # return self.filter(status=self.STATUS_ENABLED) 

class ${model_name}(models.Model):
    """
    Main entity representing ${model_name} object
    """
    ### model options - "anything that's not a field"
    class Meta:
        ordering = ['order', 'name']
        get_latest_by = 'order'
        #order_with_respect_to = 
        #permissions = [["can_deliver_pizzas", "Can deliver pizzas"]]
        #unique_together = [["driver", "restaurant"]]
        #verbose_name = "pizza"
        #verbose_name_plural = "stories"
    
    ### Python class methods
    def __unicode__(self):
  """ Retruns a unicode representation for the instance of this model """
  if settings.DEBUG:
   return u'PK%s: %s - %s' % (self.pk, self.name, self.FOO)
  return u'%s - %s' % (self.name,  self.FOO)
    
    ### Django established method
    #def get_absolute_url(self):
    #    """ Returns the relative url mapping for the instance of this model if it exists or None otherwise"""
    #    return iri_to_uri(reverse('Company', kwargs={'pk': self.pk, 'slug': urlquote(slugify(self.__uniocode__()))}))
    #    return reverse ('${model_name}', kwargs={'slug': self.slug})
    def save(self, *args, **kwargs):
        if self.project: # enforce that company matches project's company
            self.company = self.project.company
        super(Login, self).save(*args, **kwargs) # Call the "real" save() method.        

    def clean(self):
        """ Check if icon is needed or not """
        if self.project and (self.company != self.project.company):
            raise ValidationError('Project must belong to the company specified.')
        
    ### extra model functions
    def get_next_order():
        """ Returns the next available integer for the order field """
        try:
            return ${model_name}.objects.latest('order').last + 1
        except: 
            return 1
            
    def get_upload_path(instance, filename):
        """ returns a dynamic path for filefields/imagefieds """
        return '%s/%s/%s' % (instance._meta.app_label, instance._meta.module_name, filename)
    
    def get_geopoint_srid4326(self):
        """ 
        Returns a geopoint in spatial projection 4326 which is lat and lng coordinate system. 
        
        We are doing this so we can utilise lat lng in the template as sqlite dosen't support
        all distance calculations in 4326 so we used srid 900913 which is not lat lng based.
        """
        pnt = GEOSGeometry(self.geopoint.wkt) # we have to make copy of point as transform 
        pnt.set_srid(self.geopoint.get_srid())                                      # works in place and dosen't return a new point
        return pnt.transform(4326)
    ### custom managers
    objects = ${model_name}Manager()
    #objects = models.GeoManager() # geodjango objects manager
    
    ### model DB fields
    # add  blank=False, default='' if you don't want to display a 'blank choice' when rendering modelchoicefield
    status = models.IntegerField(blank=False, default=None, choices=${model_name}Manager.STATUS_CHOICES, default=${model_name}Manager.STATUS_ENABLED)
    order = models.PositiveIntegerField(help_text='Display order modifier, leave default if unsure', unique=False, default=get_next_order, validators=[MinValueValidator(1)])
    name = models.CharField(help_text='Name', max_length=20)
    # rather than database slug use PK and a dynamic slug from name, links will still work using ID lookup
 #slug = models.SlugField(help_text='A short label used in URL generation containing only letters, numbers, underscores or hyphens; hyphens being preferred for SEO optimizations',
 #           unique=True, null=True, validators=[RegexValidator(r'.+')])
    description = models.TextField(help_text='Optional HTML allowed description', blank=True)
    # content = models.TextField(help_text='Optional content', blank=True) # this is not needed keep it simple
    phone = models.CharField(help_text='Main phone number ex. 613-333-4444.', max_length=12, 
                validators=RegexValidator(regex='\d\d\d-\d\d\d-\d\d\d\d', 
                           message="Please enter phone number in following format: 613-333-4444", 
                           code="Unrecognized Phone"))
 services = models.ManyToManyField(LawyerService, limit_choices_to= {'status', LawyerService.objects.live()}, 
                    help_text='Available Family law services.')
    ### GeoDjango-specific fields
    #geopoint = models.PointField()

As always, if this has hepled you say thank you by commeting or following me on twitter.