diagramming software wireframing/prototyping software

Tuesday, January 12, 2010

Native Django on App Engine

Update: Our main branch has changed to django-nonrel which is based on a much simpler strategy that became possible with Django's new multi-db support. All links in this article have been updated. The old branch is still available.

About a few months ago we started to port Django to support non-relational databases and to implement an App Engine database backend for this port. So far we ended up supporting basic Django functionality on App Engine. This post is intended to get you started using our port and to let you know what you can do and want you can't do with it at the moment. So let's start!

Installation

In order to use our port on App Engine you have to clone a few repositories first. These are the non-relational django port, djangoappengine and the django-testapp. So what are all these repos for?
First in order to let you start a new Django project as fast as possible we created the django-testapp repo. It basically consists of a modified manage.py file to support the App Engine development server, and all corresponding settings in order to specify App Engine as the backend for Django. So the first step is to clone the django-testapp repo.
The djangoappengine repo contains all App Engine related backends for the non-relational port e.g. the email backend and the query backend for Django. So as the next step clone the djangoappengine package into the "common-apps" folder of the testapp repo.
The non-relational Django repo is our modified Django port. It contains changes in Django itself to support non-relational databases. Clone it and link the repository's "django" folder into the "common-apps" folder.
Your folder structure should now look similar to this:

.../non-relational_django_port
.../django-testapp
.../django-testapp/common-apps/djangoappengine
.../django-testapp/common-apps/link_to_django_folder

Now you should be able to create a new app in the testapp folder and use native Django models on App Engine. Try it!

Supported and unsupported features for the App Engine backend

Field types
First we added support for most Django field types but not all. Here is the list of Django field types which currently are not supported (but there are chances that a few of them will get ported in the near future):
  • FileField
  • FilePathField
  • OneToOneField
  • ManyToManyField
  • DecimalField
  • ImageField
All other ones are supported fully. In addition we support all Django field options except for options which can't be used on App Engine. These are unique, unique_for_date, unique_for_month and unique_for_year.

Additionally there exist App Engine properties which do not have been ported to Django field types yet, so you cannot use them. Maybe the most mentionable are ListPorperties. So any of your projects using ListPorperties can't be easily converted to a Django project.
Here is the list of not explicitly supported App Engine properties:
  • ByteStringProperty
  • ListProperty and StringListProperty
  • ReferenceProperty and SelfReferenceProperty
  • blobstore.BlobReferenceProperty
  • UserProperty
  • BlobProperty
  • GeoPtProperty
  • RatingProperty
Nevertheless you can use many-to-one relationships, just use a ForeignKey. But internally the ForeignKey does not store an App Engine db.Key value. The lack of a GAEKeyField in django storing db.Keys makes it impossible to create entity groups with the current App Engine backend.

The following App Engine properties can be emulated by using a CharField in Django:
  • CategoryProperty
  • LinkProperty
  • EmailProperty
  • IMProperty
  • PhoneNumberProperty
  • PostalAddressProperty
QuerySet methods
We do not support any filters that are not supported by App Engine itself. This means that you can't use all field lookup types from Django. Nevertheless you can use
  • __lt less than
  • __lte less than or equal to
  • __exact equal to
  • __gt greater than
  • __gte greater than or equal to
  • is_null
You can even use the exclude function in combination with the listed filters above. In addition you can use an IN-filter on primary keys and Q-objects to formulate more complex queries. Slicing is possible too.
But in all cases you have to keep App Engine restrictions in mind. While you can perform a filter like
Model.objects.exclude(Q(integer_field__lt=5) | Q(integer_field__gte=9))
you can't do
Model.objects.filter(Q(integer_field__lt=5) | Q(integer_field__gte=9))
The reason for this is that the first filter can be translated into an AND-filter without using a logical OR but the second one cannot.
Inequality filters are not supported for now. But many additional non return sets like count(), reverse(), ... do work.
Maybe you are wondering how to set a keyname for your model instance in Django. Just set the primary key of your entity and it will be used as the keyname.
model_instance.pk = u'keyname'
model_instance.save()
but remember to set the primary_key field option of the CharField to True.

Many Django features are not supported at the moment. A few of them are joins, many-to-many relations, multi table inheritance and aggregates. On the other hand there are App Engine features not supported by the Django port mainly because of the lack of corresponding features in Django. Two of them are entity groups and batch put.
But we plan to add some features for both sides while working on our own project. As the last point we do not support any Django transaction for the App Engine backend but we implemented a custom transaction decorator @commit_locked.

Differences in Django
itself

While changing the Django code itself we had to hack some parts in. Django should behave as described in the Django documentation in all situations except for one: when deleting a model instance the related objects will not be deleted. This had to be done because such a deletion process can take too much time.
In order to support deletion of related objects for non-relational Django in a clean manner Django should be modified to allow the backend to handle such a deletion process. For App Engine it can be done using background tasks.

Advantages using native Django

So maybe you are wondering why to use our current Django port instead of other projects providing you Django features on App Engine like our last project app-engine-patch. There are a few reasons for that.
First of all and maybe the most important advantage of using native Django is that you will avoid the vendor lock-in problem. I am not saying that you should not use App Engine but you can find yourself in a situation where App Engine is not sufficient enough for your needs. Using native Django code will allow you to switch to another hosting/cloud provider with having minimal costs related to porting the existent code to the new database. Second using native Django will result in almost full reusability of existing Django apps and makes non-relational Django apps fully portable to any platform (including SQL). Maybe these apps have to be modified a little bit but changing these apps is a lot easier than to port an App Engine project to Django.
In addition the model layer gives you many new features like select_related() which are not supported by other Django projects on App Engine. Another example is while using App Engine models we found it annoying to pass required fields to the model constructor. In Django you do not have to set required field types when creating a model instance. Only when saving the entity such fields have to be valid. So these are only a few reasons.

If you want to see more source code about all these things mentioned in this post take a look at our unit tests in djangoappengine. There you can find model definitions and queries such that you can get a better idea of how to use the App Engine backend for the non-relational Django port.

Help


A few days ago we cleaned up the non-relational django port itself. Now it should be easy to write a backend without modifying the django port. If you want to help improving our non-relational Django port, you can check out the wiki and the task list. We are happy about any other help too. You can help by testing the port, reaching out to other communities like MongoDB and SimpleDB, building a greater community around this project and also attracting developers who can contribute code.

While this post is mostly about how to use our port on App Engine it's possible to implement support for other non-relational databases using our non-relational Django port too. Again if you are planning to help, take a look at the djangoappengine repo to get an idea of how to implement your own non-relational database backend for Django and join the django non-relational group.

In order to help you write non-relational Django code we plan to write posts about how to write SQL-independent Django code and how to port existing app-engine-patch/webapp/Django apps to the new non-relational port.

So my first post got a little bit longer than i thought but i hope you enjoyed to read it. We'd love to hear your comments.

4 comments:

  1. Since there are ways to implement one-to-many and many-to-many relationships in app engine, I suppose the biggest problem is the file fields (but I'm not a Django or App Engine expert, so I may be saying something really stupid, here).
    If so, you could try the new App Engine Blobstore API to implement the FileField in models.

    ReplyDelete
  2. @Raul: Blobstore would be great, but I'm not 100% sure if this is possible. Probably it is. For now, Kyle Finley has implemented a storage backend with a BlobProperty (1MB size limit):
    http://bitbucket.org/scotch/djangoappengine/src/tip/backends/appengine.py

    ReplyDelete
  3. I have recently been trying to take my site from App Engine Patch to this project. I have gotten it running, however for some reason I cannot link to any of my static files even though the paths appear to be correct. Im fairly new to Django and App Engine so I am probably overlooking something stupid here, but any help would be appreciated.

    I have tried altering the main Django url file as well, but this did not work. Still, I thought the app.yaml was mainly responsible for this sort of stuff.


    Simplified project Structure:

    __init__.py
    app.yaml
    index.yaml
    apps/...
    common-apps/...
    media/
    2/...
    favicon.ico
    urls.py
    settings.py


    Simplified app.yaml:

    application: testApp
    version: 1
    runtime: python
    api_version: 1

    default_expiration: '365d'

    handlers:
    - url: /favicon.ico
    static_files: media/favicon.ico
    upload: media/favicon.ico

    - url: /media
    static_dir: media
    secure: optional

    - url: /remote_api
    script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
    login: admin

    - url: /.*
    script: common-apps/djangoappengine/main.py


    Example paths found on page:

    http://localhost:8080/media/2/global/960/reset.css
    http://localhost:8080/favicon.ico/

    All media files return a 404, although I can get to the favicon by going to http://localhost:8080/favicon.ico without the trailing slash. The root media folder, http://localhost:8080/media, gives me a permission denied warning but everything else just says page not found. The parts added parts of the app.yaml were directly taken from the other project as is the directory structure, so I know this has worked in the past.

    ReplyDelete
  4. How does your media folder looks like? Are you still using the media generator? Because if not it can be that you do not have a global folder generated anymore.

    Bye,
    Thomas

    ReplyDelete