OrangeScape Developer Community

A community for Application Developers on OrangeScape's Platform-as-a-Service

Filters & index errors in Google App Engine

    We can use the “Filter Data” functionality in the Process Design to enable end users to filter the data in their queues. There are different operators available to define your filters. When using with Google App Engine [GAE], there are certain limitations which limit us from defining these filters.

    Let us see how Google App Engine [GAE] executes queries. From the GAE docs,

Every datastore query uses an index, a table containing the results for the query in the desired order.

    So, whenever you wish to retrieve data from the datastore and issue a query, it looks for the corresponding indexes and fetches the data. For most of the simple queries, app engine defines and builds indexes by default. for the other queries, the application must provide the list of indexes to be built. If a required index is not there, and you try to execute the query which requires this index, then app engine will throw aNeedIndexError.

    Now, as you know, for each query, an appropriate index has to be there in the datastore. It is usually the filter conditions which require specific indexes.

    In our Filter Data, when you define a filter, with ‘n’ parameters, OrangeScape will define the required index, taking into consideration all the ‘n’ parameters. Now, what does that mean.

    consider the above filter condition, now OrangeScape would define an index for this like this,

- kind: PrimaryData
    - name: Name
    - name: Age
    - name: SheetMetadataId
    - name: Description

which GAE will build to serve queries. When you use this filter in runtime to filter records, GAE will search for this index, and based on this, it will return the filtered data, which will be displayed, in the application.

    If you had noticed, there was an option saying “Don’t Allow None in filter values”. This option enables you to say, that either all filter options have to be provided, or you can leave out some of the filter options. Regardless of whether you check this or not, the same index will be the one which will be defined.

    Now lets see how this will affect the indexes. Before that let’s look at how the query will be for when Don’t Allow None in filter values is checked and unchecked.

search values :
Name : abc
Age :
Description : software

when unChecked (default) :
select column from table where Name = ‘abc’ and Age = ‘’ and Description like ‘software’

when Checked :
select column from table where Name = ‘abc’ and Description like ‘software’

    Notice the difference, the Age parameter is left out in the query when the option is checked. Now, when this query, query without the age parameter is executed, appengine will throw theNeedIndexError. and will suggest the following index.

- kind: PrimaryData
    - name: Name
    - name: SheetMetadataId
    - name: Description

    If you notice, the Age parameter is also missing in the suggested index. This is a different index, than the one which OrangeScape had defined.

    Now why does app engine ask for this index, and why OrangeScape did not define the index?

    As we had seen previously how app engine fetches data, for each query it requires a specific index. Since the filters are different in both these cases app engine expects two different indexes. Which means, for the given example, there could be four possible complex scenarios (involves inequality criteria),

Name & Age & Description
Age & Description
Name & Description

    OrangeScape will always generate index based on all the fields mentioned in the Filter Condition. When “Don’t Allow None in filter values” is checked, the number of required indexes becomes ‘four’. Considering the fact that Google App Engine only allows to create a maximum of 200 indexes, creating these many indexes for one queue of a single model’s might deplete the available quota.

    If these combinations are required in your application, then you will have to manually add the indexes suggested by app engine.  You can read more about upload indexes from the google docs.

Views: 302


You need to be a member of OrangeScape Developer Community to add comments!

Join OrangeScape Developer Community

Comment by Kamal.S on February 20, 2014 at 1:09pm

© 2018   Created by OrangeScape Technologies.   Powered by

Badges  |  Report an Issue  |  Terms of Service