As explained at the introduction to Castor JDO, Castor has support for many advanced features such as caching. The below sections detail the features related to caching in Castor JDO, as their understanding is required to use Castor JDO in a performant and secure way.

In general, performance caches enhance the application performance by reducing the number of read operations against the persistence storage, by storing and reusing the last read or committed values of the object. Performance caches do not affect the behavior of short transactions or locking. It only affects persistence objects that have been released from any transactional context.

Starting from Castor 0.8.6, a performance cache implementation has been added. At a technical level, Castor maintains separate (performance) caches for each object type specified in the JDO mapping provided, allowing users to specify - for each object type individually - the type and capacity of the cache.

By default, the following cache types are available:

As some of these cache providers allow for allow you to use it in a distributed mode, this allows Castor JDO to be used in a clustered (multi-JVM) environment. Please see the section below for short summary of this feature.

Per definition, all build-in performance caches are write-through, because all changes to objects as part of a transaction should be persisted into the cache at commit time without delay.

For problems related to the use of performance caches, please consult with the relevant entries in the JDO F.A.Q..

As it stands currently, performance caches also serve a dual purpose as dirty checking caches for long-transactions. This limitation implies that the object's availability in the performance cache determines the allowed time span of a long transaction.

This might become an issue when performance caches of type 'count-limited' or 'time-limited' are being used, where objects will eventually be disposed. If an application tries to update an object that has been disposed from the dirty checking cache, an ObjectModifedException will be thrown.

The DTD declaration is as follows:

<!ELEMENT cache-type  ( param* )>
<!ATTLIST cache-type
    type           ( none | count-limited | time-limited | unlimited |
                     coherence | fkcache | jcache | jcs | oscache |
                     fifo | lru | ehcache | gigaspaces ) "count-limited"
    debug          (true|false) "false"
    capacity       NMTOKEN  #IMPLIED>

<!ELEMENT param EMPTY>
<!ATTLIST param
          name   NMTOKEN  #REQUIRED
          value  NMTOKEN  #REQUIRED>

With release 1.0 of Castor the DTD has changed but it is backward compatible to the old one and allows to enable debugging of cache access for a specific class as well as passing individual configuration parameters to each cache instance. Only count-limited and time-limited of the current build-in cache types support parameters. Parameter names are case sensitive and are silently ignored if they are unknown to a cache type.

It need to be noted that there are 3 parameter names that are reserved for internal use. If you specify a parameter with one of the names: type, name or debug their value will silently be overwritten with another one used internally.

<cache-type type="count-limited" capacity="500"/>

<cache-type type="count-limited"/>
    <param name="capacity" value="500"/>
</cache-type>

<cache-type type="time-limited" capacity="900"/>

<cache-type type="time-limited"/>
    <param name="ttl" value="900"/>
</cache-type>

The debug attribute can be used to enable debugging for objects of a single class. In addition to setting this attribut to true you also need to set logging level of org.castor.cache.Cache to debug.

	Note
	The default cache-type is count-limited with a capacity of 30. This will be used when no cache-type is specified in the mapping for a class.

The cache types fifo and lru are based on a set of articles in the O'Reilly Network by William Grosso, to implement a simplified and 1.3-compatible implementation of a Hashbelt algorithm.

Hashbelts are simple, in principle. Instead of walking all objects and finding out when they're supposed to expire, use a "conveyor belt" approach. At any particular point in time, objects going into the cache go into the front of the conveyor belt. After a certain amount of time or when the size limit of a container has been reached, move the conveyor belt - put a new, empty container at the front of the conveyor belt to catch new objects, and the one that drops off of the end of the conveyor belt is, by definition, ready for garbage collection.

As seen in his system, you can use a set of pluggable strategies to implement the actual hashbelt bits. A container strategy allows you to change out the implementation of the container itself - from simple hashtable-based implementations, up through more complex uses of soft referenced or hashset-based implementations, depending on what you need and what you want it to be used for. A pluggable "expire behavior" handler allows you to determine what action is taken on something which drops off of the bottom of the conveyor belt.

In difference to all other cache types the fifo and lru cache types offer various configuration options. Both of them have 6 parameters to configure their behaviour.

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
       <cache-type type="fifo"/>
       ...
    </class>
    ...
</mapping>

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
       <cache-type type="lru" capacity="300"/>
       ...
    </class>
    ...
</mapping>

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
       <cache-type type="lru">
          <param name="capacity" value="300"/>
          <param name="ttl" value="300"/>
       </cache-type>
       ...
    </class>
    ...
</mapping>

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
       <cache-type type="fifo"/>
          <param name="container" value="10"/>
          <param name="capacity" value="1000"/>
          <param name="ttl" value="600"/>
          <param name="monitor" value="5"/>
          <param name="container-class" value="org.castor.cache.hashbelt.container.WeakReferenceContainer"/>
          <param name="reaper-class" value="org.castor.cache.hashbelt.reaper.NullReaper"/>
       </cache-type>
       ...
    </class>
    ...
</mapping>

All of the cache providers added with release 0.9.9 are distributed caches per se or can be configured to operate in such a mode. This effectively allows Castor JDO to be used in a clustered J2EE (multi-JVM) environment, where Castor JDO runs on each of the cluster instances, and where cache state is automatically snychronized between these instances.

In such an environment, Castor JDO wil make use of the underlying cache provider to replicate/distribute the content of a specific cache between the various JDOManager instances. Through the distribution mechanism of the cache provider, a client of a Castor JDO instance on one JVM will see any updates made to domain objects performed against any other JVM/JDO instance.

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
        <cache-type type="coherence" />
       ...
    </class>
    ...
</mapping>

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
        <cache-type type="gigaspaces" />
       ...
    </class>
    ...
</mapping>

<mapping>
    ...
    <class name="com.xyz.MyOtherObject">
       ...
        <cache-type type="gigaspaces" >
           <param name="spaceURL" value="/./" />
           <param name="spaceProperties" value="useLocalCache" />
        </cache-type>
       ...
    </class>
    ...
</mapping>

As of release 0.9.6, Castor allows for the addition of user-defined cache implementations. Whilst Castor provides a set of pre-built cache providers, offering a variety of different cache algorithms, special needs still might require the application developer to implement a custom cache algorithm. Castor facilitates such need by making available standardized interfaces and an easy to understand recipe for integrating a custom cache provider with Castor.

As explained in org.exolab.castor.jdo.persist (API docs for the persists package), org.exolab.castor.persist.LockEngine implements a persistence engine that caches objects in memory for performance reasons and thus reduces the number of operations against the persistence storage.

The main component of this package is the interface org.castor.cache.Cache, which declares the external functionality of a (performance) cache. Existing (and future) cache implementations (have to) implement this interface, which is closely modelled after the java.util.Map interface.

Below is a summary of the steps required to build a custom cache provider and register it with Castor JDO:

/**
 * My own cache implementation
 */ 
 public class CustomCache extends AbstractBaseCache {
 
    ...
    
 }

/**
 * My own cache factory implementation
 */ 
 public class CustomCacheFactory extends AbstractCacheFactory {
 
    /**
     * The name of the factory
     */
    private static final String NAME = "custom";

    /**
     * Full class name of the underlying cache implementation.
     */
    private static final String CLASS_NAME = "my.company.project.CustomCache"; 
    
    /**
     * Returns the short alias for this factory instance.
     * @return The short alias name. 
     */
    public String getName() {
        return NAME;
    }
    
    /**
     * Returns the full class name of the underlying cache implementation.
     * @return The full cache class name. 
     */
    public String getCacheClassName() {
        return CLASS_NAME;   
    }
    
 }

# 
# Cache implementations
# 
org.castor.cache.Factories=\
  org.castor.cache.simple.NoCacheFactory,\
  org.castor.cache.simple.TimeLimitedFactory,\
  org.castor.cache.simple.CountLimitedFactory,\
  org.castor.cache.simple.UnlimitedFactory,\
  org.castor.cache.distributed.FKCacheFactory,\
  org.castor.cache.distributed.JcsCacheFactory,\
  org.castor.cache.distributed.JCacheFactory,\
  org.castor.cache.distributed.CoherenceCacheFactory,\
  org.castor.cache.distributed.OsCacheFactory,\
  org.castor.cache.hashbelt.FIFOHashbeltFactory,\
  org.castor.cache.hashbelt.LRUHashbeltFactory

# 
# Cache implementations
# 
org.castor.cache.Factories=\
  org.castor.cache.simple.NoCacheFactory,\
  org.castor.cache.simple.TimeLimitedFactory,\
  org.castor.cache.simple.CountLimitedFactory,\
  org.castor.cache.simple.UnlimitedFactory,\
  org.castor.cache.distributed.FKCacheFactory,\
  org.castor.cache.distributed.JcsCacheFactory,\
  org.castor.cache.distributed.JCacheFactory,\
  org.castor.cache.distributed.CoherenceCacheFactory,\
  org.castor.cache.distributed.OsCacheFactory,\
  org.castor.cache.hashbelt.FIFOHashbeltFactory,\
  org.castor.cache.hashbelt.LRUHashbeltFactory,\
  org.whatever.somewhere.nevermind.CustomCache

Sometimes it is necessary to interact with Castor's (performance) caches to e.g. (selectively) clear a Castor cache of its content, or inquire about whether a particular object instance (as identified by its identity) is cached already.

For this purpose a org.exolab.castor.jdo.CacheManager can be obtained from a org.exolab.castor.jdo.Database instance by issuing the following code:

	  
JDO jdo = ....;
Database db = jdo.getDatabase();
CacheManager manager = db.getCacheManager();
	  

This instance can subsequently be used to selectively clear the Castor performance cache using one of the following methods:

To inquire whether an object has already been cached, please use the following method:

Please note that once you have closed the Database instance from which you have obtained the CacheManager, the CacheManager cannot be used anymore and will throw a org.exolab.castor.jdo.PersistenceException.

Release 0.9.6:

The Castor OQL implementation is currently in phase 3 of development.

	Note
	This documentation is not yet finished

This document describes an OQL to SQL translator to be added to the Castor JDO Java object Persistence API. The translator will accept OQL queries passed as strings, and generate a parse tree of the OQL. It will then traverse the tree creating the appropriate SQL. The user will then be able to bind parameters to parameterized queries. Type checking will be performed on the bound parameters. When the user executes the query, the system will submit the query to the SQL database, and then postprocess the SQL resultset to create the appropriate result as a Java Object or literal. The current org.exolab.castor.mapping and org.exolab.castor.persist packages will be used for metadata and RDBMS communication.

Four of the (now defunct) SourceXchange milestones for this project call for java source code. These milestones will be referred to here as phase 1, 2, 3, and 4. There are many possible OQL features that can be supported, but weren't discussed in the proposal or RFP. Many of these are probably unwanted. These additional features are specified as phase 5, which is out of the scope of this SourceXChange project.

The parser will construct a parse tree as output from an OQL query string given as input. The OQL syntax is a subset of the syntax described in the ODMG 3.0 specification section 4.12, with some additional constructs. Following is a description of the supported OQL syntax, and its SQL equivalent.

Certain features of OQL may not be directly translatable to SQL, but may still be supported, by post processing the query. For example, the first() and last() collection functions supported in OQL are not directly translatable to standard SQL, but a resultset can be post-processed to return the appropriate values. Features requiring post-processing of SQL resultsets will be documented as such below.

Currently the OQLQuery checks for correct syntax at the same time as it does type checking and other types of error checking. The new code will involve a multiple pass strategy, with the following passes:

This section describes the first pass which will be done by the parser. The parser will create a StringTokenizer like this:

StringTokenizer tokenizer
    = new StringTokenizer(oql,
    "\n\r\t,.()[]+-*/<>=:|$", true);

This will create a StringTokenizer with the delimiter characters listed in the second argument, and it will return delimeters as well as tokens. The parser will also create a Vector to be used as a token buffer. As tokens are returned from the StringTokenizer they will be added to the Vector. Older tokens will be removed from the Vector when it reaches a certain size. The Vector will also be modified when the StringTokenizer returns multi character operators as seperate tokens, for example the -> method invocation operator.

The parser will consume tokens from the StringTokenizer, generating a ParseTree. Each ParseTree node will have a nodeType corresponding to its symbol in the OQL syntax. After each node is created it will look at the next token and act acordingly, either modifying its properties (i.e. for DISTINCT property of selectExpr), creating a new child node or returning an error. If the error travels up to the root node of the ParseTree, there is a syntax error in the OQL submitted. At the end of this pass, the ParseTree will contain an apropriate representation of the query, which will be analyzed, and used to create SQL. Below is the modified EBNF which will be the Castor OQL syntax.

query                   ::= selectExpr
                            | expr

selectExpr              ::= select [distinct] 
                            projectionAttributes
                            fromClause
                            [whereClause]
                            [groupClause]
                            [orderClause]
                            [limitClause [offsetClause]]

projectionAttributes    ::= projectionList
                          | *

projectionList          ::= projection {, projection }

projection              ::= field
                          | expr [as identifier]

fromClause              ::= from iteratorDef {, iteratorDef}

iteratorDef             ::= identifier [ [as ] identifier ]
                          | identifier in identifier

whereClause             ::= where expr

groupClause             ::= group by fieldList {havingClause}

havingClause            ::= having expr

orderClause             ::= order by sortCriteria

limitClause             ::= limit  queryParam

offsetClause            ::= offset queryParam

sortCriteria            ::= sortCriterion {, sortCriterion }

sortCriterion           ::= expr [ (asc | desc) ]

expr                    ::= castExpr

castExpr                ::= orExpr
                            | ( type ) castExpr

orExpr                  ::= andExpr {or andExpr}

andExpr                 ::= quantifierExpr {and quantifierExpr}

quantifierExpr          ::= equalityExpr
                            | for all inClause : equalityExpr
                            | exists inClause : equalityExpr

inClause                ::= identifier in expr

equalityExpr            ::= relationalExpr
                            {(= | !=) 
                                [compositePredicate] relationalexpr }
                            | relationalExpr {like relationalExpr}

relationalExpr          ::= additiveExpr
                            {(< | <= 
                                | > | >= ) 
                                    [ compositePredicate ] additiveExpr }
                                | additiveExpr between 
                                    additiveExpr and additiveExpr

compositePredicate      ::= some | any | all

additiveExpr            ::= multiplicativeExpr
                                {+ multiplicativeExpr}
                            | multiplicativeExpr
                                {- multiplicativeExpr}
                            | multiplicativeExpr
                                {union multiplicativeExpr}
                            | multiplicativeExpr
                                {except multiplicativeExpr}
                            | multiplicativeExpr
                                {|| multiplicativeExpr}

multiplicativeExpr      ::= inExpr {* inExpr}
                            | inExpr {/ inExpr}
                            | inExpr {mod inExpr}
                            | inExpr {intersect inExpr}

inExpr                  ::= unaryExpr {in unaryExpr}

unaryExpr               ::= + unaryExpr
                            | - unaryExpr
                            | abs unaryExpr
                            | not unaryExpr
                            | postfixExpr

postfixExpr             ::= primaryExpr{[ index ]}
                            | primaryExpr
                            {(. | ->)identifier[arglist]}

index                   ::= expr {, expr}
                            | expr : expr

argList                 ::= ([ valueList ])

primaryExpr             ::= conversionExpr
                            | collectionExpr
                            | aggregateExpr
                            | undefinedExpr
                            | collectionConstruction
                            | identifier[ arglist ]
                            | queryParam
                            | literal
                            | ( query )

conversionExpr          ::= listtoset( query )
                            | element( query )
                            | distinct( query )
                            | flatten( query )

collectionExpr          ::= first( query )
                            | last( query )
                            | unique( query )
                            | exists( query )

aggregateExpr           ::= sum( query )
                            | min( query )
                            | max( query )
                            | avg( query )
                            | count(( query | * ))

undefinedExpr           ::= is_undefined( query )
                            | is_defined( query )

fieldList               ::= field {, field}

field                   ::= identifier: expr

collectionConstruction  ::= array([valueList])
                            | set([valueList])
                            | bag([valueList])
                            | list([valueList])
                            | list(listRange)

valueList               ::= expr {, expr}

listRange               ::= expr..expr

queryParam              ::= $[(type)]longLiteral

type                    ::= [unsigned] short
                            | [unsigned] long
                            | long long
                            | float
                            | double
                            | char
                            | string
                            | boolean
                            | octet
                            | enum [identifier.]identifier
                            | date
                            | time
                            | interval
                            | timestamp
                            | set <type>
                            | bag <type>
                            | list <type>
                            | array <type>
                            | dictionary <type, type>
                            | identifier

identifier              ::= letter{letter| digit| _}

literal                 ::= booleanLiteral
                            | longLiteral
                            | doubleLiteral
                            | charLiteral
                            | stringLiteral
                            | dateLiteral
                            | timeLiteral
                            | timestampLiteral
                            | nil
                            | undefined

booleanLiteral          ::= true
                            | false

longLiteral             ::= digit{digit}

doubleLiteral           ::= digit{digit}.digit{digit}
                            [(E | e)[+|-]digit{digit}]

charLiteral             ::= 'character'

stringLiteral           ::= "{character}"

dateLiteral             ::= date
                                'longliteral-longliteral-longliteral'

timeLiteral             ::= time
                                'longliteral:longLiteral:floatLiteral'

timestampLiteral        ::= timestamp
                                'longLiteral-longLiteral-longLiteral
                                longliteral:longLiteral:floatLiteral'

floatLiteral            ::= digit{digit}.digit{digit}

character               ::= letter
                            | digit
                            | special-character

letter                  ::= A|B|...|Z|
                            a|b|...|z

digit                   ::= 0|1|...|9

special-character       ::= ?|_|*|%|\
               

The following symbols were removed from the standard OQL Syntax for the following reasons:

The following symbols were added or modified.

The rest of the standard OQL syntax remains unchanged. Certain syntactically correct queries may not be supported in Castor. For example, top level expressions which do not contain a selectExpr anywhere in the query may not be supported. This will be discussed further in the next section.

The first pass over the ParseTree will do type checking, and create some structures used in the SQL generation pass. It will also check whether the identifiers used are valid, and whether the query uses unsupported features. The following table describes each type of node in the ParseTree, and how it will be processed in the first pass.

After the first pass, the ParseTree is free of errors, and ready for the SQL generation step. The existing implementation of the OQLParser uses the persistence API for SQL generation. This API lacks the necessary features to generate SQL from any OQL. The SQLEngine class which implements Persistence is used to create a JDBCQueryExpression. The SQL is derived from the finder, which is a JDBCQueryExpression produced by the SQLEngine. The problem is that the SQLEngine only supports single objects. It cannot generate SQL for path expressions like this:

select p.address from Person p 

This query requires a SQL statement like this:

select address.* from person, address
    where person.address_id = address.address_id

The buildFinder method should not be used to generate a queryExpression. The SQLEngine should be used to get a ClassDescriptor, and to create a new QueryExpression. The OQLParser should use the methods in the QueryExpression to generate the SQL. The JDBCQueryExpression which is an implementation of QueryExpression is also lacking in necessary features. This class should continue to be used, but the following features will need to be added:

The following table lists each type of tree node, and how it will be processed in the SQL generation pass.

Please see the OQL section of the JDO FAQ.

The Parser and ParseTree classes will be improved through the phases of this project. The top level of OQL to SQL translation will look very simple, like this:

OQLParser parser = new OQLParser(query);
ParseTree pt = parser.getParseTree();
pt.checkPass();
//the SQL generation pass
_expr = pt.getQueryExpr();
            

These methods will have some additional parameters passed for storing and retrieving data relevant to the query. Following is a table containing a list of what will be introduced in each coding phase of the project.

Please find below various examples of OQL queries using the Java class files as outlined below.

package myapp;

public class Product 
{
    private int       _id;

    private String    _name; 

    private float     _price; 

    private ProductGroup _group;


    public int getId() { ... }
    public void setId( int anId ) { ... }

    public String getName() { ... }
    public void setName( String aName ) { ... }

    public float getPrice() { ... }
    public void setPrice( float aPrice ) { ... }

    public ProductGroup getProductGroup() { ... }
    public void setProductGroup( ProductGroup aProductGroup ) { ... }
}
	  

public class ProductGroup
{

    private int       _id;

    private String    _name;

    public int getId() { ... }
    public void setId( int id ) { ... }

    public String getName() { ... }
    public void setName( String name ) { ... }

}
         

query = db.getOQLQuery("select p from ProductGroup as p LIMIT $1");
query.bind(10);
         

query = db.getOQLQuery ("select p from ProductGroup as p LIMIT $1 OFFSET $2");
query.bind(10);
         

In order to understand how the JDO transaction model affects applications performance and transactional integrity, you must first understand the semantics of the Java Data Objects. Java Data Objects are objects loaded from and stored to the database, but are disassociated from the database itself. That is, once an object has been loaded into memory, changes to the object in memory are not reflect in the database until the transaction commits.

The following table shows a sample code and the state of the JDO object and the relevant database field:

Concurrency conflicts do not occur when an object is changed in memory, but rather when the transaction commits and changes are saved back to the database. No changes are saved if the transaction rolls back.

Conflicts could occur if two threads attempted to modify the same object, or the same thread receives two objects associated with the same database record and performs different changes to each object. Castor solves these issues through a transaction-object-identity association.

When the same transaction attempts to query the same database record twice, (e.g. as the result of two different queries) the same object is returned, assuring that different changes will be synchronized through the same object.

When two transactions attempt to query the same database record, each transaction receives a different object mapping to the same record, assuring that changes done in one transaction are not visible to the other transactions. Changes should only become visible when the transaction commits and all its changes are made durable.

This approach places the responsibility of synchronization and deadlock detection on the Castor persistence engine, easing the life of the developer.

Concurrent access requires use of locking to synchronize two transactions attempting to work with the same object. The locking mechanism has to take into account several possible use of objects, as well as help minimize database access through caching and is-modified checking.

Locking modes are declared in the class element of the XML mapping on a per class basis.

The visibility of changes occuring in one transaction to other transactions depends upon the transaction isolation level specified for the database connection. Whether or not the changes are visible in the current transaction depends upon the operation being done.

There are four types of changes, the following table summarizes the affect of each change in one transaction on other queries in that transaction as well as other transactions.

The Castor persistence engine is based on a layer architecture allowing different APIs to be plugged on top of the system, and different persistence engines to be combined in a single environment.

At the top level are the application level APIs. These are industry standard APIs that allow an application to be ported in and to other environments. These APIs consist of interfaces as well as semantics that make them suitable for a particular type of applications.

At the medium level is the Castor persistence mechanism. The persistence mechanism exposes itself to the application through the application level APIs. These typically have a one to one mapping with the persistence mechanism API. The persistence mechanism takes care of object caching and rollback, locking and deadlock detection, transactional integrity, and two phase commit.

At the bottom level are the Castor service providers. SPIs provide the persistence and query support into a variety of persistence mechanism. This version of Castor is bundled with an SQL 92 and LDAP persistence SPIs. Additional SPIs can be added, for example, alternative engines streamlined for Oracle, Sybase, DB2 and other databases.

This document will describe the persistence mechanism API and SPI to allow for those interested in adding new top level APIs or service providers.

The persistence mechanism is responsible for object caching and rollback, locking and deadlock detection, transaction integrity and two phase commit. All data access goes through the persistence mechanism. All operations are performed in the context of a transaction, even if the underlying SPI does not support transactions (e.g. LDAP).

The persistence API is defined in the package org.exolab.castor.persist. The persistence mechanism implements the org.exolab.castor.persist.PersistenceEngine interface, which allows objects to be loaded, created, deleted and locked in the context of a transaction. The actual implementation is obtained from org.exolab.castor.persist.PersistenceEngineFactory.

All operations are performed through the context of a transaction. A transaction is required in order to properly manage locking and caching, and to automatically commit or rollback objects at transaction termination (write-at-commit). Persistence operations are performed through the org.exolab.castor.persist.TransactionContext interface.

The actual implementation of a transaction context is specific to each application API and set of SPIs. One is created from an org.exolab.castor.persist.XAResourceSource which abstracts the data source for the purpose of connection pooling and XA transaction enlistment. A default implementation of XAResource is available from org.exolab.castor.persist.XAResourceImpl.

Castor supports a service provider architecture that allows different persistence storage providers to be plugged in. The default implementation includes an SQL 92 provider and an and an LDAP provider. Additional providers will be available optimized for a particular database server or other storage mechanisms.

The service provider is defined through two interfaces, org.exolab.castor.persist.spi.Persistence and org.exolab.castor.persist.spi.PersistenceQuery. The first provides creation, deletion, update and lock services, the second is used to process queries and result sets. Service providers are obtained through the org.exolab.castor.persist.spi.PersistenceFactory interface.

TBD

The key generator gives a possibility to generate identity field values automatically. During create the value of the identity field is set to the value obtained from the key generator. Different algorithms may be used here, both generic and specific for database server.

The key generator for the given class is set in the mapping specification file (mapping.xml), in the key-generator attribute of the class element, for example:

<class name="myapp.ProductGroup"
       identity="id" key-generator="MAX">
  <field name="id">
  </field>
</class>
      

The following key generator names are supported in Castor 1.0:

Some of these algorithms have parameters, which can be specified in the key-generator element of the mapping specification file, for example:

<key-generator name="HIGH-LOW">
  <param name="table" value="SEQ"/>
  <param name="key-column" value="SEQ_TableName"/>
  <param name="value-column" value="SEQ_MaxPKValue"/>
  <param name="grab-size" value="1000"/>
</key-generator>

<class name="myapp.ProductGroup"
       identity="id" key-generator="HIGH-LOW">
  <field name="id">
  </field>
</class>

It is possible to create several key generators with the same algorithms but different parameters. In this case you have to specify the alias attribute in the key-generator element, for example:

<key-generator name="SEQUENCE" alias="A">
  <param name="sequence" value="a_seq"/>
</key-generator>

<key-generator name="SEQUENCE" alias="RETURNING">
  <param name="sequence" value="b_seq"/>
  <param name="returning" value="true"/>
</key-generator>

<class name="myapp.ProductGroup"
       identity="id" key-generator="RETURNING">
  <field name="id">
  </field>
</class>
      

Below all supported key generators a described in details.

MAX key generator fetches the maximum value of the primary key and lock the record having this value until the end of transaction. Then the generated value is set to (MAX + 1). Due to the lock concurrent transactions which perform insert to the same table using the same key generator algorithm will wait until the end of the transaction and then will fetch new MAX value. Thus, duplicate key exception is almost impossible (see below). Note, that it is still possible to perform multiple inserts during the same transaction.

There is one "singular" case of this algorithm: the case when the table is empty. In this case there is nothing to lock, so duplicate key exception is possible. The generated value in this case is 1.

This algorithm has no parameters. Primary key must have type integer, bigint or numeric.

This key generator uses one of the variants of the generic HIGH-LOW algorithm. It is needed a special auxiliary table ("sequence table") which has the unique column which contains table names (of the type char or varchar) and the column which is used to reserve values of the primary keys (of the type integer, bigint or numeric). The key generator seeks for the given table name, reads the last reserved value and increases it by some number N, which is called "grab size". Then the lock on the auxiliary table is released, so that concurrent transactions can perform insert to the same table. The key generator generates the first value from the grabbed interval. During the next (N - 1) invocations it generates the other grabbed values without database access, and then grabs the next portion. Note, that the auxiliary table must be in the same database as the table for which key is generated. So, if you work with multiple databases, you must have one auxiliary table in each database.

If the grab size is set to 1, the key generator each time stores the true maximum value of the primary key to the auxiliary table. In this case the HIGH-LOW key generator is basically equivalent to the MAX key generator. On you want to have LOW part of the key consisting of 3 decimal digits, set the grab size to 1000. If you want to have 2 LOW bytes in the key, set the grab size to 65536. When you increase the grab size, the speed of key generation also increases because the average number of SQL commands that are needed to generate one key is (2 / N). But that average number of key values that will be skipped (N / 2) also increases.

The HIGH-LOW key generator has the following parameters:

If the parameter global is set to true, the sequence table contains only one row with the value set in parameter global-key (or "<GLOBAL>" if "global-key was not set") instead of the table name. This row serves for all tables.

Don't forget to set same-connection="true" if you are working in EJB environment!

Note, that the class HighLowKeyGenerator is not final, so you can extend it in order to implement other variants of HIGH-LOW algorithm (for example, HIGH/MID/LOW or char key values).

This key generator generates global unique primary keys. The generated key is a combination of the IP address, the current time in milliseconds since 1970 and a static counter. The complete key consists of a 30 character fixed length string.

This algorithm has no parameters. Primary key must have type char, varchar or longvarchar.

IDENTITY key generator can be used only with autoincrement primary key columns (identities) with Sybase ASE/ASA, MS SQL Server, MySQL and Hypersonic SQL.

After the insert the key generator selects system variable @@identity which contains the last identity value for the current database connection.

In the case of MySQL and Hypersonic SQL the system functions LAST_INSERT_ID() and IDENTITY() are called, respectively.

This algorithm has no parameters.

The SEQUENCE key generator type is supported in conjunction with the following DBMS: Derby, Interbase, Oracle, PostgreSQL, and SAP DB.

It generates keys using database sequence objects.

The key generator has the following parameters:

Usually one sequence is used for one table, so in general you have to define one key generator per table.

But if you use some naming pattern for sequences, you can use one key generator for all tables.

For example, if you always obtain sequence name by adding "_seq" to the name of the correspondent table, you can set "sequence" parameter of the key generator to "{0}_seq" (the default value).

In this case the key generator will use sequence "a_seq" for table "a", "b_seq" for table "b", etc. Castor also allows for inserting the primary key into the sequence name as well. This is accomplished by including the {1} tag into the "sequence" parameter. Example: "{0}_{1}_seq"

Actually the SEQUENCE key generator is "4 in 1". With PostgreSQL it performs "SELECT nextval(sequenceName)" before INSERT and produces identity value that is then used in INSERT. Similarly, with Interbase it performs "select gen_id(sequenceName, increment) from rdb$database" before INSERT.

With Oracle by default (returning=false) and with SAP DB it transforms the Castor-generated INSERT statement into the form "INSERT INTO tableName (pkName,...) VALUES (sequenceName.nextval,...)" and after INSERT it performs "SELECT seqName.currval FROM tableName" to obtain the identity value. With Oracle8i it is possible to use more efficient RETURNING mode: to the above INSERT statement is appened "RETURNING primKeyName INTO ?" and the identity value is fetched by Castor during INSERT, so that only one SQL query is needed.

In case when your table has an on_Insert trigger which already generates values for your key, like the following Oracle example:

create or replace trigger "trigger_name"
before insert on "table_name" for each row 
begin 
   select "seq_name".nextval into :new."pk_name" from dual;
end;

you may set the "trigger" parameter to "true". This will prevent the "Sequence_name".nexval from being pulled twice (first time in the insert statement (see above), then in the trigger). Also usefull in combination with the "returning" parameter set to "true" for Oracle (in this case you may not specify the sequence name).

The usual Castor transactions are called here short transactions: an object is read, modified and written within the bounds of one transaction. Long transactions consist of two Castor transactions: an object is read in the first and written in the second. Between them the object is sent "outwards" and modified there. For example, the object may be send to a client application and dispayed to a user, or it may be sent to a servlet engine and is displayed on a web page. After that the modified object returns back and is written to the database in the second transaction. At this point the object is usually not the same physical instance as one that was read in the first transaction. The example code for writing the object in the second Castor transaction follows:

    // a customer go to a webpage to review her personal information.
    // The servlet then call this server side function: getCustomerInfo
    public CustomerInfo getCustomerInfo( Integer customNum ) {
         
         // in most case, users simply review information and
         // make no change. Even if they make changes, it often
         // takes time for them to decide. We don't want to
         // lock the database row, so commit right after we load.
         db.begin();
         CustomerInfo info = (CustomerInfo)
            db.load( CustomerInfo.class, customNum );

         
         // we also want to keep track of customers patterns
         // well...it helps us provide better service.
         info.setLastVisit( today );
         db.commit();
         return info;
    }

    // Three days passed, the indecisive customer finally agrees to
    // marry Joe. She changes her last name in the webpage and
    // clicked the "Submit" button on the webpage.
    
    // The servlet then calls updateCustomerInfo to update the
    // last name for the indecisive customer.
    public void updateCustomerInfo( CustomerInfo info ) {
        db.begin();
        db.update(info);
        db.commit();
    }
         

Note, that it is natural to read the object in the first transaction in the read-only mode.

Since the time interval between the first and the second transaction is relatively big, it is desirable to perform dirty checking, i.e. to check that the object has not been modified in the database during the long transaction. For that the object must hold a timestamp: it is set by Castor during the first Castor transaction and is checked during the second one. In order to enable the dirty checking for long transactions, the object should implement the interface org.exolab.castor.jdo.TimeStampable having two methods: long jdoGetTimeStamp() and void jdoSetTimeStamp(long timeStamp)

The advantage of the bounded dirty checking is that it doesn't require any changes to the database schema. It uses the Castor cache to store object timestamps. The disadvantage of this algorithm is that it is bounded by a lifetime of the cached copy of the object. After the cached copy has been purged, db.update() causes ObjectModifiedException.

Thus, parameters of the cache define dirty checking capabilities. The cache-type attribute is part of the <class> element in the XML mapping. Consider the existing cache types:

For long transactions (detached objects) it was required that the entity has been kept in cache from being loaded until its update. If the entity was expired from cache before the update an ObjectModifiedException had been thrown. While this is no problem if all entities of an application can be kept in cache all the time, it is one for large scale applications with millions of entities.

With release 1.3 we have changed the handling of timestamps. While it is still possible to rely on cache only it is now also possible to persist the timestamp together with the other properties of the entity. Doing so will ensure that the timestamp do not change even if the entity got expired from cache from being loaded until it gets updated. If this happens the entity gets reloaded during update which also loads the previous timestamp. Having said that it still is possible that an ObjectModifiedException is thrown when another user has changed the same entity in the meantime.

See an example entity and its mapping below:

public class Entity implements TimeStampable {
    private Integer _id;
    private String _name;
    private long _timeStamp;
    
    public Integer getId() { return _id; }
    public void setId(final Integer id) { _id = id; }
    
    public String getName() { return _name; }
    public void setName(final String name) { _name = name; }
    
    public long getTimeStamp() { return _timeStamp; }
    public void setTimeStamp(final long timeStamp) {
        _timeStamp = timeStamp;
    }
    
    public long jdoGetTimeStamp() { return _timeStamp; }
    public void jdoSetTimeStamp(final long timestamp) {
        _timeStamp = timestamp;
    }
}       

<class name="Entity">
    <cache-type type="time-limited" capacity="300"/>
    <map-to table="entity"/>
    <field name="id" type="integer" identity="true">
        <sql name="id" type="integer"/>
    </field>
    <field name="name" type="string">
        <sql name="name" type="char"/>
    </field>
    <field name="timeStamp" type="long">
        <sql name="timestamp" type="numeric" />
    </field>
</class>
        

In some cases it is desirable to map a plain sequence of fields in a database record to more complicated structure of attributes in a Java object, where the target attributes are contained (nested) in other attributes. In brief, you can specify a path to the target attribute as a name of the field in a configuration file, and Castor is able to handle such nested attributes. For example:

<field name="address.country.code"...>
  <sql name="person_country"/>
</field>
	        

The first case is an attribute of an application type that is a container for some value of a Java type supported by Castor. Usually the application type also has some business methods. Examples are: class Balance that contains a BigDecimal value and has some accounting-specific methods; class CountryCode that contains a String value and has methods validate(), getDisplayName(), etc.; class Signature that contains a byte[] value and has some security-specific methods. In order to use such type with Castor you should provide a pair of methods to get/set the value of the Castor-supported type, e.g. getBigDecimal/setBigDecimal, getCode/setCode, getBytes/setBytes.

Assume that you have the class Address

public class Address {
    private CountryCode _country;
    private String      _city;
    private String      _street;

    public CountryCode getCountry() {
        return _country;
    }

    public void setCountry(CountryCode country) {
        _country = country;
    }
    ...
}
	        

where the class CountryCode is like this

public class CountryCode {
    private String _code;
    private static String[] _allCodes;
    private static String[] _allDisplayNames;

    public String getCode() {
        return _code;
    }

    public void setCode(String code) {
        _code = code;
    }

    public void getDisplayName() {
    ...
}
	        

then write in the configuration file:

<class name="Address"...>
  <field name="country.code"...>
    <sql name="addr_country"/>
  </field>
  ...
</class>
	        

When reading the object from the database Castor will use

object.getCountry().setCode(value);
	        

to set the nested attribute value. Moreover, if object.getCountry() is null, Castor will create the intermediate object of the application type:

country = new CountryCode();
country.setCode(value);
object.setCountry(country);
	        

When writing the object to the database Castor will use

value = object.getCountry().getCode();
	        

to get the value of the correspondent database field.

The second case is an attribute that is a part of a compound attribute, which contains several database fields. For example, database fields person_country, person_city, person_street of the table PERSON correspond to one compound attribute "address" of the class Person:

public class Person {
    private String  _firstName;
    private String  _lastName;
    private Address _address;

    public Address getAddress() {
        return _address;
    }

    public void setAddress(Address address) {
        _address = address;
    }
    ...
}
        

where the class Address is the same as in the previous section. Then write in the configuration file:

<class name="Person"...>
  <field name="address.country.code"...>
    <sql name="person_country"/>
  </field>
  <field name="address.city"...>
    <sql name="person_city"/>
  </field>
  <field name="address.street"...>
    <sql name="person_street"/>
  </field>
  ...
</class>
        

Similarly to the previous section, Castor will use a proper sequence of get/set methods to access the nested attributes and will create the intermediate objects when necessary. Don't forget to provide parameterless constructors for the container classes.

There is no mechanism within Castor JDO to provide pooling of JDBC drivers. Rather, Castor JDO relies on the drivers or external driver wrappers to implement a pooling mechanism. Some drivers, such as Oracle, provides a pooling mechanism in the driver. For those that do not, there are tools such as Proxool and Jakarta's DBCP project.

Here, I'll go over the various usage of the PostgreSQL driver with Castor. We start with the most basic configurations that do not use any pooling, to those with pooling via DBCP. I'll include how to configure the pooling version of the PostgreSQL JDBC driver ths will be usable with PostgreSQL 7.3 and later, how to setup a Tomcat JNDI context that Castor can use to get a pooled JDBC connection. Finally, I'll explain how to configure a BasicDataSource from the DBCP package using the <data-source> element.

A standard jdo-conf.xml entry for using PostgreSQL without pooling looks like this:

<driver class-name="org.postgresql.Driver"
        url="jdbc:postgresql://localhost/app">
  <param name="user" value="smith"/>
  <param name="password" value="secret" />
</driver>
        

On the other hand, if you wanted to use the PostgresqlDataSource, you would use the data-source tag instead, and the connection entry would look like this:

<data-source class-name="org.postgresql.PostgresqlDataSource">
 <param name="server-name" value="localhost" />
 <param name="database-name" value="app" />
 <param name="user" value="smith" />
 <param name="password" value="secret" />
</data-source>
      

(Note that only versions before 7.3 of the PostgreSQL JDBC driver include this class)

Finally, I want to show the configuration for using a pooling data-source for Castor which is retrieved from a JNDI context that Apache fills. The first example is using the PostgreSQL pooling data-source, and the second is using Castor. The information to gain here is that we did not need to change the jdo-conf.xml file or the webapp's web.xml file to achieve this.

First, we modify the deployment context for the webapp in Tomcat >= 4.0 for our webapp in the conf/server.xml directory. (With Tomcat/Catalina releases 4.0 and higher there's more than one way of adding a <Resource> entry. Please consult with the manuals for more and more detailed information).

We add the following information (using the PostgreSQL JDBC DataSource implementations as introduced above.):

	        
<Context path="/webapp" docBase="test" debug="10">
 <Resource name="jdbc/appDb" auth="Container"
            type="org.postgresql.jdbc2.optional.PoolingDataSource"/>
   <ResourceParams name="jdbc/appDb">
    <parameter>
      <name>factory</name>
      <value>org.postgresql.jdbc2.optional.PGObjectFactory</value>
    </parameter>
    <parameter>
     <name>dataSourceName</name>
     <value>appDb</value>
    </parameter>
    <parameter>
     <name>initialConnections</name>
     <value>2</value>
    </parameter>
    <parameter>
      <name>maxConnections</name>
      <value>5</value>
    </parameter>
    <parameter>
      <name>databaseName</name>
      <value>app</value>
    </parameter>
    <parameter>
      <name>user</name>
      <value>smith</value>
    </parameter>
    <parameter>
      <name>password</name>
      <value>secret</value>
    </parameter>
    <parameter>
      <name>serverName</name>
      <value>localhost</value>
    </parameter>
  </ResourceParams>
</Context>
	     

Here, we are using the PostgreSQL PGObjectFactory which provides the JNDI server (Tomcat) the ability to create the correct data source. Now, the web.xml file for the webapp needs to be updated too.

        
	        
<resource-env-ref>
  <description>PostgreSQL pooling check</description>
  <resource-env-ref-name>jdbc/appDb</resource-env-ref-name>
  <resource-env-ref-type>javax.sql.DataSource</resource-env-ref-type>
</resource-env-ref>
	
	      

Note that we are only calling the ref type a DataSource object, not using the PostgreSQL class name. This will enable us to make changes easily. Now, in the jdo-conf.xml file that Castor uses, we no longer list the driver or data-source tag, but use the JNDI one, and it is simply this:

	         
<jndi name="java:comp/env/jdbc/appDb"/>
	     

Commons-DBCP provides database connection pooling services, and together with Commons-Pool it is the default JNDI datasource provider for Tomcat.

With release 1.1 of the Jakarta Commons DBCP component, one of the major new features of the JDBC 3.0 API has (finally) been added to BasicDataSource, support for prepared statement pooling.

To configure Castor for the use of DBCP, please provide the following <data-source> entry in the jdo-conf.xml file.

       
<data-source class-name="org.apache.commons.dbcp.BasicDataSource">
  <param name="driver-class-name" value="com.mysql.jdbc.Driver" />
  <param name="username" value="test" />
  <param name="password" value="test" />
  <param name="url" value="jdbc:mysql://localhost/test" />
  <param name="max-active" value="10" />
</data-source>
        

            
<data-source class-name="org.apache.commons.dbcp.BasicDataSource">
  <param name="driver-class-name" value="com.mysql.jdbc.Driver" />
  <param name="username" value="test" />
  <param name="password" value="test" />
  <param name="url" value="jdbc:mysql://localhost/test" />
  <param name="max-active" value="10" />
  <param name="pool-prepared-statements" value="true" />
</data-source>
            

PostgreSQL's blob support has evolved over the years. Today PostgreSQL fields can be of unlimited length. And there are specific data types for character and binary large objects. The current Castor support for blobs, however, uses an earlier PostgreSQL blob support. This support places the blob data in the pg_largeobject table and a object id in the referring table. For most practical purposes using this earlier support does not matter.

Database version and the JDBC driver version matter greatly. To get everything to work I eventually built and installed PostgreSQL 7.2.2 and used the JDBC driver from this build (i.e. not the one from http://jdbc.postgresql.org.

Since Castor is using the earlier blob support the JDBC has to be placed in PostgreSQL 7.1 comparability mode. To do this use the following JDBC URL

	      jdbc:postgresql://host:port/database?compatible=7.1      

Once you have resolved the PostgreSQL version issues Castor works as documented.

Here are the details of an example configuration.

	      Client Windows 2000, Sun Java Standard Edition 1.3.1_03, Castor 0.9.3.21
	      Server RedHat 7.2, PostgreSQL 7.2.2     

The interface I am using is

	    
	    public interface Document {
	        String      getTitle();
	        void        setTitle( String title );
	        Date        getCreatedOn();
	        void        setCreatedOn( Date createdOn );
	        String      getContentType();
	        void        setContentType( String contentType );
	        InputStream getContent();
	        void        setContent( InputStream content );
	    }
	    

and this is implemented by the class DocumentImpl.

The mapping file is

	    
  <?xml version="1.0"?>
    <mapping>
     <class
         name="com.ingenta.DocumentImpl"
         identity="id"
         key-generator="SEQUENCE" >
         <description />
         <cache-type type="none" />  
         <map-to table="documents" />
         <field name="id" type="integer" >
             <sql name="id" type="integer" dirty="check" required="true"/>
         </field>
         <field name="title" type="string">
             <sql name="title" type="longvarchar" dirty="check" />
         </field>
         <field name="createdOn" type="date">
             <sql name="createdon" type="date" dirty="check" />
         </field>
         <field name="contentType" type="string">
             <sql name="contenttype" type="longvarchar" dirty="check" />
         </field>
         <field name="content" type="stream">
             <sql name="content" type="blob" dirty="ignore" />
         </field>
     </class>
 </mapping>
	 

Note that the blob is not dirty checked.

And the SQL is

	    
    create table documents (
        id          serial    not null,
        title       text      null,
        createdon   timestamp null,
        contenttype text      null,
        content     oid       null,
        primary key ( id )
    );
	    

Castor caches objects between transactions for performance. With a blob however the cached object's InputStream is not reusable. To workaround this I have told the cache to not cache any objects of this class by adding to the class mapping, as noted above.

There's many users of Castor JDO out there, who (want to) use Castor JDO in in high-volume applications. To fine-tune Castor for such environment, it is necessary to understand many of the product features in detail and to be able to balance their use according to the application needs. Even though many of these features are detailed in various places, people have frequently been asking for a 'best practise' document, a document that brings together these technical topics (in one place) and presents them as a set of easy-to-use recipes.

Please be aware that this document is under construction, but still we believe that - even when in its conception phase - it provides valuable information to users of Castor JDO.

Let's start with some general suggestions that you should have a look at. Please don't feel upset if some are really primitive but there may be users reading this document that are not aware of them.

We hope above suggestions help you to resolve the problems you have. If you still need more performance there are areas of improvement that are more difficult to resolve. For further ideas to improve your applications performance you should take a loock at out performance test suite (PTF) which you can find in Castor's source distribution under: src/tests/ptf/jdo.

Now, there's lots left to do - there is still the issue, for example, of dependent objects being slightly sub-optimal in performance both in terms of the SQL that gets generated and the way it gets managed - but there will be improvements over time to the way that this and other operations are performed.

But performance should be good right now. If it isn't, you'll need to think about whether you are using the optimal set of operations. No environment can predict your requirements - hinting to the system when objects can be safely assumed to be read-only is vital to a high-performance implementation.