The DataCacheManager in Kepler

The DataCacheManager (DCM) was designed primarily to "cache" data that comes from Ecogrid. The design consists of a abstract base class where the primary function of the derived classes is to implement "how" the data is to be "download." The data is then dowloaded on its own thread, whne the download is complete the registered listeners are notified. Each data item in the cache is stored as a separate file. The DataCachManager stores the "index" or "directory" of all the items in a simple XML file.  A small separate application the DataCacheViewer will read the index file and produce a very simple table like representation of the entries. The contents of any of the Textual entries can be viewed in a free flowing text control or a table.

The initial usage pattern was for the consumer to ask the DCM for a specific piece of data or metadata and depending on the type of data or where the data may be stored the consumer would ask the cache to return a specific deriied cache object that could handle getting that type of data. The consumer should never make any assumptions about whether the data is in the cache or out on the network.

The signature of method call for getting the data in a transparent way is as follows:

    public static DataCacheItem getCacheItem(DataCacheListener aListener,
                                             String aName,
                                             String aResourceName,
                                             String aClassName)

The cache item is identified within the cache manager by the Name, ResourceName, and the ClassName. The Name, is any arbitrary name, the ResourceName, is usually the actual URI of where the data is to be retrieved from, the ClassName is the type of object that should be created to retrieve the information. You can always cast the return object to DataCachItem or to the class of the name you passed in. The listener will be notified when the data retrival completes.

For example, here is the code from the Eml200DataSource that is search metadata in Metacat:

_cacheMetaDataItem = (EcogridMetaDataCacheItem) DataCacheManager.getCacheItem(this, "MetaData " + getRecordId(), getEndpoint(), EcogridMetaDataCacheItem.class.getName());
_cacheMetaDataItem.setEndPoint(getEndpoint());
_cacheMetaDataItem.setRecordId(getRecordId());
_cacheMetaDataItem.start();

Here the call is saying that it wants to have returned to it an object of class EcogridMetaDataCacheItem and the the "endpoint" and "recordid" are used to construct the "final" URI that used to download the data. The "start" method call is for the thread and this kicks off the asynchronous retrieval on the thread. You can also call "doWork()" and it will not create a thread and perform the retrieval sychronously.

NOTE: When using asynchronous retrieval no assumptions can be made about when the consumer will be notified via the listener, so this can cause problems with timing and the retrieval of other information. For example, if you need two different datasets and "kick off" two retrievals, you can never be sure of the order of their return and the order may be different depending if they are in the cache or out on the net.

Note that there are helper methods on the DCM for retrieving objects by a named/value pair.

Generic "Look up"
Looking up an item without having it be created is as follows:
public static DataCacheItem lookupCacheItem(String  aName,
                                            String  aResourceName,
                                            String  aClassName,
                                            boolean aIgnoreResOnSearch)


The last argument indicates that the look up should NOT match on the ResourceName.


The DataCacheItem

The DataCacheItem (DCI) is responsible for the thread that does the work to get the data, the loading and saving of the data into a file, and the generation and consumption of the XML for its directory entry.

The DCI has several preset attributes that are used to describe it within the cache directory. More recently the each DCI can have one or more "named" attributes with text values. These are automatically saved and loaded for any derived class. The DCI also provides some abstract methodsfor derived classes that enable them to have their own "preset" attributes be read and written as actual attributes of the class.

Here are the methods for getting and setting of the named value pair attributes:
public setAttr(String aName, String aValue)
public removeAttr(String aName)
public String getAttr(String aName)


(Currently is does NOT escape the name or the value, I may want to add that.)



Using the DCM File Cache Features

There are times within Kepler that an actor may construct or obtain some data that should be cached either temporarily or permanently. There are a handful of methods and a support derived class to assist with this functionality.

The motivation is that there may be a dat in memory that needs to be saved into the cache as a file, or there may already be some data on the disk that needs to be "cached."

In general there is a "putFile" and a "getFile" method, but there are also other methods to assist when working with files. When working with files and the DCM there is still a notion of the "ResourceName" but it is really just an additional way to identify a file. If the ResourceName is not null then it is used in addition to the Name arg when looking up and identifying a file caache item.

External Files
A file that is already on the disk can be physically copied into the cache or merely referenced by the cache.

To "copy" the file into the cache simple call the "putFile" indicating the file should be copied:

item = DataCacheManager.putFile(physicalFileName,
                                "logical filename",
                                "file type",
                                DataCacheFileItem.COPY_FILE_TO_CACHE);

(After the file is copied the "getFileType()" will return INTERNAL)

Note that the caller is responsible for deleting the external file after this call, but before doing so the caller should check the return value of "getStatus()" to make sure it equals "CACHE_COMPLETE" or it can check the "isReady()" ready method for true.

If the consumer (or caller) want to just reference an external file then it can put a file into the cache using "EXTERNAL" for the type argument.

item = DataCacheManager.putFile(physicalFileName,
                                "logical filename",
                                "file type"
,
                                DataCacheFileItem.EXTERNAL);

This is not recommended from the standpoint that a user may delete or move the file.


Creating Cache Files from Memory Data

If the caller has data that needs to be persisted then it can call:
DataCacheFileItem createFileItemWithBytes(String aLogicalName, String aFileType, byte[] aData)



Finding File Items in the Cache

The original API design of the cache was use the Name, ResourceName, and ClassName to do the lookup, if the item did not exist then the ResourceName was used to obtain the data. Calling getCacheItem for a file will work, but if the item is not already in the cache then DataCacheFileItem will have the status of "CACHE_EMPTY."

The best and most appropriate way to retrieve a DataCacheFileItem is to "getFile" which uses the Name and ResourceName to look up the item and return it or return "null" if it doesn't exist.

public static DataCacheFileItem getFile(String aLogicalName, String aFileType)

or

public static DataCacheFileItem getFile(String aLogicalName)

To look up an item by name AND a named/value pair use:
public static DataCacheFileItem getFile(String aLogicalName, String aAttr, String aValue)

This approach enables you assign some metadata to the object such as a description of its file type.


Or you could have it return a "list" of items that match a named/value pair:

public static Vector getCacheItemsByAttr(String aAttr, String aValue)

Note: This will return a Vector of DataCachItems not just DataCacheFileItems so it is recommended to test for "instanceof" before cast the object in the vector.

Using a DataCacheFileItem
The DataCacheFileItem has a helper method that returns a java.io.File object from the physical location of the cache object.

public File getFile()



Removing Items from the Cache

To remove an item from the cache:
public void removeItem(DataCacheItem aItem)