Virtual File System for the Access Grid

Some existing data-grid projects:

• Biomedical Informatics Research Network (BIRN)
• National Virtual Observatory (NVO)
• Real-time Observatories, Applications, and Data management Network (ROADNet)

These projects provide access to large amounts of distributed data (reaching terabytes or petabytes) for scientific and engineering applications. The size of data collections can only grow as new experiments and computational systems require higher capacity storage for results and simulation of large systems. The Access Grid (AG) [1], [2] is a collaborative research environment with potential for distributed data and visualization across networks and platforms. However currently AG is used predominantly for videoconferencing with PowerPoint presentations. To extend the Access Grid beyond simple meeting scenarios and towards true collaborative working, we need to enable the incorporation of large data sets and collections into the AG environment.

The Project

We plan to achieve this goal by augmenting the collaborative capabilities offered by Access Grid with the Storage Resource Broker (SRB) [3], [4]. The SRB is a middle-ware data-grid technology that provides a uniform interface to large heterogeneous data collections. With it powering the majority of data-grid projects around the world, augmenting AG with SRB allows potential access to extensive amount of existing distributed data (Figure 1).

Figure 1. Merging of the Access Grid with Storage Resource Broker

There will be several stages for the integration of the SRB technology into the Access Grid: shared application, data store service, and search service. Each are described in detail below.

Shared Application
The first stage will be the development of a file-system manager as a shared application for use in the Access Grid. The client initiated shared application links the Access Grid with the SRB provided data. The initial shared application will essentially provide shared data provided by one user to one or more users. With the addition of zoneSRB, these shared data could be linked together through federation of SRB servers and/or zones [6]. See Figure 2 for the overall system architecture.

Figure 2. Shared Application System Architecture

Data Store Service
The next stage is a more seamless interface between AG and SRB - the integration of SRB technology into the AG Data Store service. This makes heterogeneous and distributed data available to users as a hierarchical file-system in the 'Data' section of the AG Venue Client. Shared applications within the venue could then easily access the data provided by SRB.

Search Service
The ability to quickly and easily search through large data sets and collections is highly desirable. In the future we plan to investigate innovative search methods that are suitable to distributed data in data-grids. These could be SQL based or natural language based queries.

Development Log

• Since stating in August 2006, I have spent time becoming familiar with the AG toolkit and the Argonne National Laboratory's (ANL) shared application API.
• As a learning exercise, I ported Shared Rasmol and Shared GRASS [7] to the new AG Version 3.0. These ports promote user migration to the updated version of Access Grid with improved robustness, performance, and interoperability.
• I have also spent time understanding the SRB implementation and investigated methods to integrate it as the virtual file-system of the AG.
• I have written a prototype SRB Browser tool independent of slashManager in wxPython for Linux. It can connect to, upload, and download files from the SRB managed data (see Figure 3 below).

Figure 3. Prototype SRB Browser Interface

• I modified the existing Python-SRB bindings to allow sharing of data through ticket-based access.
• The SRB Browser tool is now a shared application within the AccessGrid environment (see Figure 4) and AG users can use this to manage/share SRB data.

Figure 4. Updated SRB Browser Interface in the AG

• Improving the SRB Browser GUI's usability (Figure 5):
  
• Hierachical navigation with tree view of personal and shared SRB space.
  
• Right-click menu functionality i.e. common I/O operations and properties.
  
• Status bar showing relevant information e.g. number of data selected and total size.
  

Figure 5. AG3 SRB Browser Interface

• Further improvements to SRB Browser functionality (Figure 6):
  
• Able to upload directories and download collections to personal SRB space.
• Able to upload and download collections to be shared across the AG.
  
• Easy renaming, Tango icons and progress bar improves GUI usability.
• agSRB Caveats: 
1. When you share or remove items from the shared space, the tree-control selection reverts to 'All Shared Data'.
2. Currently can only navigate in shared space using the tree-control.
   

Figure 6. agSRB Browser (current)

• The Python-SRB Bindings have been further modified with SRB error msg handling and renaming of files/collections.
• A presentation about this project was done by Christoph Willing at the 2007 Access Grid Retreat in Chicago [7].
• The Python-SRB bindings can now be built on Linux and Mac OSX with gcc, and Windows with Visual Studio. It is a relatively simple and documented process making it easy for developers to further extend/improve the bindings.
  

Desired Functionality

Downloads

Instructions for building from source

Installation

• Download then add the appropriate .agpkg3 to the Venue data store.
  
• Double click (or right click and choose 'Open') the newly uploaded files.

Applications

''Global'' Applications
With a tight integration between the AG and SRB we bring new and exciting possibilities for the Access Grid. The seamless access to data-grids and the large data sets within them enables the development of "global" applications. Many innovative applications are possible.

They could include:

• Google Maps
• Geographic Information System (GIS)
• Astronomical
• Medical

References

[1]    The Access Grid Project @ UQ VisLab
        http://www.vislab.uq.edu.au/research/accessgrid/index.html

[2]    Access Grid official site
        http://www.accessgrid.org/

[3]    The SRB Project @ UQ VisLab
        http://www.vislab.uq.edu.au/research/SRB/index.html

[4]    SDSC Storage Resource Broker official site
        http://www.sdsc.edu/srb/index.php/Main_Page

[5]    SRB zones @ UQ VisLab
        http://www.vislab.uq.edu.au/research/SRB/zone.html

[6]    Shared Scientific Applications for the Access Grid @ UQ VisLab
        http://www.vislab.uq.edu.au/research/accessgrid/shared_app/index.html

[7]    SRB for large data handling in the Access Grid, 2007 AG Retreat
        http://www.accessgrid.org/files/retreat07/presentations/willing_srb.pdf

Acknowledgements

- Last updated August 3, 2007 -