The University of Queensland Homepage
UQ VisLab UQ VisLab

 Virtual File System for the Access Grid
The availability of high bandwidth networks is growing rapidly in recent years and enables innovative approaches to linking computers around the world. One approach is global computing, where powerful supercomputers and data stores around the world are being connected together to form a 'Grid' of computing and data.

Some existing data-grid projects:

  • Biomedical Informatics Research Network (BIRN)
  • National Virtual Observatory (NVO)
  • Real-time Observatories, Applications, and Data management Network (ROADNet)

These projects provide access to large amounts of distributed data (reaching terabytes or petabytes) for scientific and engineering applications. The size of data collections can only grow as new experiments and computational systems require higher capacity storage for results and simulation of large systems. The Access Grid (AG) [1], [2] is a collaborative research environment with potential for distributed data and visualization across networks and platforms. However currently AG is used predominantly for videoconferencing with PowerPoint presentations. To extend the Access Grid beyond simple meeting scenarios and towards true collaborative working, we need to enable the incorporation of large data sets and collections into the AG environment.

 

The Project

We plan to achieve this goal by augmenting the collaborative capabilities offered by Access Grid with the Storage Resource Broker (SRB) [3], [4]. The SRB is a middle-ware data-grid technology that provides a uniform interface to large heterogeneous data collections. With it powering the majority of data-grid projects around the world, augmenting AG with SRB allows potential access to extensive amount of existing distributed data (Figure 1).

 

Figure 1. Merging of the Access Grid with Storage Resource Broker


There will be several stages for the integration of the SRB technology into the Access Grid: shared application, data store service, and search service. Each are described in detail below.

Shared Application
The first stage will be the development of a file-system manager as a shared application for use in the Access Grid. The client initiated shared application links the Access Grid with the SRB provided data. The initial shared application will essentially provide shared data provided by one user to one or more users. With the addition of zoneSRB, these shared data could be linked together through federation of SRB servers and/or zones [6]. See Figure 2 for the overall system architecture.


Figure 2. Shared Application System Architecture


Data Store Service
The next stage is a more seamless interface between AG and SRB - the integration of SRB technology into the AG Data Store service. This makes heterogeneous and distributed data available to users as a hierarchical file-system in the 'Data' section of the AG Venue Client. Shared applications within the venue could then easily access the data provided by SRB.

Search Service
The ability to quickly and easily search through large data sets and collections is highly desirable. In the future we plan to investigate innovative search methods that are suitable to distributed data in data-grids. These could be SQL based or natural language based queries.

 

Development Log

  • Since stating in August 2006, I have spent time becoming familiar with the AG toolkit and the Argonne National Laboratory's (ANL) shared application API.
  • As a learning exercise, I ported Shared Rasmol and Shared GRASS [7] to the new AG Version 3.0. These ports promote user migration to the updated version of Access Grid with improved robustness, performance, and interoperability.
  • I have also spent time understanding the SRB implementation and investigated methods to integrate it as the virtual file-system of the AG.
  • I have written a prototype SRB Browser tool independent of slashManager in wxPython for Linux. It can connect to, upload, and download files from the SRB managed data (see Figure 3 below).

 

Figure 3. Prototype SRB Browser Interface


  • I modified the existing Python-SRB bindings to allow sharing of data through ticket-based access.
  • The SRB Browser tool is now a shared application within the AccessGrid environment (see Figure 4) and AG users can use this to manage/share SRB data.

new SRB browser

Figure 4. Updated SRB Browser Interface in the AG


  • Improving the SRB Browser GUI's usability (Figure 5):
    • Hierachical navigation with tree view of personal and shared SRB space.
    • Right-click menu functionality i.e. common I/O operations and properties.
    • Status bar showing relevant information e.g. number of data selected and total size.

AG3 SRB browser

Figure 5. AG3 SRB Browser Interface

  • Further improvements to SRB Browser functionality (Figure 6):
    • Able to upload directories and download collections to personal SRB space.
    • Able to upload and download collections to be shared across the AG.
    • Easy renaming, Tango icons and progress bar improves GUI usability.
    • agSRB Caveats: 
      1. When you share or remove items from the shared space, the tree-control selection reverts to 'All Shared Data'.
      2. Currently can only navigate in shared space using the tree-control.

AG3 SRB browser

Figure 6. agSRB Browser (current)

  • The Python-SRB Bindings have been further modified with SRB error msg handling and renaming of files/collections.
  • A presentation about this project was done by Christoph Willing at the 2007 Access Grid Retreat in Chicago [7].
  • The Python-SRB bindings can now be built on Linux and Mac OSX with gcc, and Windows with Visual Studio. It is a relatively simple and documented process making it easy for developers to further extend/improve the bindings.

Desired Functionality

CORE (Completed)

EXTENDED (In development)

  • Connect to a SRB server
  • Navigation within the SRB Server
  • Navigation within the AG Shared data
  • Upload/Download Data from SRB server
  • Upload/Download Collection from SRB server
  • Share Data from SRB server to AG Users
  • Sharing via DataStore for non-SRB-capable users
  • General GUI Ease-of-Use

  • Integration into or Companion of AG Data Store
  • Manipulation of Data Permissions
  • Exposing of SRB Metadata
  • Searching Capabilities
  • Incorporation of Bittorrent or P2P program


Downloads

Instructions for building from source

Quick guide for testing the bindings

for AG Toolkit 3.02 (or later) and SRB Version 3.4.2  

agSRB Client (with bindings):
AG package for Linux distros (.agpkg3) - coming soon!
Latest code from SVN repository - coming soon!

Python-SRB bindings source code:
Linux and OS X source distribution (.tar.gz)     
Windows source distribution (.zip)
Latest code from SVN repository - coming soon!

Python-SRB bindings binary packages built with Python 2.3:       


Installation

Shared Applications can be installed by the typical AccessGrid Shared Applications installation method:
  • Download then add the appropriate .agpkg3 to the Venue data store.
  • Double click (or right click and choose 'Open') the newly uploaded files.
The Shared Application should then be installed.


Applications

''Global'' Applications
With a tight integration between the AG and SRB we bring new and exciting possibilities for the Access Grid. The seamless access to data-grids and the large data sets within them enables the development of "global" applications. Many innovative applications are possible.

They could include:

  • Google Maps
  • Geographic Information System (GIS)
  • Astronomical
  • Medical

 

References

[1]    The Access Grid Project @ UQ VisLab
        http://www.vislab.uq.edu.au/research/accessgrid/index.html

[2]    Access Grid official site
        http://www.accessgrid.org/

[3]    The SRB Project @ UQ VisLab
        http://www.vislab.uq.edu.au/research/SRB/index.html

[4]    SDSC Storage Resource Broker official site
        http://www.sdsc.edu/srb/index.php/Main_Page

[5]    SRB zones @ UQ VisLab
        http://www.vislab.uq.edu.au/research/SRB/zone.html

[6]    Shared Scientific Applications for the Access Grid @ UQ VisLab
        http://www.vislab.uq.edu.au/research/accessgrid/shared_app/index.html

[7]    SRB for large data handling in the Access Grid, 2007 AG Retreat
       
http://www.accessgrid.org/files/retreat07/presentations/willing_srb.pdf


Acknowledgements

Created by Eric LIAO <e.liao at uq.edu.au>
Funding through UQVislab and QCIF.


- Last updated August 3, 2007 -