Some existing data-grid projects:
- Biomedical Informatics Research Network (BIRN)
- National Virtual Observatory (NVO)
- Real-time Observatories, Applications, and Data management Network (ROADNet)
These projects provide access to large amounts of distributed data (reaching terabytes or petabytes) for scientific and engineering applications. The size of data collections can only grow as new experiments and computational systems require higher capacity storage for results and simulation of large systems. The Access Grid (AG) [1], [2] is a collaborative research environment with potential for distributed data and visualization across networks and platforms. However currently AG is used predominantly for videoconferencing with PowerPoint presentations. To extend the Access Grid beyond simple meeting scenarios and towards true collaborative working, we need to enable the incorporation of large data sets and collections into the AG environment.
The Project
We plan to achieve this goal by augmenting the collaborative capabilities offered by Access Grid with the Storage Resource Broker (SRB) [3], [4]. The SRB is a middle-ware data-grid technology that provides a uniform interface to large heterogeneous data collections. With it powering the majority of data-grid projects around the world, augmenting AG with SRB allows potential access to extensive amount of existing distributed data (Figure 1).
Figure 1. Merging of the Access Grid with Storage Resource Broker
There will be several stages for the integration of the SRB technology
into the Access Grid: shared application, data store service, and
search service. Each are described in detail below.
Shared Application
The first stage will be the development of a file-system manager as a
shared application for use in the Access Grid. The client initiated
shared application links the Access Grid with the SRB provided data.
The initial shared application will essentially provide shared data
provided by one user to one or more users. With the addition of
zoneSRB, these shared data could be linked together through federation
of SRB servers and/or zones [6]. See Figure 2 for the overall system
architecture.
Figure 2.
Shared Application System Architecture
Data Store
Service
The next stage is a more seamless
interface between AG and SRB - the
integration of SRB technology into the AG Data Store service. This
makes heterogeneous and distributed data available to users as a
hierarchical file-system in the 'Data' section of the AG Venue Client.
Shared applications within the venue could then easily access the data
provided by SRB.
Search Service
The ability to quickly and easily search through large data sets and
collections is highly desirable. In the future we plan to investigate
innovative search methods that are suitable to distributed data in
data-grids. These could be SQL based or natural language based queries.
Development Log
- Since stating in August 2006, I have spent time becoming familiar with the AG toolkit and the Argonne National Laboratory's (ANL) shared application API.
- As a learning exercise, I ported Shared Rasmol and Shared GRASS [7] to the new AG Version 3.0. These ports promote user migration to the updated version of Access Grid with improved robustness, performance, and interoperability.
- I have also spent time understanding the SRB implementation and investigated methods to integrate it as the virtual file-system of the AG.
- I have written a prototype SRB Browser tool independent of slashManager in wxPython for Linux. It can connect to, upload, and download files from the SRB managed data (see Figure 3 below).
Figure 3. Prototype SRB Browser
Interface
- I modified the existing Python-SRB bindings to allow sharing of data through ticket-based access.
- The SRB Browser tool is now a shared application within the AccessGrid environment (see Figure 4) and AG users can use this to manage/share SRB data.
Figure 4. Updated SRB Browser
Interface in the AG
- Improving the SRB Browser GUI's usability (Figure 5):
- Hierachical navigation with tree view of personal and shared
SRB space.
- Right-click menu functionality i.e. common I/O operations and
properties.
- Status bar showing relevant information e.g. number of data
selected and total size.
- Further improvements to SRB Browser functionality (Figure 6):
- Able to upload directories and download collections to personal SRB space.
- Able to upload and download collections to be shared across the
AG.
- Easy renaming, Tango icons and progress bar improves GUI usability.
- agSRB Caveats:
- When you share or remove items from the shared space, the tree-control selection reverts to 'All Shared Data'.
- Currently can only navigate in shared space using the
tree-control.
Figure 6. agSRB Browser (current)
- The Python-SRB Bindings have been further modified with SRB error msg handling and renaming of files/collections.
- A presentation about this project was done by Christoph Willing at the 2007 Access Grid Retreat in Chicago [7].
- The Python-SRB bindings can now be built on Linux and Mac OSX
with gcc, and Windows with Visual Studio. It is a relatively simple and
documented process making it easy for developers to further
extend/improve the bindings.
Desired Functionality
CORE (Completed) |
EXTENDED (In development) |
|
|
Downloads
Instructions for building from source
Quick guide for testing the bindings
for AG Toolkit 3.02 (or later) and
SRB Version 3.4.2
|
|
Python-SRB bindings source code:
Linux
and OS X source distribution (.tar.gz)
Windows source distribution (.zip) Latest code from SVN repository - coming soon! |
|
Python-SRB bindings binary
packages built with Python 2.3:
|
Installation
Shared Applications can be installed by the typical AccessGrid Shared Applications installation method:- Download then add the appropriate .agpkg3 to the Venue data store.
- Double click (or right click and choose 'Open') the newly uploaded files.
Applications
''Global'' Applications
With a tight integration between the AG and SRB we bring new and
exciting possibilities for the Access Grid. The seamless access to
data-grids and the large data sets within them enables the development
of "global" applications. Many innovative applications are possible.
They could include:
- Google Maps
- Geographic Information System (GIS)
- Astronomical
- Medical
References
[1] The Access Grid Project @ UQ VisLab
http://www.vislab.uq.edu.au/research/accessgrid/index.html
[2] Access Grid official site
http://www.accessgrid.org/
[3] The SRB Project @ UQ VisLab
http://www.vislab.uq.edu.au/research/SRB/index.html
[4] SDSC Storage Resource Broker official site
http://www.sdsc.edu/srb/index.php/Main_Page
[5] SRB zones @ UQ VisLab
http://www.vislab.uq.edu.au/research/SRB/zone.html
[6] Shared Scientific Applications for the
Access Grid @ UQ VisLab
http://www.vislab.uq.edu.au/research/accessgrid/shared_app/index.html
[7] SRB for large data handling in the Access
Grid, 2007 AG Retreat
http://www.accessgrid.org/files/retreat07/presentations/willing_srb.pdf
Acknowledgements
Created by Eric LIAO <e.liao at uq.edu.au>Funding through UQVislab and QCIF.
- Last updated August 3, 2007 -