SAGE web browser

Overview

Until now, the only way to use an ordinary web browser in the Scalable Adaptiver Graphics Environment (SAGE) has been to run it on a workstation who's screen output is captured (typically using VNC or Blackmagic capture card) and streamed to the tiled display. This has been necessary because there has been no browser modified to render directly to an offscreen buffer suitable for SAGE. The technique of somehow streaming a desktop with browser to a tiled display is reasonably convenient, however the displayed component still has only the resolution of the physical display of the workstation on which it is being run. We would much prefer to run a browser at the much higher resolution available on the tiled display, rather than be limited to the resolution of the desktop.

We show here some images demonstrating a proof of concept of a technique to run a web browser at high resolution within a tiled display utilising SAGE.



Professors Tom Defanti (Calit2, EVL) and Bernard Pailthorpe (UQVislab) viewing an unscaled 5760x4000 Google map of Brisbane	Detail of the unscaled 5760x4000 Google map of Brisbane


Using a 3D wireless mouse to interact with a 5760x4000 genome browser	Detail of 5760x4000 genome browser. Many of the genome features are themselves links which can be followed by clicking with the 3D mouse.


The technique is not restricted to web browsers. Here, an unscaled 3840x2400 instance of Paraview is running alongside the genome browser.	Without scaling, the large rendering windows result in quite small looking menus and other features. Notice the yellow SAGE pointer for application interaction.

Method

The pictures above demonstrate a proof of concept stage of development. At the moment, while looking into various optimisations, the individual steps involved in this technique are kept separate (recall Donald Knuth's premature optimization is the root of all evil conjecture); these steps are detailed below.

The technique entails running the target application inside a large X virtual frame buffer, the contents of which are delivered to SAGE using the screamer (SCReen strEAMER) application. As well as streaming the contents of the frame buffer, screamer also synthesizes pointer events (click, drags) inside the frame buffer environment based on the activity of the SAGE pointer. Since an X virtual frame buffer has no event mechanism of its own, it is screamer's ability to synthesize events which provides interactivity with whatever application is running in the X virtual frame buffer environment.

The overall steps are as follows:

create an X virtual fame buffer of whatever size is required e.g.
Xvfb :1 -screen 0 5760x4000x24
start a window manager for the Xvfb instance e.g.
fvwm2 -display :1
(in fact, whether to run a window manager is questionable. There's still some tuning to do synthesizing the Xvfb events from the SAGE pointer events so the target application may run better without the window manager)
run the target application inside the XVfb instance e.g.
seamonkey --display=:1
run the screen streaming screamer software with:
screamer :1
or
screamer :1 -show_original
(no DXT compression in this case)

Of course, the screamer appplication needs to be available first. To build it, first download the latest screamer source tarball. Unpack it somewhere and then install the included DXT shared library before building screamer itself. Typically, the following sequence of commands will do everything required (change version numbers as necessary):
         tar xvf screamer-0.2.3.tar.gz
         cd screamer-0.2.3
         make install-lib
         make install

Issues

Everything seems to run very slowly
The sluggishness is related to the size of the virtual frame buffer. Using a 1920x1080 frame buffer results in reponsiveness similar to a regular desktop. As larger frame buffer sizes are employed, the reponsiveness is correspondingly reduced and there are two main causes for this. Firstly, in applications like a web browser, the downloading of rich content e.g. a Google map, takes much longer for a large viewport than for a regular vewport. There's just a lot more data to download and render. We really have no control about this aspect of the problem. Secondly, the contents of the virtual frame buffer need to be regularly transferred to the SAGE buffer. The time to complete this transfer is also ralated to the buffer size. For a 1920x1080 buffer, we achieve about 30fps; for a 5760x4000 buffer, we achieve about 3fps with DXT compression enabled (about 1fps without). Since this data transfer is performed by the screamer application, it is an area ripe for further work. Some optimisations which immediately spring to mind are:

separate the event processing from the data transfer loop (we currently look for SAGE event messages only at the end of each data transfer)
only transfer the frame buffer data if there has been some change that makes it worthwhile to transfer
only transfer parts of the buffer that have changed
parallelize data transfer - we have started on this but there is an XGetImage() call on the virtual fame buffer root window which is the greatest bottleneck at the moment. Even grabbing parts of the root window for transfer in separate threads only works if the multiple XGetImage() calls are consecutive i.e. no faster than a single grab of the whole window
Bypass XGetImage to fill the SAGE buffer directly from the underlying root window data structure
If Xvfb is invoked with the -fbdir /some/dir argument, then the root window is available at that point in the file system in XWD format. Is it faster to retrieve the window data from there?

What about keyboard input? We can now click on links etc., but how do we input a new url into the browser?
Keyboard events aren't yet exposed to SAGE appplications - now that there is a use case, perhaps they will be soon.

My target application (paraview) doesn't support the "--display" option
Set the environment DISPLAY variable e.g.
DISPLAY=:1 paraview

Contact

Please send any comments, corrections, advice etc., to Chris Willing <c.willing _at_ uq.edu.au>