Dealing with Large Files

Large astronomical FITS data files present a challenge for browser-based image display, in two ways:

The nightmare scenario is to run out of browser memory after taking a long time to download the file!

One possibility for dealing with large files is to download a small representation file for display and quick-look analysis, while supporting detailed analysis and image extraction on the original parent file. In JS9, you can:

These schemes are described below.

Automatic Generation of a FITS Representation File

Consider a 12000 x 12000 Hubble Deep Field image. Even using gzip compression, this 300Mb file could take several minutes to download and would require more than 500Mb of memory to display. But you can bin this file by a factor of 24 into a gzip'ed 512 x 512 FITS image with file size of approximately 500Kb. This FITS representation file would download quickly and give a good overview of the entire data set. You can perform quick-look analysis on this file using JS9 plugins. But if you also could re-extract a section from the original file at a higher resolution, you would, in essence, be panning around the original image without the transmission and memory overhead associated with processing the whole file.

JS9 supports FITS representation file generation and display, as well as extraction of binned image sections from the original parent file, using the fits2fits scheme. To utilize it at your site, first configure a Node.js-based Server-side Helper using the standard GNU configure/make setup:

  # where to find cfitsio and install binaries, what sort of helper to build
  ./configure  --with-cfitsio=/usr/local --prefix=$HOME --with-helper=nodejs
  make
  make install

Once the js9Helper is running, configure fits2fits by setting the following global parameters in your js9prefs.js file:

The server-side helper must support temporary user work directories in which to store the FITS representation files. This is accomplished by setting the globalOpts.workDir property in js9prefs.js to a top-level directory which will contain the individual work sub-directories for each JS9 connection. The workDir path should either be an absolute path, or one that is relative to the JS9 helper directory.

Optionally, you probably will want to set the globalOpts.workDirQuota property (current default value of 50Mb) to limit the amount of space that can be utilized by each user. Note that the workDirQuota is not a hard quota: the js9helper only checks whether the quota has been exceeded before extracting the next section. This means that users can always extract one section that will bring the total size above the quota. Thus, if the quota is 10Mb and the work directory is empty, imsection will extract a 50Mb file. But it will not extract any more files until already-loaded files have been closed in order to bring the total under the quota. The globalOpts.fits2fits property controls whether a FITS representation file is automatically generated. It accepts the following values:

Of course, this property is only checked if JS9 is connected to a server-side helper which has a working directory configured, as described above.

The current default for fits2fits is "size>100", which means that representation files will be generated if the original FITS file has a size greater than 100 Mb. The globalOpts.image and globalOpts.table objects specify the image dimensions and bin factor used to generate the FITS representation file:

Binning is applied after the image section is extracted or the table is binned into an image. Thus, dimensions of 4096 x 4096 and bin 4 will produce a 1024 x 1024 image.

When fit2fits is set up in this way, JS9 will ask the helper to check the file size when loading a FITS URL. If the check returns false, the original FITS file will be loaded internally into JS9. If true, the JS9 helper will extract a representation image (i.e, an image section) from the parent file, store it in the temporary work directory, and then tell JS9 to display the representation file.

The smaller in-memory representation file will be used when performing browser-based analysis such as WCS reprojection, 3D plots, and imexam. But the representation file knows its parent, and will send the parent filename to run server-side analysis tasks. Thus JS9 performs quick-look analysis on the smaller file and more detailed server-side analysis on the original parent file.

In this context, perhaps the most important server-side task is the Bin/Filter/Section plugin available in the View menu. This plugin allows you to extract sections from FITS binary tables and images, with options to choose the image center, image dimensions, and bin factor. For tables, the plugin also allows you to filter rows of the table by means of arithmetic and boolean operations on the table columns.

The Bin/Filter/Section plugin works with both FITS representation files and with in-memory FITS files. If the displayed FITS file is a representation file (i.e., it has a parent file, a JS9 helper is connected, etc.), the plugin will ask the JS9 helper to extract the new image section from the parent. Otherwise, it extracts the section from the in-memory FITS file. This behavior is meant to be transparent to the user. See the Bin/Filter/Section plugin for more information.

NB: Unfortunately, the fits2fits scheme is not available with FITS files that are drag/dropped onto JS9 or files that are opened through the file menu dialog box. Both of these methods call for the browser to load the whole file directly and pass it to JS9 for display. There is some discussion within the browser development community of supporting on-the-fly processing, but nothing definite is planned.

Manual Specification of a FITS Parent File

Assuming the JS9 helper is set up, but fits2fits is set to "never", you can load a small FITS file using JS9.Load() and manually specify a parentFile property in the options object:

    JS9.Load("js9/fits/f814w_bin24.fits.gz", {colormap:"cool", parentFile: "fits/f814w.fits.gz"})
One important note: the pathname of the URL to load is (as usual) relative to the web page, while the pathname of the parentFile must be relative to the JS9 install directory (i.e., where the JS9 helper was started). In the above example, the web page is one level up from the JS9 install directory. Once the parentFile has been specified, the in-memory file should act as a representation file.

Of course, the two files need to be tied to one another spatially. For server-side data analysis, having agreement between WCS parameters is sufficient. In order to use the Bin/Filter/Section plugin, however, the representation file also must be connected to the parent file using IRAF LTM/LTV keywords, so that the physical coordinates of the representation file point to the file coordinates of the parent. The js9helper program will generate a FITS image section with the correct keywords. The syntax is:

    js9helper -i parentFile imsection repFile section filter
where section can be: and filter specifies a column filter for binary tables. For example:
    js9helper -i f814w.fits.gz imsection f814_bin24.fits "12000@0,12000@0,24"
was used to generate the binned representation image in the JS9 Large demo.

Automatic Generation of the PNG Representation File

NB: The use of PNG representation files has been deprecated in favor of FITS representation files described above.

Generating a representation file from an X-ray binary table for download is one solution to the large file problem. It does require setting up server-side parent file processing (including temporary storage of image section files). An alternative that does not require a server-side helper is to create a special PNG representation file explicitly tied to the original event data. Once downloaded into JS9, the image can be reconstructed from this file for display. The file maintains information about the original FITS binary table for use in remote analysis. The idea is that you create a representation file from the original FITS binary table and use that file for display in JS9, while supporting server-side analysis on the original FITS event file.

Display of a representation file is fully integrated into JS9:

Using this representation file, image data values are available to the client JavaScript code without having to download the much larger binary table. This means that the contrast/bias conversion can be performed, image positions/values displayed, etc. using standard browser capabilities and relatively straight-forward array manipulation code. Browser-based image plugins can process the image data, and JS9 can communicate with a server to provide more sophisticated FITS-based analysis support.

Note that JS9 is able to distinguish between PNG representation files and ordinary PNG files and will correctly load either type.

The PNG Representation File

The PNG representation file contains the following information:

Generating the PNG file using tpos and fits2png

[NB: A maximum image size is enforced by Apple on Safari mobile devices. (See https://discussions.apple.com/thread/4975106 for a discussion.) If you want your JS9 PNG representation files to display on iPhones and iPads, they should be restricted to a size of less than approximately 1900x1900, depending on the size of the FITS header. So a PNG file of size 1024x1024 should be fine, while a file of size 2048x2048 certainly will be too large. (April 9, 2015)] A PNG representation file can be created either statically or dynamically at display time as needed. The low-level tpos program and high-level fits2png script facilitate this conversion:

    tpos [ifits] [opng]
    switches:
    -b bl       # block factor for binary tables (cfitsio only)
    -e list     # list of event extensions to look for (cfitsio only)
    -f filter   # filter for binary tables (cfitsio only)
    -s sect     # section string for binary tables (cfitsio only)
or
    fits2png [switches] [ifits] [opng]
    switches:
    -abs	# convert relative path to absolute path for storage in PNG
    -gzip       # assume the file is in gzip format, regardless of its extension
    -dim	# dimension for PNG image (use to calculate block factor)
    -js9	# set dim, odir, repl parameters for automatic JS9  conversion
    -odir	# specify output directory (default: from opng path)
    -repl	# replace existing PNG? (default: true)

The tpos program can be linked against either the cfitsio library or the funtools library, depending on how you configured your JS9 build. (Note that funtools is deprecated. Please use cfitsio where possible.)

The fits2png switches are meant to provide flexibility when running tpos in batch mode. The fits2png script also is used to generate PNG files on the fly (i.e., when offering FITS files on your web page directly instead of the PNG representation.) Note that if you have linked against cfitsio, you must ensure that the HEASOFT environment is set up properly when running fits2png.

By default, tpos and fits2png will convert a FITS image or binary table to a PNG representation file having the same image dimensions:

    tpos m13.fits m13.png
However, current browsers have a working limit to the image size they can reasonably handle: 1024x1024 is recommended, although 2048x2048 works as well; anything larger can be subject to performance degradation. For maximum flexibility, you can use section syntax to limit the size of the generated PNG file. For example:
    tpos -s "1024@4027,1024@3944" -b 2 casa.fits casa.png    # cfitsio
or:
    tpos 'casa.fits[1024@4027,1024@3944,2]' casa.png         # funtools
Here, the PNG file is generated by extracting a 1024x1024 image centered on a specific point (4027,3944) and then blocking it by a factor of two to create a final 512x512 image.

For automatic processing with fits2png, specifying the -dims switch will tell fits2png to calculate the block factor in such a way as to generate an image of the specified size:

    fits2png -dims 512 512 casa.fits casa.png
This will block the 8192x8192 Chandra ACIS data by a factor of 16 to make a 512x512 image centered on the the point (4096,4096).

In addition to the image, the (generated) FITS image header and the FITS pathname are stored in the PNG file for use by both the browser and the server-side helper programs. The image header is used to provide WCS information for individual pixels as the mouse is moved across the image. It also is used to provide region information in WCS format (i.e. independent of block factor). The FITS pathname is used to locate the original FITS file in order to execute analysis tasks.

In order to run analysis tasks, the FITS data must be accessible to the server-side helper program. Therefore, it seems preferable to store an absolute pathname in the PNG file, but more experience is needed on this score. You can specify the absolute pathname on the tpos command line, or else specify the -abs switch on fits2png and it will be determined by the script.

If you store a relative path in the PNG file, note that this path will be utilized by the server-side JS9 helper to locate the FITS file. This means that if the helper is not started in the same directory that tpos (fits2png) was run, the relative path will not be correct. In this case, you can set the JS9.analOpts.dataDir parameter in the js9prefs.js file to a path that will be used instead of the relative path.

You also must decide where to store the PNG files, e.g. with the FITS data or in a separate directory. Clearly, this will depend on how your archive is set up and where you have write access. By default, PNG files will simply be stored in the path specified on the tpos command line. You can use the -odir switch on the fits2png script to specify a generic location:

    tpos ./png ./fits/casa.fits ./png/casa.png    # store in png subdirectory
    tpos -odir ./png ./fits/casa.fits casa.png    # same

The -odir switch can be used to specify a session-based directory in which to store PNG files. This facility could be used, for example, in a case where each user has files stored in her or her own directory, which might be removed at the end of the session. To facilitate session-based directory specification, a standard HTML cookie is used to pass the session information between client and server as follows:

Note that the name of the cookie is arbitrary, but is case sensitive.

Finally, the special value of @IFILE@ tells fits2png to store the PNG file in the same directory as the FITS file:

    tpos -odir @IFILE@ .../casa.fits casa.png     # store PNG with FITS file
Again, these switches are meant to provide initial flexibility while we gain more experience with community needs. Please let us know how to improve this part of the system.

Regardless of where the PNG files are stored, you must adhere to the W3C "same-origin" security policy, i.e. the PNG data files must be hosted on the same web server as the JS9 code. This means that your PNG files must either be stored in the web site directory tree, or you must make a link within the web site tree to the top level PNG directory. Also note that Google Chrome's implementation of this policy prevents you from using JS9 with a file:// URL. (On Linux and Macs, you can start Chrome with the --allow-file-access-from-files switch to work around this restriction.)

Loading a PNG File From Your Web Page

Once you have generated your PNG files, you can make them available on your JS9 web page by creating URLs that call the JS9.Load() routine:

    href='javascript:JS9.Load(filename, opts);'
where filename is the file to load and opts is an object containing params to be merged with the default JS9.imageOpts object.

For example:

    href='javascript:JS9.Load("png/casa.png", {scale: "log"});'
Clicking on this URL will cause the casa.png file (here, stored in the png directory) to be loaded into JS9, using the log scale as a default. If the JS9 server-side helper is running, it will be alerted about this new file so that it can run analysis, etc.

See JS9 Public API for complete documentation about the JS9.Load() routine.

Run-time Conversion of FITS to PNG

If you don't generate PNG files beforehand, you can still specify a FITS file as the first argument to JS9.Load() and have the JS9 server-side helper run fits2png at run-time.. To do this, you set the fits2png property in js9prefs.js:

{
  "globalOpts": {"fits2png": true ...}
}
This tells JS9 to convert FITS files instead of downloading them directly. Then you can use JS9.Load() as usual:
    href='javascript:JS9.Load("fits/3c58.fits");'
The default command for this conversion is contained in the js9Helper.js file:
    fits2png -js9 -odir ./png
The -js9 switch will calculate a block factor to generate a 512x512 PNG image. It will not replace an existing PNG file, if it exists. Note that the output directory is set explicitly to be the png subdirectory of the sample installation. You can change this command to suite your site needs.
Last updated: July 10, 2017