[[TOC]]

YAFITS

Stands for Yet Another FITS viewer

Introduction

YAFITS is a distributed application to visualize and to study the content of a FITS file (with IMAGE extension) located on a remote server. On the client's side all the activity takes place in a WEB browser. Material is provided to traverse the server's file space dedicated to the FITS files service or to explore that space with SQL queries. Once a file is found and selected it can be visualized and studied. Inspired from tools like ds9 or goview ( part of GILDAS software). It adds the "remote" dimension in that sense that absolutely no software installation on the final user's machine is required. It simply requires a reasonably modern browser.

Related links.

• A integrated documentation of the YAFITS project can be browsed at https://yafits.obspm.fr/
• YAFITS has its own DOI,
• YAFITS is one of the main components of the ARTEMIX project --- http://artemix.obspm.fr

Requirements

Have Python installed ( python 3.7 works ) with astropy and sqlite3 packages.

Have the applications docker (https://docs.docker.com/install/) and docker-compose (https://docs.docker.com/compose/install/) available on the host where you plan to deploy the project.

If your operating system is Linux please check that your docker's installation has been done completely and in particular that the post-installation steps have not been neglected. It's mandatory if you want to manage Docker as a non-root user, see https://docs.docker.com/install/linux/linux-postinstall/

Even if it's a not a requirement, having the Docker management and monitoring tool portainer (https://www.portainer.io/) installed is indiscutably a bonus.

Design considerations

Docker's choice

After an initial phase where the project was built as a set of applications running on a (possibly virtual ) machine in a classical way, it's been decided to package those applications in a Docker container mostly to ease the distribution and the installation of the product.

Architecture

The facility is architectured as a composition of four Docker images :

• yafitsv : an HTTP server (NodeJS) that reacts to the user's requests such that navigating in the FITS collection or visually exploring the content of one selected FITS file.
• yafitss : an HTTP server (Python+Bottle) which actually performs all the hard work with FITS files ( browsing, loading in memory, extract parts to display, etc...) and is callable via a serie of REST APIs
• spectro : an HTTP server (NodeJS) that searches spectroscopy data in a mongo database and sends results as JSON files
• seed : a container that simply copies spectroscopic data in the mongo container. It is only used in case spectroscopic data are updated
• mongo : a mongodb database

The relations between the user's WEB browser and the containers can be depicted as follows :

graph LR

A{web browser} --> |queries| B{yafitsv}  
B -->|queries| C{yafitss}
B -->|queries| D{spectro}
D -->|search| F{mongo}
E{seed} -->|loads data into| F
E -->|copies| H{spectro data}
C -->|reads|G{files}

subgraph User
A
end

subgraph Docker containers
B & C & D & E & F
end

subgraph File system
G & H
end

Getting the source code

Clone the project.

cd <..>
git clone https://gitlab.obspm.fr/artemix/yafits.git

Before building the composition of Docker images

The steps below are absolutely mandatory for a successful build and execution of the images. Do not skip them.

Adapt the configuration file

yafits is designed to be easily shared and deployed on a platform where the docker and docker-compose applications are available after having defined a couple of environment variables. Prior to any other action :

• Set the project's directory as the current directory :

cd <..>/yafits

• Copy the configuration file template yafits.bashrc.dist to .yafits.bashrc (still in the same directory) and edit its content accordingly with your local environment.

• Append a call to .yafits.bashrc in your $HOME/.bashrc

• If needed (e.g. the first time) re-execute $HOME/.bashrc or alternatively logout and re login.

At the end of this process an examination of the YAFITS specific environment variables should look like this :

(base) -bash-4.2$ printenv | grep YAFITS
YAFITS_OBJDIR=/tmp
YAFITS_SQLITE_DIR=/datartemix/ALMA/FITS
YAFITSS_MAXIDLE=300
YAFITSV_HOST=juliette.obspm.fr
YAFITSS_HOST=juliette.obspm.fr
YAFITS_LOGDIR=/tmp
YAFITS_FITSDIR=/datartemix/ALMA/FITS
YAFITS_PNGDIR=/tmp
YAFITSS_PORT=4252
YAFITSV_URLROOT=artemix.obspm.fr/fits
YAFITS_IMGDIR=/datartemix/ALMA/FITS/IMG
YAFITS_FITSDIR2=/datartemix2/ALMA/FITS
YAFITS_SAMPDIR=/tmp
YAFITS_SMOOTHDIR=/datartemix/ALMA/FITS/SMOOTH
YAFITSV_TTL_SMOOTH_IMG=3600
YAFITSV_PORT=3022
YAFITSV_USESAMP=true
(base) -bash-4.2$ 

When you perform an upate ( a git pull ) do not forget to check that your local configuration file .yafits.bashrc needs to be udpated given that the distributed file yafits.bashrc.dist may have changed.

Customize the Welcome message file

Copy the file <..>/yafits/yafitsv/public/Welcome.html.dist to <..>/yafits/yafitsv/public/Welcome.htmland possibly adapt its content to your specific service.

Preparing the SQL database.

This section is very important in that it describes in details the process of creation and maintenance of the SQL database used by the query based selection tool (see). It's the duty of the YAFITS administrator to understand its principles and follows the creation/maintenance steps.

SQLite.

Why SQLite ?

Considering the very simple nature of the envisioned queries SQLite appears to be a well suited solution. It requires no specific server and a database uses a single file for its content. Beyond its simplicity of utilization the fact that SQLite has a JSON extension allowing to create, parse and query JSON content has been the other major reason to choose it. This feature is the perfect solution to fill a table row with standard fields such as path or size plus a FITS header translated in a JSON in a obvious way for one given FITS file.

The SQLite database for YAFITS

For the specific purpose of YAFITS, the following conventions and structure have been adopted :

• The database is named fitsinfos.db and must reside in a directory defined by the environment variable YAFITS_SQLITE_DIR (see),
• The database has a unique table fitsinfos,
• The table fitsinfo has the following fields :

Creating/Updating the YAFITS SQLite database

The brute force approach consists in traversing the FITS files hierarchy of directories and for each FITS file found to create a row in the table fitsinfos with three fields : Path (string ), Size ( integer ) and Header ( JSON representation the FITS header considered as a dictionary ).

This work is done by the script fs2sqlite.py located in the yafits project toplevel directory. In order to run correctly the script requires that the environment is defined (see). In particular it uses two environment variables YAFITS_FITSDIR (FITS files toplevel directory) and YAFITS_SQLITE_DIR (SQLite database directory).

• Create. Run the script once the first time you install YAFITS before starting it. Depending on the number of FITS files that are served the script's execution can take a long time.

cd <yafits-project-directory>
./fs2sqlite.py

Note that a file fs2sqlite.log in that same directory keeps a trace of the script's activity.

• Update. The need of updating the database or the frequency of its updates depends on the site which can serve a completely frozen collection of FITS files or a collection that varies more of less often (daily, weekly, randomly,...).

A solution maybe to run it routinely under the control of the cron daemon. Here is an example extracted from the crontab of the process executing YAFITS once a day at night:

6 0 * * *  . $HOME/.bashrc; cd $HOME/yafits; ./fs2sqlite.py 2>&1 cron.log

Building and running.

The YAFITS application is designed and built as a composition of Docker images. The tool docker-compose is used both for the building and running steps.

An integrated tool : the yafits script

This is the standard way to install and run the application.

A bash script called yafits is located in the application toplevel directory. It aims at hiding the docker commands from the user. It provides the following options :

• help : Show this help
• build : Build the application
• buildrun : Build and run the application
• run : Run the application without building it
• updatespectro : Will build and run yafits, spectroscopy database will be updated

Manually building the images composition

Of course it is possible to execute the process manually too. Firstly, you need to copy one docker compose configuration file from the docker-composers directory into the yafits directory itself. There are two yaml files : docker-compose-spectroupdate.yml which is to be chosen when you want to update the content of the spectroscopic database and docker-compose.yml in all the other cases.

Once the yaml file has been copied, launch the build command : it'll take some time due to the Debian base and Anaconda installations (at the first build)

docker-compose build

Running the images compositions in containers

docker-compose up -d

Two containers should be running ready to process your requests.

You can start your work, e.g. http://${YAFITSV_HOST}:${YAFITSV:PORT}/browse ( the two env variables are the ones you have defined in the configuration script. You will very likely use their value instead of their names as above. E.g. http://juliette.obspm.fr:3022/browse )

Note that the build step can be omitted if for some reason the containers had to be restarted.

How FITS files can be selected with YAFITS

The link

The URL that directs YAFITS to open a FITS file for its vizualization has the following form:

http://${YAFITSV_HOST}:${YAFITSV:PORT}/visit/?relFITSFilePath=<path-to-FITS-file>

where <path-to-FITS-file> is the path to the FITS file in the server's file system relative to the top level directory of the FITS files collection. Example :

http://some.machine.1234/visit/?relFITSFilePath=2012/2012.1.00185.S/science_goal.uid___A002_X5a9a13_X724/group.uid___A002_X5a9a13_X725/member.uid___A002_X5a9a13_X726/product/HCO+_line.fits

Most of the time this path is unknown by the user who needs ways to explore the FITS files collection to obtain the path to the desired file. Two methods exposed below are available; one allows to traverse the FITS files collection and the other one look for FITS files meeting some criteria expressed in an SQL SELECT query.

Browsing available files

This is a simple (sub)directory un/folding mechanism allowing to traverse the hierarchy of FITS files on the server's dedicated space until you find the file of interest. The file names are displayed as links; clicking one of them directs YAFITS to work on the corresponding file.

Open your browser at http://${YAFITSV_HOST}:${YAFITSV:PORT}/browse

Searching in available files

This is an SQL query interface allowing to retrieve informations about FITS files meeting the criteria expressed in the query. Amongst the informations a query can return there is one fitsinfos.Path which contains the path to the FITS file in the server's dedicated space. Should this field be specified in the query ( SELECT fitsinfos.Path [,...] FROM fitsinfos WHERE ...) that the result will contain column of links to the corresponding files. Clicking on one of these links will trigger the YAFITS activity on the corresponding file.

Open your browser at http://${YAFITSV_HOST}:${YAFITSV:PORT}/query

Attention the SQL query interface operates at the express condition the SQL database of informations about the FITS files exists and is hopefully up to date. It's the YAFITS administrator's responsibility to create this database out of the collection of FITS files and keep it updated. Reading this section is absolutely mandatory for a YAFITS administrator when s/he installs YAFITS on a server.

Technical details

The informations below are given for the curious of technical details. They may be useful in case of problems, but normally reading this section is not required.

Account considerations

Instead of leaving the default account (root, uid==0) running the code, one uses a specific account created during the image build. Its name is partemix with uid==1000, gid==1001.

Data files organization

Data files are on the one hand the FITS files (readonly) and on the other hand the PNG files (read/write). The software expects to find these files respectively in a FITS directory and in a PNG directory both subdirectories of a same toplevel directory. The table below depicts the situation of the example of the docker runcommand ( see above) where the host directory /datartemix/datartemix with its two subdirectories FITS and PNG is mounted on the container's directory /home/partemix/dataroot .

Exemple :

Files considerations.

A characteristic of the container is that all the data that it accesses to are residing on the host filesystem. The container has NO data on his own; in other words all the data are persistent. A particular attention must therefore be paid to the protections defined on the data and the permissions to be set up for the applications executed by the container have the necessary access to them.

The informations given below are pertinent with the assumption that the image has been built with the Dockerfile left unchanged.

FITS files.

The FITS files which are the raw material of yafits are accessed to as a readonly resource of the host filesystem. The most permissive way to ensure their readability by the container is to make them readable by anyone (chmod a+r). A less permissive approach is to allow them to be read by the group gid==1001 and to put their owner into that group, possibly at the cost of creating such a group. The most restrictive access would be to have a user with uid==1000 ( which is not always possible ) and the FITS files belonging to that user.

PNG files.

The PNG files are temporary files generated on the fly for the visualization of FITS files contents. Consequently the directory used as a repository for these files must be writable by (uid==1000,gid==1001). As for the FITS files one can choose between the three options 1) writable by anyone 2) writable by group==1001 with the PNG directory's owner in that group and 3) the PNG directory belonging to uid==1000 and writable by the owner.

LOG files.

Both servers yafitsv and yafitss generates log files in a directory of the host which must be writable by uid=1000,gid=1001. Again one can use one of the strategies presented above for the PNG and FITS files.

Related links.

• A integrated documentation of the YAFITS project can be browsed at https://yafits.obspm.fr/
• YAFITS has its own DOI,
• YAFITS is one of the main components of the ARTEMIX project --- http://artemix.obspm.fr