Changes

Page history
Update Technical informations authored by Moreau Nicolas's avatar Moreau Nicolas
Hide whitespace changes
Inline Side-by-side
Technical-informations.md
View page @ 022163d6
\[\[_TOC_\]\] \[\[_TOC_\]\]
This page is intended to provide technical informations hopefully useful for anybody in charge of running the yafits containers. The commands indicated below were run under SL Linux. Feel free to introduce examples of commands used on different flavours of Linux or on Mac OSX where it's relevant. This page is intended to provide technical informations hopefully useful for anybody in charge of running the yafits containers. The commands indicated below were run under SL Linux. Feel free to introduce examples of commands used on different flavours of Linux or on Mac OSX where it's relevant.
# Prerequisites # Prerequisites
You have cloned/downloaded the `yafits` project and you can run the `docker-compose` command You have cloned/downloaded the `yafits` project and you can run the `docker-compose` command
# Starting docker daemon (possibly) # Starting docker daemon (possibly)
### Manually on demand ### Manually on demand
Before trying to launch any container, it's advised to ensure that the docker daemon is running. E.g. it may happen that the daemon has not been registered to be launched during a (re)boot sequence. Before trying to launch any container, it's advised to ensure that the docker daemon is running. E.g. it may happen that the daemon has not been registered to be launched during a (re)boot sequence.
```plaintext ```plaintext
ps aux|grep dockerd ps aux|grep dockerd
``` ```
If the above command returns nothing then start a session on an account belonging to sudoers and start the daemon. Example : If the above command returns nothing then start a session on an account belonging to sudoers and start the daemon. Example :
```plaintext ```plaintext
[caillat-adm@juliette ~]$ sudo service docker start [caillat-adm@juliette ~]$ sudo service docker start
[sudo] Mot de passe de caillat-adm : [sudo] Mot de passe de caillat-adm :
Redirecting to /bin/systemctl start docker.service Redirecting to /bin/systemctl start docker.service
[caillat-adm@juliette ~]$ ps aux|grep dockerd [caillat-adm@juliette ~]$ ps aux|grep dockerd
root 14212 0.2 0.0 2064984 65392 ? Ssl 11:59 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 14212 0.2 0.0 2064984 65392 ? Ssl 11:59 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
``` ```
### Having `dockerd` automatically started during the future boot sequences ### Having `dockerd` automatically started during the future boot sequences
```plaintext ```plaintext
sudo systemctl enable docker.service sudo systemctl enable docker.service
``` ```
# Starting the yafits containers # Starting the yafits containers
## Change the current directory to the yafits project's : ## Change the current directory to the yafits project's :
```plaintext ```plaintext
cd <..>/yafits cd <..>/yafits
``` ```
## Build and run the containers with docker-compose ## Build and run the containers with docker-compose
```plaintext ```plaintext
docker-compose build docker-compose build
docker-compose up -d docker-compose up -d
``` ```
## Build and run the containers with the control script ## Build and run the containers with the control script
Yafits provides a bash control script called yafits installed in the root directory. It accepts the following arguments : Yafits provides a bash control script called yafits installed in the root directory. It accepts the following arguments :
* help : Show help * help : Show help
* build : Build the application * build : Build the application
* buildrun : Build and run the application * buildrun : Build and run the application
* down : stops all containers * down : stops all containers
* prune : Prune existing containers * prune : Prune existing containers
* run : Run the application without build * run : Run the application without build
* updatespectro : Will build and run yafits, spectroscopy database will be updated
# Starting portainer
# Starting portainer
```plaintext
```plaintext docker volume create portainer_data
docker volume create portainer_data docker run -d -p 8000:8000 -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
docker run -d -p 8000:8000 -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer ```
```
## Activating `jupyterhub` sur `romeo`
## Activating `jupyterhub` sur `romeo`
Start a session under the `jupyterhub` account and then type :
Start a session under the `jupyterhub` account and then type :
```plaintext
```plaintext nohup jupyterhub &
nohup jupyterhub & ```
```
and logout.
and logout.
or in a more system admin way :
or in a more system admin way :
```plaintext
```plaintext sudo systemctl start jupyterhub.service
sudo systemctl start jupyterhub.service ```
```
and possibly check the status of the service :
and possibly check the status of the service :
```plaintext
```plaintext sudo systemctl status jupyterhub.service
sudo systemctl status jupyterhub.service ● jupyterhub.service - Jupyterhub
● jupyterhub.service - Jupyterhub Loaded: loaded (/usr/lib/systemd/system/jupyterhub.service; disabled; vendor preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/jupyterhub.service; disabled; vendor preset: disabled) Active: active (running) since mar. 2020-12-15 10:19:27 CET; 4min 15s ago
Active: active (running) since mar. 2020-12-15 10:19:27 CET; 4min 15s ago Main PID: 115277 (jupyterhub)
Main PID: 115277 (jupyterhub) Tasks: 7
Tasks: 7 Memory: 99.1M
Memory: 99.1M CGroup: /system.slice/jupyterhub.service
CGroup: /system.slice/jupyterhub.service ├─115277 /home/jupyterhub/anaconda3/bin/python /home/jupyterhub/anaconda3/bin/jupyterhub -f /home/jupyterhub/jupyterhub_config.py
├─115277 /home/jupyterhub/anaconda3/bin/python /home/jupyterhub/anaconda3/bin/jupyterhub -f /home/jupyterhub/jupyterhub_config.py └─115287 node /home/jupyterhub/anaconda3/bin/configurable-http-proxy --ip --port 8500 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0....
└─115287 node /home/jupyterhub/anaconda3/bin/configurable-http-proxy --ip --port 8500 --api-ip 127.0.0.1 --api-port 8001 --error-target http://127.0....
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.244 - info: [ConfigProxy] 200 GET /api/routes
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.244 - info: [ConfigProxy] 200 GET /api/routes déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.244 JupyterHub proxy:301] Checking routes
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.244 JupyterHub proxy:301] Checking routes déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.244 JupyterHub proxy:370] Adding default route for Hub: / => http://127.0.0.1:8081
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.244 JupyterHub proxy:370] Adding default route for Hub: / => http://127.0.0.1:8081 déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.247 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.247 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081 déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.249 - info: [ConfigProxy] 201 POST /api/routes/
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: 10:19:31.249 - info: [ConfigProxy] 201 POST /api/routes/ déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.249 JupyterHub app:1906] JupyterHub is now running at http://:8500
déc. 15 10:19:31 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:19:31.249 JupyterHub app:1906] JupyterHub is now running at http://:8500 déc. 15 10:20:10 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:10.826 JupyterHub log:158] 302 GET / -> /hub (@::ffff:145.238.200.65) 1.56ms
déc. 15 10:20:10 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:10.826 JupyterHub log:158] 302 GET / -> /hub (@::ffff:145.238.200.65) 1.56ms déc. 15 10:20:11 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:11.526 JupyterHub log:158] 302 GET /hub -> /hub/ (@::ffff:145.238.200.65) 0.89ms
déc. 15 10:20:11 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:11.526 JupyterHub log:158] 302 GET /hub -> /hub/ (@::ffff:145.238.200.65) 0.89ms déc. 15 10:20:11 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:11.746 JupyterHub log:158] 302 GET /hub/ -> /hub/login (@::ffff:145.238.200.65) 1.02ms
déc. 15 10:20:11 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:11.746 JupyterHub log:158] 302 GET /hub/ -> /hub/login (@::ffff:145.238.200.65) 1.02ms déc. 15 10:20:12 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:12.529 JupyterHub log:158] 200 GET /hub/login (@::ffff:145.238.200.65) 68.62ms
déc. 15 10:20:12 romeo.obspm.fr jupyterhub[115277]: [I 2020-12-15 10:20:12.529 JupyterHub log:158] 200 GET /hub/login (@::ffff:145.238.200.65) 68.62ms ```
```
## Having `jupyterhub` active concurrently with `docker` on `romeo`
## Having `jupyterhub` active concurrently with `docker` on `romeo`
By default `jupyterhub`'s proxy uses the port 8000 which the port used by `docker`'s. See below a way to have `jupyterhub`'s proxy using a different port ( e.g. 8500 ) :
By default `jupyterhub`'s proxy uses the port 8000 which the port used by `docker`'s. See below a way to have `jupyterhub`'s proxy using a different port ( e.g. 8500 ) :
* In `jupyterhub`'s account on `romeo` ( username : jupyterhub, password: askme) edit the file `jupyterhub_config.py` in the home directory and type `c.JupyterHub.bind_url = 'http://:8500'` at line 94
* In `jupyterhub`'s account on `romeo` ( username : jupyterhub, password: askme) edit the file `jupyterhub_config.py` in the home directory and type `c.JupyterHub.bind_url = 'http://:8500'` at line 94 * Start `jupyterhub`. In the same account type `nohup jupyterhub &` . Be careful that a `nohup.out` file logging the activity is going to grow.
* Start `jupyterhub`. In the same account type `nohup jupyterhub &` . Be careful that a `nohup.out` file logging the activity is going to grow. * As root or from a sudo enabled account open the port 8500 once forever :
* As root or from a sudo enabled account open the port 8500 once forever :
```plaintext
```plaintext sudo firewall-cmd --zone=public --permanent --add-port=8500/tcp
sudo firewall-cmd --zone=public --permanent --add-port=8500/tcp sudo firewall-cmd --reload
sudo firewall-cmd --reload ```
```
At this point you should be able to access `jupyterhub` on `romeo` via the link <https://romeo.obspm.fr:8500>
At this point you should be able to access `jupyterhub` on `romeo` via the link <https://romeo.obspm.fr:8500>
# Link between romeo and juliette
# Link between romeo and juliette
There is a dedicated ethernet card to create the link between the two machines.
There is a dedicated ethernet card to create the link between the two machines. If it is not mounted on juliette after a reboot :
If it is not mounted on juliette after a reboot :
**as root** or **sudo** on juliette
**as root** or **sudo** on juliette nmcli connection up em2 # romeo
nmcli connection up em2 # romeo nmcli connection up em3 # benvolio
nmcli connection up em3 # benvolio
Then call automount command as root on romeo or benvolio
Then call automount command as root on romeo or benvolio
If romeo or benvolio is shut down while juliette stays up, the connection must be restarted
If romeo or benvolio is shut down while juliette stays up, the connection must be restarted
em2 is the name of the card on juliette (p7p2 on romeo)
em2 is the name of the card on juliette (p7p2 on romeo)
automount is enabled, it is configured in :
automount is enabled, it is configured in : - /etc/auto.master
- /etc/auto.master - /etc/auto.datartemix
- /etc/auto.datartemix - /etc/auto.datartemix2
- /etc/auto.datartemix2
# Hardware configuration
# Hardware configuration
This section describes the network, storage and computing resources dedicated to ARTEMIX and YAFITS on their development site i.e. the LERMA/Observatoire de Paris.
This section describes the network, storage and computing resources dedicated to ARTEMIX and YAFITS on their development site i.e. the LERMA/Observatoire de Paris.
## juliette\[.obspm.fr\]
## juliette\[.obspm.fr\]
The host for artemix ( artemix.obspm.fr ) and yafits ( artemix.obspm.fr/fits ).
The host for artemix ( artemix.obspm.fr ) and yafits ( artemix.obspm.fr/fits ).
* machine : DELL R740, 2 processors, 32 cores, 384GB RAM
* machine : DELL R740, 2 processors, 32 cores, 384GB RAM * disks :
* disks : - /datartemix ( 83TB )
- /datartemix ( 83TB ) - /datartemix2 ( 101TB )
- /datartemix2 ( 101TB )
# Web access configuration
# Web access configuration
Both the artemix.obspm.fr and yafits.obspm.fr domains are accessed through the vm-weblerma.obspm.fr proxy server. More generally, all the artemix related services are proxied this way (almastaging, almaspectro, dev-artemix, dev2-artemix).
Both the artemix.obspm.fr and yafits.obspm.fr domains are accessed through the vm-weblerma.obspm.fr proxy server. More generally, all the artemix related services are proxied this way (almastaging, almaspectro, dev-artemix, dev2-artemix).
This configuration can be modified in the netmagis service provided by the DIO (netmagis.obspm.fr).
This configuration can be modified in the netmagis service provided by the DIO (netmagis.obspm.fr).
Access restrictions are applied to "restricted" and "RRL" data directories. The configuration is done directly in the vhost configuration file of the proxy server.
Access restrictions are applied to "restricted" and "RRL" data directories. The configuration is done directly in the vhost configuration file of the proxy server.
# House cleaning
# House cleaning
The execution of YAFITS servers generates a lot of temporary files amongst which are the FITS files generated a
The execution of YAFITS servers generates a lot of temporary files amongst which are the FITS files generated a
# Network considerations
# Network considerations
## Open a port
## Open a port
Example :
Example :
```plaintext
```plaintext [caillat-adm@juliette ~]$ sudo sudo firewall-cmd --zone=public --permanent --add-port=3023/tcp
[caillat-adm@juliette ~]$ sudo sudo firewall-cmd --zone=public --permanent --add-port=3023/tcp [sudo] Mot de passe de caillat-adm :
[sudo] Mot de passe de caillat-adm : success
success [caillat-adm@juliette ~]$ sudo firewall-cmd --reload
[caillat-adm@juliette ~]$ sudo firewall-cmd --reload success
success ```
```
to add the port 3023 and have it available immediately.
to add the port 3023 and have it available immediately.
# Monitoring activity
# Monitoring activity
Yafits provides a page that summarizes the current status of the application ( loaded files, used memory ).
Yafits provides a page that summarizes the current status of the application ( loaded files, used memory ). It is available at the address :
It is available at the address :
http://yafitshome/monitor
http://yafitshome/monitor
# yafitss documentation
# yafitss documentation
It is available at $SERVER_URL:5252/apidoc/index.html
It is available at $SERVER_URL:5252/apidoc/index.html
* 5252 is the port number at which yafitss is visible (can be changed)
* 5252 is the port number at which yafitss is visible (can be changed) * apidoc prefix is a reference to the tool uses to generate the documentation. Its activation can be found in yafitss's Dockerfile :
* apidoc prefix is a reference to the tool uses to generate the documentation. Its activation can be found in yafitss's Dockerfile :
```
``` # Generate the bottle server documentation
# Generate the bottle server documentation WORKDIR /home/partemix/yafitss
WORKDIR /home/partemix/yafitss RUN apidoc -f serverWsgi.py -o /home/partemix/dataroot/apidoc
RUN apidoc -f serverWsgi.py -o /home/partemix/dataroot/apidoc
``` ```
\ No newline at end of file