Architecture

Architecture

Requirements

  • docker software installed on your computer for helping you to build your model image
  • a registry for hosting your docker images, we can provide one if needed
  • an account on a coconut instance

Global Workflow

  • create your model image
  • send it to your registry
  • log into to the plaform
  • create a new project
  • send some parameters data
  • ask for a simple simulation
  • see the results
  • download the report

Context

You have a command-line simulator compatible with Linux and you want to perform some analysis or parameter estimation.

Your simulator will be viewed as a black-box for COCONUT.

There are two main modes of execution in COCONUT : full time execution when you go from 0 to t_max step by step execution when you from n to n+1 cycle, which is what we want for the data assimilation. run.sh, run-step.sh and file formats run.sh and run-step.sh There are two kinds of scripts : run.sh is dedicated for execution from first cycle to last cycle of the simulation run-step.sh is dedicated for execution from one cycle to another cycle

File Formats

File formats are comma seperated values (CSV) but the structure itself is specific. We need a file format for parameters, states and observations. Files formats are described below with examples.

  • Parameters and states
param1, value1 # for set 1
param2, value2 # for set 1
param3, value3 # for set 1
param1, value1 # for set 2
param2, value2 # for set 2
param3, value3 # for set 2
param1, value1 # for set 3
param2, value2 # for set 3
param3, value3 # for set 3
…
  • Observations
0,observation_model1,time1,variable_name1,value1 # for set 1
1,observation_model1,time2,variable_name1,value2 # for set 1
2,observation_model2,time3,variable_name1,value3 # for set 1
3,observation_model2,time1,variable_name2,value4 # for set 1
4,observation_model2,time2,variable_name2,value5 # for set 1
5,observation_model2,time3,variable_name2,value6 # for set 1
0,observation_model1,time1,variable_name1,value1 # for set 2
1,observation_model1,time2,variable_name1,value2 # for set 2
2,observation_model2,time3,variable_name1,value3 # for set 2
3,observation_model2,time1,variable_name2,value4 # for set 2
4,observation_model2,time2,variable_name2,value5 # for set 2
5,observation_model2,time3,variable_name2,value6 # for set 2
0,observation_model1,time1,variable_name1,value1 # for set 3
1,observation_model1,time2,variable_name1,value2 # for set 3
2,observation_model2,time3,variable_name1,value3 # for set 3
3,observation_model2,time1,variable_name2,value4 # for set 3
4,observation_model2,time2,variable_name2,value5 # for set 3
5,observation_model2,time3,variable_name2,value6 # for set 3
…
  • Examples If your simulator can’t read/write list file, It is necessary to split merge file to work with COCONUT. Below we provide examples for splitting and merging files

  • Splitting files

split -ln $1 p_
# $1 is input given by COCONUT. For run.sh script if only $1, for run-step.sh script it can be $1 and/or $2
# -ln is the split line with n the number of lines to split
# $1 is a file path
# p_ is a prefix to splitted files
  • Merging files
cat p_* > $2
# p_ is a prefix of files you want merge
# $2 is an output path given by COCONUT. For run.sh script if only $2, for run-step.sh script it can be $3 and/or $4

How to create a script run.sh for a simulation from first cycle to last cycle ?

run.sh script is a bash script for launching your simulator in the COCONUT environment with our input from 0 to t_max.

run.sh

  • Introduction COCONUT uses bash syntax to execute one simulation, the goal of run.sh script is to allow to launch simulator in one command line with parameters list file path and output list file path in arguments, like :
./run.sh /data/pl.csv /data/yl.csv

Thus we have two Inputs :

# $1 path to initial parameters set list (csv format) [pl_in]
# $2 path to output observation list(csv format) [yl_out]
  • Script file At the end, your run.sh file must be like :
#!/bin/bash

# INPUT
# $1 path to initial parameters set list [pl_in]
# $2 path to output observation list [yl_out]

# if it is necessary convert input file to your format for your simulator
# for example
awk '{ gsub(",","=",$1); print $1 }' $1 > $1.new
mv $1.new $1

# launch your simulator with input and write in output
./simulator --input-parameters $1 --output-observation $2

# if it is necessary convert output file from your simulator to format for COCONUT
# for example
awk '{ gsub("=",",",$1); print $1 }' $2 > $2.new
mv $2.new $2

How to create a script run-step.sh for a simulation from one cycle to another cycle ?

run-step.sh script is a bash script for launching your simulator in the COCONUT environment with our input from n to n+1.

run-step.sh

  • Introduction COCONUT uses bash syntax to execute one simulation, the goal of run-step.sh script is to allow to launch simulator in one command line with input file paths, output file paths, number of step, time to begin and number of simulations integer in arguments, like :
./run-step.sh /data/pl.csv /data/xl.csv /data/yl.csv /data/xlnew.csv 5 1 10000

Thus we have two Inputs :

# $1 path to initial parameters set list (csv format) [pl_in]
# $2 path to initial state list (csv format) [xl_in]
# $3 path to observation output list (csv format) [yl_out]
# $4 path to all states output list (csv format) [xlnew_out]
# $5 nbr of step to do (integer) [(n+1) - n]
# $6 start time (integer) [n]
# $7 nbr simulation (integer) [number of particles]
  • Script file At the end, your run-step.sh file must be like :
#!/bin/bash

# INPUT
# $1 path to initial parameters set list (csv format) [pl_in]
# $2 path to initial state list (csv format) [xl_in]
# $3 path to observation output list (csv format) [yl_out]
# $4 path to all states output list (csv format) [xlnew_out]
# $5 nbr of step to do (integer) [(n+1) - n]
# $6 start time (integer) [n]
# $7 nbr simulation (integer) [number of particles]

# if it is necessary convert input files to your format for your simulator
# for example
awk '{ gsub(",","=",$1); print $1 }' $1 > $1.new
mv $1.new $1
awk '{ gsub(",","=",$1); print $1 }' $2 > $2.new
mv $2.new $2

# launch your simulator with input and write in output
./simulator --input-parameters $1 --input-states $2 --output-observation $3 --output-states $4 --nbr-step $5 --start-time $6 --nbr-simulation $7

# if it is necessary convert output files from your simulator to format for COCONUT
# for example
awk '{ gsub("=",",",$2); print $1 }' $3 > $3.new
mv $3.new $3
awk '{ gsub("=",",",$2); print $1 }' $4 > $4.new
mv $4.new $4

How to create the Docker image ?

  • Introduction Docker is used to emulate a system with your simulator. Thus COCONUT can use any simulator independently of the operating system. Your Dockerfile must start with an image of the system you want to emulate, like ubuntu 16.04. It also must contain your simulator but has to be executable by a simple user thus there are three steps :
  • initialize with the distribution
  • create a user
  • copy your simulator into the container

Attention : we assume that folder “/data” was reserve to COCONUT to share files between COCONUT and your container, thus your image must not use this “/data” folder for managing something.

1 - Initialization Here you can find an example for using a python based simulator :

FROM python:latest
MAINTAINER your-id

2 - Add a user COCONUT uses specific user to run script on container, we assume user name is user01 and it is in staff group. Add the following in your Dockerfile:

RUN useradd -ms /bin/bash -G staff user01
USER user01
WORKDIR /home/user01

3 - Copy your simulator Now, you can add your instructions to copy files necessary to execute simulation. Don’t forget to copy your run scripts (run.sh and/or run-step.sh) and your simulator if it is not in docker distribution. Make sure your static file have read write for all and your simulator and run scripts have read/exec write for all.

For example :

# copy static file like environment, static configuration …
COPY environment.txt /home/user01/
# if your simulator if not in the container distribution
COPY my_simulator.py /home/user01/
# copy run scripts
COPY run.sh /home/user01/
COPY run-step.sh /home/user01/
  • Dockerfile example At the end, your run-step.sh file must be like :
FROM python:latest
MAINTAINER your-id

RUN useradd -ms /bin/bash -G staff user01
USER user01
WORKDIR /home/user01

# copy static file like environment, static configuration …
COPY environment.txt /home/user01/
# if your simulator if not in the container distribution
COPY my_simulator.py /home/user01/
# copy run scripts
COPY run.sh /home/user01/
COPY run-step.sh /home/user01/
  • Push to your registry If everything is ok, you can now push the image to your registry.