HTTP Status reporting

STEMsalabim simulations may take a long time, even when running them in parallel on many processors. In order to ease tracking of the status of running simulations, we built reporting via HTTP POST requests into the program.

In order to use that feature, the libCURL library has to be installed and STEMsalabim needs to be linked against it.

Configure HTTP reporting

To configure HTTP reporting, please add the http_reporting: {} block to your simulations’s parameter file, containing at least reporting = true; and the url to report to, url = "http://my_server_address:port/path";.

If you want to use HTTP basic authentication, you may also specify the options auth_user = "your_user"; and auth_pass = "your_pass";. Note, that HTTP basic auth will be enabled as soon as auth_user is not empty. You should therefore only fill in that field when you want to use authentication.

Additional, custom payload for the HTTP requests may be specified in the sub-block parameters: {}. Each key-value pair in this block is translated into JSON and appended to each request. This allows you to use custom authentication techniques, such as token-based authentication etc.

An example configuration block with HTTP basic authentication may look like this:

http_reporting: {
    reporting = true;
    url = "http://my_api_endpoint:8000/stemsalabim-reporting";
    auth_user = "my_user";
    auth_pass = "my_pass";
    parameters: {
        simulation_category = "suitable for many Nature papers";
    }
}

The status requests

In each request that STEMsalabim sends, some JSON payload is common. In addition to the JSON values specified in the parameter file (see Configure HTTP reporting), the following parameters are always reported:

time: // The currrent date and time
id: // the UUID of the simulation
num_threads: // the number of threads of each MPI processor
num_processors: // the number of MPI processors
num_defoci: // the total number of defoci to calculate
num_configurations: // the total number of frozen phonon configurations to calculate
event: // A code for what event is reported. see below.

The following four different event codes, each for a different event, are reported:

START_SIMULATION: A simulation is started

This request is sent at the beginning of a simulation. Additional key/value pairs sent are:

event: "START_SIMULATION"
version: // program version
git_commit: // git commit hash of the program version
title: // simulation title

START_DEFOCUS: A defocus iteration is started

This request is sent at the beginning of a defocus iteration. Additional key/value pairs sent are:

event: "START_DEFOCUS"
defocus: // the defocus value in nm
defocus_index: // the index of the defocus, between 0 and num_defoci

START_FP_CONFIGURATION: A frozen phonon iteration is started

This request is sent at the beginning of a frozen phonon configuration. Additional key/value pairs sent are:

event: "START_FP_CONFIGURATION"
defocus: // the defocus value in nm
defocus_index: // the index of the defocus, between 0 and num_defoci
configuration_index: // the index of the configuration, between 0 and num_configurations

PROGRESS: Progress report

This request is sent during the calculation, typically after each integer percent of the simulation finished. Additional key/value pairs sent are:

event: "START_CONFIGURATION"
defocus: // the defocus value in nm
defocus_index: // the index of the defocus, between 0 and num_defoci
configuration_index: // the index of the configuration, between 0 and num_configurations
progress: // progress between 0 and 1 of this configuration iteration within this defocus iteration

FINISH: Simulation finished

This request is sent when the simulation finished. Additional key/value pairs sent are:

event: "FINISH"

How to process the reports

Obviously, in order to register the requests, an HTTP(S) server needs to be running on the target machine. For example, a very simple server in python using the http://flask.pocoo.org/ package, that only echos the requests, can be implemented as:

#!/usr/bin/env python

from flask import Flask
from flask import request
import json

app = Flask(__name__)

@app.route('/', methods=['POST'])
def echo():
    content = request.get_json()
    print(json.dumps(content, indent=4))
    return ""

if __name__ == "__main__":
    app.run()

Run the script and then start a STEMsalabim simulation to see requests imcoming.