Data Handlers#

BinaryFileWritingDataHandler#

This Data Handler class writes serialized data into a file on the filesystem. The name of each file is generated by joining a string identifying the MPI rank of the process writing the file with a counter that increases by one for each file written by the same process. A suffix is appended to the resulting string, and optionally a prefix can be prepended to it.

Configuration Parameters for BinaryFileWritingDataHandler#

  • file_prefix (str): This parameter is optional. It defines a prefix that is prepended to the name of each file written by the Data Handler. If the prefix does not already end with an underscore, one is added automatically as a separator. The default value is "" (an empty string, meaning no prefix is added). Example: processed_

  • file_suffix (str): This parameter is optional. It defines the suffix (extension) for the files written by the Data Handler, without the leading dot. The default value is h5 (i.e. an .h5 extension is used for the files). Example: data

  • write_directory (str): This parameter is optional. It defines the directory where the Data Handler writes the files. The directory is created, including any missing parent directories, if it does not already exist. The default value of this parameter is the current working directory. Example: /data/output

BinaryDataStreamingDataHandler#

This Data Handler class sends serialized data through a network socket, to be consumed by an external application. The handler creates or connects to sockets using the ZMQ library. The socket provides backstop capabilities: if nothing is receiving data on the other side of the socket, sending will block and eventually time out rather than silently dropping data.

Configuration Parameters for BinaryDataStreamingDataHandler#

  • urls (list of str): This parameter defines the list of URIs where the Data Handler opens or connects its sockets. Each URI must be in a format compatible with the ZMQ library (e.g. tcp://localhost:5555). This parameter must always be provided as a list, even if only one URI is specified. Example:

    urls:
  - tcp://localhost:5555

  • role (str): This parameter is optional. It specifies whether the Data Handler should create a socket or connect to an already existing one. The parameter can only take two values: server (the handler binds the socket, i.e. performs a ZMQ bind operation) or client (the handler connects to an existing socket, i.e. performs a ZMQ connect operation). The default value of this parameter is server. Example: client

  • library (str): This parameter is optional. It dictates which library the handler uses to manage the socket. Currently only the value zmq is supported. The default value of this parameter is zmq. Example: zmq

  • socket_type (str): This parameter is optional. It determines the type of socket that the handler creates. Currently only the PUSH socket type is supported, and this parameter can only take the value push. The default value of this parameter is push. Example: push