Infrastructure to deploy Research Projects

The Infrastructure expects to be deployed in the same Pod / local environment as the analysis container and handles all outbound communication.

Configuration

A configuration is located in /config.yaml. All relevant variables can be configured via exporting environment variables.

Environment Variable	Default	Description
LOGGING_LEVEL_ROOT	DEBUG	Logging level for service logger
PROBING_WEBSERVER_HOST	"0.0.0.0"	Probe webserver address
PROBING_WEBSERVER_PORT	8080	Probe webserver port
PROBING_WEBSERVER_MODE	production	Webserver mode: {development, production}
RABBITMQ_HOST	localhost	RabbitMQ host address
RABBITMQ_PORT	5672	RabbitMQ host port
RABBITMQ_USERNAME	user	RabbitMQ username
RABBITMQ_PASSWORD	bitnami	RabbitMQ password
RABBITMQ_HEARTBEAT	7200	Controls AMQP heartbeat timeout in seconds
REQUEST_QUEUE	request_queue	Requests to service
RESPONSE_QUEUE	response_queue	Responses by service
DEAD_LETTER_QUEUE	dead_letter_queue	Messages that failed to process
ANALYSIS_ENDPOINT	"http://127.0.0.1:5000"	Endpoint for analysis container
STORAGE_BACKEND	s3	The type of storage to use {s3, azure}
STORAGE_BUCKET	"redaction"	The bucket / container to pull files specified in queue requests from
STORAGE_ENDPOINT	"http://127.0.0.1:9000"	Endpoint for s3 storage
STORAGE_KEY	root	User for s3 storage
STORAGE_SECRET	password	Password for s3 storage
STORAGE_AZURECONNECTIONSTRING	"DefaultEndpointsProtocol=..."	Connection string for Azure storage
STORAGE_AZURECONTAINERNAME	"redaction"	AKS container

Response Format

Expected AMQP input message:

{
   "dossierId": "",
   "fileId": "",
   "targetFileExtension": "",
   "responseFileExtension": ""
}

Optionally, the input message can contain a field with the key "operations".

AMQP output message:

{
  "dossierId": "",
  "fileId": "",
   ...
}

Setup

Install project dependencies

 make poetry

You don't have to install it independently in the project repo, just import pyinfra in any .py-file

or install form another project

poetry add git+ssh://git@git.iqser.com:2222/rr/pyinfra.git#TAG-NUMBER