PyInfra

1. About
2. Configuration
3. Response Format
4. Usage & API
5. Scripts
6. Tests

About

Common Module with the infrastructure to deploy Research Projects. The Infrastructure expects to be deployed in the same Pod / local environment as the analysis container and handles all outbound communication.

Configuration

A configuration is located in /config.yaml. All relevant variables can be configured via exporting environment variables.

Environment Variable	Default	Description
LOGGING_LEVEL_ROOT	DEBUG	Logging level for service logger
PROBING_WEBSERVER_HOST	"0.0.0.0"	Probe webserver address
PROBING_WEBSERVER_PORT	8080	Probe webserver port
PROBING_WEBSERVER_MODE	production	Webserver mode: {development, production}
RABBITMQ_HOST	localhost	RabbitMQ host address
RABBITMQ_PORT	5672	RabbitMQ host port
RABBITMQ_USERNAME	user	RabbitMQ username
RABBITMQ_PASSWORD	bitnami	RabbitMQ password
RABBITMQ_HEARTBEAT	7200	Controls AMQP heartbeat timeout in seconds
REQUEST_QUEUE	request_queue	Requests to service
RESPONSE_QUEUE	response_queue	Responses by service
DEAD_LETTER_QUEUE	dead_letter_queue	Messages that failed to process
ANALYSIS_ENDPOINT	"http://127.0.0.1:5000"	Endpoint for analysis container
STORAGE_BACKEND	s3	The type of storage to use {s3, azure}
STORAGE_BUCKET	"redaction"	The bucket / container to pull files specified in queue requests from
STORAGE_ENDPOINT	"http://127.0.0.1:9000"	Endpoint for s3 storage
STORAGE_KEY	root	User for s3 storage
STORAGE_SECRET	password	Password for s3 storage
STORAGE_AZURECONNECTIONSTRING	"DefaultEndpointsProtocol=..."	Connection string for Azure storage
STORAGE_AZURECONTAINERNAME	"redaction"	AKS container

Response Format

Expected AMQP input message:

{
   "dossierId": "",
   "fileId": "",
   "targetFileExtension": "",
   "responseFileExtension": ""
}

Optionally, the input message can contain a field with the key "operations".

AMQP output message:

{
  "dossierId": "",
  "fileId": "",
   ...
}

Usage & API

Setup

Install project dependencies

 make poetry

You don't have to install it independently in the project repo, just import pyinfra in any .py-file

or install form another project

poetry add git+ssh://git@git.iqser.com:2222/rr/pyinfra.git#TAG-NUMBER

API

TBA

Scripts

Run pyinfra locally

Shell 1: Start minio and rabbitmq containers

$ cd tests && docker-compose up

Shell 2: Start pyinfra with callback mock

$ python scripts/start_pyinfra.py 

Shell 3: Upload dummy content on storage and publish message

$ python scripts/mock_process_request.py

Tests

The tests take a bit longer than you are probably used to, because among other things the required startup times are quite high. The test runtime can be accelerated by setting 'autouse' to 'False'. In that case, run 'docker-compose up' in the tests dir manually before running the tests.