Julius Unverfehrt f5b7203778 Pull request #20 : Pika encapsulation

Merge in RR/pyinfra from pika_encapsulation to master

Squashed commit of the following:

commit 251fea4094062a72f5f0b1f8a54f959a1d7309ec
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 15:02:25 2022 +0100

    Adjusted readme and config to the actual changes.

commit d00a0aadc02be8ab1343dbaa2a8df82418e77673
Merge: ded29bb c34a1ce
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 14:06:21 2022 +0100

    Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into pika_encapsulation

commit ded29bb8d0a78c1d5fd172edb74f0b120da05d5f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 14:02:37 2022 +0100

    blackkkkkkkkkkkkkkkk

commit de3899c69f4e83fa5b8dc2702a18be66661201e9
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:59:43 2022 +0100

    Judgement Day - The Great Refactoring

commit 48eb2bb792dd45539e7db20949fedae03316c2b3
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:30:41 2022 +0100

    blacky

commit 38fb073291c6989fcc5560ef9f97e7dafd9570be
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:29:46 2022 +0100

    update mock client/publish

commit 8744cdeb6d21adab81fb82c7850a4ada31a1a3f9
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:29:19 2022 +0100

    quickfix consumer.consume_and_publish bug

commit 0ce30cafbf15142e3f773d0e510b7d22927ee86d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 11:42:04 2022 +0100

    updated serving logic for new queue, visior and storage logic

commit 5ac5fbf50bf9d8a6b6466cd04d82d21525250b4c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 11:41:18 2022 +0100

    changed callback signature to expect a dict with a key 'data'

commit 4c7c4b466e2b5228a6098c9130cd6c3334315153
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 09:46:39 2022 +0100

    set dead letter queue related params

commit f8b60aad16eb0f5c91bed857ac3b352c4bcb7468
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 20:22:57 2022 +0100

    added republishing logic

commit a40f07139af9121d8b72efa52b6b2588e7d233e8
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:10:08 2022 +0100

    removed obsolete import

commit d3605b0913f8b155961b8c8e90e7c1d442a1da4f
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:09:50 2022 +0100

    removed obsolete code

commit 9130b9fc753c63bfe4abd3cc26b31caf5c5bf2cb
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:05:59 2022 +0100

    test refac

commit 1205adecdbd42a46dcc264a46086aab94c285b94
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:59 2022 +0100

    removed obsolete import

commit 59d0de97e39646b3fa6c6ca91644dbf8adbd0de3
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:21 2022 +0100

    applied black

commit f78a6894bc99b42d53a7c265ffd8c7869e9d606d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:03 2022 +0100

    applied black

commit 32740a914f8be15eed954389f6ddbb565e1567e6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:47:47 2022 +0100

    removed obsolete code

commit f21cbc141efc10d0e779f683dcc452e6c889dd6b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:47:27 2022 +0100

    consumer with visitor test fixed (added connection.close() in case of pika queue manager)

commit c415dcd8e6375e015b959a9c3cdee61b8ffe7d60
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 13:56:54 2022 +0100

    consumer with visitor test WIP 2

... and 15 more commits

2022-03-15 15:05:14 +01:00

4.3 KiB

Executable File

Raw Blame History

Infrastructure to deploy Research Projects

The Infrastructure expects to be deployed in the same Pod / local environment as the analysis container and handles all outbound communication.

Configuration

A configuration is located in /config.yaml. All relevant variables can be configured via exporting environment variables.

Environment Variable	Default	Description
LOGGING_LEVEL_ROOT	DEBUG	Logging level for service logger
PROBING_WEBSERVER_HOST	"0.0.0.0"	Probe webserver address
PROBING_WEBSERVER_PORT	8080	Probe webserver port
PROBING_WEBSERVER_MODE	production	Webserver mode: {development, production}
RABBITMQ_HOST	localhost	RabbitMQ host address
RABBITMQ_PORT	5672	RabbitMQ host port
RABBITMQ_USERNAME	user	RabbitMQ username
RABBITMQ_PASSWORD	bitnami	RabbitMQ password
RABBITMQ_HEARTBEAT	7200	Controls AMQP heartbeat timeout in seconds
REQUEST_QUEUE	request_queue	Requests to service
RESPONSE_QUEUE	response_queue	Responses by service
DEAD_LETTER_QUEUE	dead_letter_queue	Messages that failed to process
ANALYSIS_ENDPOINT	"http://127.0.0.1:5000"	Endpoint for analysis container
STORAGE_BACKEND	s3	The type of storage to use {s3, azure}
STORAGE_BUCKET	"pyinfra-test-bucket"	The bucket / container to pull files specified in queue requests from
STORAGE_ENDPOINT	"http://127.0.0.1:9000"	Endpoint for s3 storage
STORAGE_KEY	root	User for s3 storage
STORAGE_SECRET	password	Password for s3 storage
STORAGE_AZURECONNECTIONSTRING	"DefaultEndpointsProtocol=..."	Connection string for Azure storage

Response Format

Expected AMQP input message:

{
   "dossierId": "",
   "fileId": "",
   "targetFileExtension": "",
   "responseFileExtension": ""
}

Optionally, the input message can contain a field with the key "operations".

AMQP output message:

{
  "dossierId": "",
  "fileId": "",
   ...
}

Development

Either run src/serve.py or the built Docker image.

Setup

Install module.

 pip install -e .
 pip install -r requirements.txt

or build docker image.

docker build -f Dockerfile -t pyinfra .

Usage

Shell 1: Start a MinIO and a RabbitMQ docker container.

docker-compose up

Shell 2: Add files to the local minio storage.

python scripts/manage_minio.py add <MinIO target folder> -d path/to/a/folder/with/PDFs

Shell 2: Run pyinfra-server.

python src/serve.py

or as container:

docker run --net=host pyinfra

Shell 3: Run analysis-container.

Shell 4: Start a client that sends requests to process PDFs from the MinIO store and annotates these PDFs according to the service responses.

python scripts/mock_client.py

4.3 KiB Executable File Raw Blame History