pyinfra/tests/payload_test.py
Julius Unverfehrt d6de45d783 Pull request #65: RED-6205 monitoring
Merge in RR/pyinfra from RED-6205-monitoring to master

Squashed commit of the following:

commit 529cedfd7c065a3f7364e4596b923f25f0af76b5
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date:   Thu Mar 16 14:57:26 2023 +0100

    Remove unnecessary default argument to dict.get

commit b718531f568e89df77cc05039e5e7afe7111b9a4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 16 14:56:50 2023 +0100

    refactor

commit c039b0c25a6cd2ad2a72d237d0930c484c8e427c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 16 13:22:17 2023 +0100

    increase package version to reflect the recent changes

commit 0a983a4113f25cd692b68869e1f33ffbf7efc6f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 16 13:16:39 2023 +0100

    remove processing result conversion to a list, since ner-predicion service actually returns a dictionary. It is now expected that the result is sized to perform the monitoring and json dumpable to upload it.

commit 541bf321410471dc09a354669b2778402286c09f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 16 12:48:07 2023 +0100

    remove no longer needed requirements

commit cfa182985d989a5b92a9a069a603daee72f37d49
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 16 11:14:58 2023 +0100

    refactor payload formatting

    - introduce PayloadFormatter class for better typehinting and bundling
    of functionality
    - parametrize payload formatting so the PayloadProcesser can adapt
    better to differnt services/products
    - move file extension parsing to its own module

commit f57663b86954b7164eeb6db013d862af88ec4584
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Mar 15 12:22:08 2023 +0100

    refactor payload parsing

    - introduce QueueMessagePayloadParser for generality
    and typehinting
    - refactor file extension parsing algorithm

commit 713fb4a0dddecf5442ceda3988444d9887869dcf
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 14 17:07:02 2023 +0100

    fix tests

commit a22ecf7ae93bc0bec235fba3fd9cbf6c1778aa13
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 14 16:31:26 2023 +0100

    refactor payload parsing

    - parameterize file and compression types allowed for files to download
    and upload via config
    - make a real value bag out of QueueMessagePayload and do the parsing
    beforehand
    - refector file extension parser to be more robust

commit 50b578d054ca47a94c907f5f8b585eca7ed626ac
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 14 13:21:32 2023 +0100

    add monitoring

    - add an optional prometheus monitor to monitor the average processing
    time of a service per relevent paramater that is at this point defined
    via the number of resulting elements.

commit de525e7fa2f846f7fde5b9a4b466039238da10cd
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 14 12:57:24 2023 +0100

    fix bug in file extension parser not working if the file endings have prefixes
2023-03-16 16:08:44 +01:00

45 lines
1.5 KiB
Python

import pytest
from pyinfra.config import get_config
from pyinfra.payload_processing.payload import (
QueueMessagePayload,
get_queue_message_payload_parser,
)
from pyinfra.utils.file_extension_parsing import make_file_extension_parser
@pytest.fixture(scope="session")
def payload_config():
return get_config()
class TestPayload:
def test_payload_is_parsed_correctly(self, request_payload, payload_config):
parse_payload = get_queue_message_payload_parser(payload_config)
payload = parse_payload(request_payload)
assert payload == QueueMessagePayload(
dossier_id="test",
file_id="test",
target_file_extension="json.gz",
response_file_extension="json.gz",
target_file_type="json",
target_compression_type="gz",
response_file_type="json",
response_compression_type="gz",
target_file_name="test/test.json.gz",
response_file_name="test/test.json.gz",
)
@pytest.mark.parametrize(
"extension,expected",
[
("json.gz", ("json", "gz")),
("json", ("json", None)),
("prefix.json.gz", ("json", "gz")),
],
)
@pytest.mark.parametrize("allowed_file_types,allowed_compression_types", [(["json", "pdf"], ["gz"])])
def test_parse_file_extension(self, extension, expected, allowed_file_types, allowed_compression_types):
parse = make_file_extension_parser(allowed_file_types, allowed_compression_types)
assert parse(extension) == expected