- Upgrades python version to 3.10 and sync packages with isaacs list.
- Changes loguru logger to kn_utlis logger.
- Overrides python version in CI script (temporarily until all services
are updated and CI template can be adjusted).
If a processing failure occures in the processing callback, pyinfra now
additionally to the exeption prints the stack trace.
Also removes knutils logging for now, since it still contains bugs and
it should be tested first in a non-production environment if
production-readiness is given.
All components from payload processing downwards are tested.
Tests that depend on docker compose have been disabled by default
because they take too long to use during development. Furthermore, the
queue manager tests are not stable, a refactoring with inversion of
control is urgently needed to make the components properly testable. The
storage tests are stable and should be run once before releasing, this
should be implemented via the CI script.
Also adds, if present, tenant Id and operation kwargs to storage and
queue response.
PyInfra now reports exceptions that happen inside the processing
callback.
Also refactors queue manager logging to fit new logger by
changing "%s", var logic to f string, since this syntax is not supported
with knutlis logging.
Introduces new payload parsing logic to be able to process absolute file
paths. The queue message is expected to contain the keys
"targetFilePath" and "responseFilePath".
To ensure backward-compatibility, the legacy "dossierId", "fileId"
messages are still supported.
Merge in RR/pyinfra from bugfix/RED-6273-forward-processing-kwargs to master
Squashed commit of the following:
commit 2f45f7329dc6fd6166e08bad720e022e722737ad
Merge: 0a6d5df 0f4646e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:55:24 2023 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into bugfix/RED-6273-forward-processing-kwargs
commit 0a6d5dfc1a6edd8e6d171b50334b812a79f9288d
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:51:05 2023 +0200
update pyinfra version
commit cd417c4b515d2a5d190af883af770bc660e15bb8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:48:12 2023 +0200
Revert poetry update
- adds strange rust dependency for some reason
Merge in RR/pyinfra from bugfix/RED-6273-forward-operation-kwargs to master
Squashed commit of the following:
commit 347add07f8ea6e085064660ae79f0df9013dd9d6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:16:41 2023 +0200
update pyinfra version
commit 3c17047377aca666a015eaf0f06190d3dfa28c1c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:13:59 2023 +0200
fix monitoring preventing operation kwargs for processing fn getting forwarded
Merge in RR/pyinfra from RED-6273-multi-tenant-storage to master
Squashed commit of the following:
commit 0fead1f8b59c9187330879b4e48d48355885c27c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 15:02:22 2023 +0200
fix typos
commit 892a803726946876f8b8cd7905a0e73c419b2fb1
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Tue Mar 28 14:41:49 2023 +0200
Refactoring
Replace custom storage caching logic with LRU decorator
commit eafcd90260731e3360ce960571f07dee8f521327
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 12:50:13 2023 +0100
fix bug in storage connection from endpoint
commit d0c9fb5b7d1c55ae2f90e8faa1efec9f7587c26a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 11:49:34 2023 +0100
add logs to PayloadProcessor
- set log messages to determine if x-tenant
storage connection is working
commit 97309fe58037b90469cf7a3de342d4749a0edfde
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 10:41:59 2023 +0100
update PayloadProcessor
- introduce storage cache to make every unique
storage connection only once
- add functionality to pass optional processing
kwargs in queue message like the operation key to
the processing function
commit d48e8108fdc0d463c89aaa0d672061ab7dca83a0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Mar 22 13:34:43 2023 +0100
add multi-tenant storage connection 1st iteration
- forward x-tenant-id from queue message header to
payload processor
- add functions to receive storage infos from an
endpoint or the config. This enables hashing and
caching of connections created from these infos
- add function to initialize storage connections
from storage infos
- streamline and refactor tests to make them more
readable and robust and to make it easier to add
new tests
- update payload processor with first iteration
of multi tenancy storage connection support
with connection caching and backwards compability
commit 52c047c47b98e62d0b834a9b9b6c0e2bb0db41e5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:35:57 2023 +0100
add AES/GCM cipher functions
- decrypt x-tenant storage connection strings
Merge in RR/pyinfra from bugfix/RED-6205-prometheus-port to master
Squashed commit of the following:
commit e97d81bebfe34c24d8da4e4392ff7dbd3638e685
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:48:04 2023 +0100
increase package version
commit c7e181a462e275c5f2cbf1e6df4c88dfefbe36b7
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:43:46 2023 +0100
fix prometheus address
- change loopback address to all available network interfaces to enable
external metric scraping
- disable ENV input for prometheus address and port since they should
not be set in HELM
Merge in RR/pyinfra from add-safety-measure to master
* commit 'b985679d6b30b3a983c7b1df5fb23eef0dc95cd3':
add safety mesasure for monitoring in case a service didn't find any results.
