Merge in RR/pyinfra from bugfix/RED-6273-forward-processing-kwargs to master
Squashed commit of the following:
commit 2f45f7329dc6fd6166e08bad720e022e722737ad
Merge: 0a6d5df 0f4646e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:55:24 2023 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into bugfix/RED-6273-forward-processing-kwargs
commit 0a6d5dfc1a6edd8e6d171b50334b812a79f9288d
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:51:05 2023 +0200
update pyinfra version
commit cd417c4b515d2a5d190af883af770bc660e15bb8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:48:12 2023 +0200
Revert poetry update
- adds strange rust dependency for some reason
Merge in RR/pyinfra from bugfix/RED-6273-forward-operation-kwargs to master
Squashed commit of the following:
commit 347add07f8ea6e085064660ae79f0df9013dd9d6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:16:41 2023 +0200
update pyinfra version
commit 3c17047377aca666a015eaf0f06190d3dfa28c1c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 17:13:59 2023 +0200
fix monitoring preventing operation kwargs for processing fn getting forwarded
Merge in RR/pyinfra from RED-6273-multi-tenant-storage to master
Squashed commit of the following:
commit 0fead1f8b59c9187330879b4e48d48355885c27c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 28 15:02:22 2023 +0200
fix typos
commit 892a803726946876f8b8cd7905a0e73c419b2fb1
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Tue Mar 28 14:41:49 2023 +0200
Refactoring
Replace custom storage caching logic with LRU decorator
commit eafcd90260731e3360ce960571f07dee8f521327
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 12:50:13 2023 +0100
fix bug in storage connection from endpoint
commit d0c9fb5b7d1c55ae2f90e8faa1efec9f7587c26a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 11:49:34 2023 +0100
add logs to PayloadProcessor
- set log messages to determine if x-tenant
storage connection is working
commit 97309fe58037b90469cf7a3de342d4749a0edfde
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 24 10:41:59 2023 +0100
update PayloadProcessor
- introduce storage cache to make every unique
storage connection only once
- add functionality to pass optional processing
kwargs in queue message like the operation key to
the processing function
commit d48e8108fdc0d463c89aaa0d672061ab7dca83a0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Mar 22 13:34:43 2023 +0100
add multi-tenant storage connection 1st iteration
- forward x-tenant-id from queue message header to
payload processor
- add functions to receive storage infos from an
endpoint or the config. This enables hashing and
caching of connections created from these infos
- add function to initialize storage connections
from storage infos
- streamline and refactor tests to make them more
readable and robust and to make it easier to add
new tests
- update payload processor with first iteration
of multi tenancy storage connection support
with connection caching and backwards compability
commit 52c047c47b98e62d0b834a9b9b6c0e2bb0db41e5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:35:57 2023 +0100
add AES/GCM cipher functions
- decrypt x-tenant storage connection strings
Merge in RR/pyinfra from bugfix/RED-6205-prometheus-port to master
Squashed commit of the following:
commit e97d81bebfe34c24d8da4e4392ff7dbd3638e685
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:48:04 2023 +0100
increase package version
commit c7e181a462e275c5f2cbf1e6df4c88dfefbe36b7
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 21 15:43:46 2023 +0100
fix prometheus address
- change loopback address to all available network interfaces to enable
external metric scraping
- disable ENV input for prometheus address and port since they should
not be set in HELM
Merge in RR/pyinfra from RED-6205-monitoring to master
Squashed commit of the following:
commit 529cedfd7c065a3f7364e4596b923f25f0af76b5
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Thu Mar 16 14:57:26 2023 +0100
Remove unnecessary default argument to dict.get
commit b718531f568e89df77cc05039e5e7afe7111b9a4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 16 14:56:50 2023 +0100
refactor
commit c039b0c25a6cd2ad2a72d237d0930c484c8e427c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 16 13:22:17 2023 +0100
increase package version to reflect the recent changes
commit 0a983a4113f25cd692b68869e1f33ffbf7efc6f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 16 13:16:39 2023 +0100
remove processing result conversion to a list, since ner-predicion service actually returns a dictionary. It is now expected that the result is sized to perform the monitoring and json dumpable to upload it.
commit 541bf321410471dc09a354669b2778402286c09f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 16 12:48:07 2023 +0100
remove no longer needed requirements
commit cfa182985d989a5b92a9a069a603daee72f37d49
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 16 11:14:58 2023 +0100
refactor payload formatting
- introduce PayloadFormatter class for better typehinting and bundling
of functionality
- parametrize payload formatting so the PayloadProcesser can adapt
better to differnt services/products
- move file extension parsing to its own module
commit f57663b86954b7164eeb6db013d862af88ec4584
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Mar 15 12:22:08 2023 +0100
refactor payload parsing
- introduce QueueMessagePayloadParser for generality
and typehinting
- refactor file extension parsing algorithm
commit 713fb4a0dddecf5442ceda3988444d9887869dcf
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 14 17:07:02 2023 +0100
fix tests
commit a22ecf7ae93bc0bec235fba3fd9cbf6c1778aa13
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 14 16:31:26 2023 +0100
refactor payload parsing
- parameterize file and compression types allowed for files to download
and upload via config
- make a real value bag out of QueueMessagePayload and do the parsing
beforehand
- refector file extension parser to be more robust
commit 50b578d054ca47a94c907f5f8b585eca7ed626ac
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 14 13:21:32 2023 +0100
add monitoring
- add an optional prometheus monitor to monitor the average processing
time of a service per relevent paramater that is at this point defined
via the number of resulting elements.
commit de525e7fa2f846f7fde5b9a4b466039238da10cd
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Mar 14 12:57:24 2023 +0100
fix bug in file extension parser not working if the file endings have prefixes
Merge in RR/pyinfra from RED-6366-refactor to master
Squashed commit of the following:
commit 8807cda514b5cc24b1be208173283275d87dcb97
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 13:15:15 2023 +0100
enable docker-compose autouse for automatic tests
commit c4579581d3e9a885ef387ee97f3f3a5cf4731193
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 12:35:49 2023 +0100
black
commit ac2b754c5624ef37ce310fce7196c9ea11bbca03
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 12:30:23 2023 +0100
refactor storage url parsing
- move parsing and validation to config where the connection url is
actually read in
- improve readability of parsing fn
commit 371802cc10b6d946c4939ff6839571002a2cb9f4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 10:48:00 2023 +0100
refactor
commit e8c381c29deebf663e665920752c2965d7abce16
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 09:57:34 2023 +0100
rename
commit c8628a509316a651960dfa806d5fe6aacb7a91c1
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 09:37:01 2023 +0100
renaming and refactoring
commit 4974d4f56fd73bc55bd76aa7a9bbb16babee19f4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 10 08:53:09 2023 +0100
refactor payload processor
- limit make_uploader and make_downloader cache
- partially apply them when the class is initialized with storage and
bucket to make the logic and behaviour more comprehensive
- renaming functional pipeline steps to be more expressive
commit f8d51bfcad2b815c8293ab27dd66b256255c5414
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 9 15:30:32 2023 +0100
remove monitor and rename Payload
commit 412ddaa207a08aff1229d7acd5d95402ac8cd578
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Mar 2 10:15:39 2023 +0100
remove azure connection string and disable respective test for now for security reasons
commit 7922a2d9d325f3b9008ad4e3e56b241ba179f52c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Mar 1 13:30:58 2023 +0100
make payload formatting function names more expressive
commit 7517e544b0f5a434579cc9bada3a37e7ac04059f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Mar 1 13:24:57 2023 +0100
add some type hints
commit 095410d3009f2dcbd374680dd0f7b55de94c9e76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Wed Mar 1 10:54:58 2023 +0100
Refactoring
- Renaming
- Docstring adjustments
commit e992f0715fc2636eb13eb5ffc4de0bcc5d433fc8
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Wed Mar 1 09:43:26 2023 +0100
Re-wording and typo fixes
commit 3c2d698f9bf980bc4b378a44dc20c2badc407b3e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Feb 28 14:59:59 2023 +0100
enable auto startup for docker compose in tests
commit 55773b4fb0b624ca4745e5b8aeafa6f6a0ae6436
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Feb 28 14:59:37 2023 +0100
Extended tests for queue manager
commit 14f7f943f60b9bfb9fe77fa3cef99a1e7d094333
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Feb 28 13:39:00 2023 +0100
enable auto startup for docker compose in tests
commit 7caf354491c84c6e0b0e09ad4d41cb5dfbfdb225
Merge: 49d47ba d0277b8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Feb 28 13:32:52 2023 +0100
Merge branch 'RED-6205-prometheus' of ssh://git.iqser.com:2222/rr/pyinfra into RED-6205-prometheus
commit 49d47baba8ccf11dee48a4c1cbddc3bbd12471e5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Feb 28 13:32:42 2023 +0100
adjust Payload Processor signature
commit d0277b86bc54994b6032774bf0ec2d7b19d7f517
Merge: 5184a18 f6b35d6
Author: Christoph Schabert <christoph.schabert@iqser.com>
Date: Tue Feb 28 11:07:16 2023 +0100
Pull request #61: Change Sec Trigger to PR
Merge in RR/pyinfra from cschabert/PlanSpecjava-1677578703647 to RED-6205-prometheus
* commit 'f6b35d648c88ddbce1856445c3b887bce669265c':
Change Sec Trigger to PR
commit f6b35d648c88ddbce1856445c3b887bce669265c
Author: Christoph Schabert <christoph.schabert@iqser.com>
Date: Tue Feb 28 11:05:13 2023 +0100
Change Sec Trigger to PR
... and 20 more commits
Merge in RR/pyinfra from RED-6118-multi-tenancy-patch to master
Squashed commit of the following:
commit 02e471622e59baf5d2bb5c61980cea43ca1c6d61
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Feb 16 16:36:19 2023 +0100
move acknowledgment function to outer scope
commit f9efffd8e6d90d5e371c66574b1afe361a1da146
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Feb 16 16:04:07 2023 +0100
adjust response headers
- change response formatting: only forward the
request message headers instead of all properties
- adjust build script to only increase patch
version on master push
Merge in RR/pyinfra from bugfix/RED-5277-investigate-missing-heartbeat-error to master
Squashed commit of the following:
commit 9e139e79e46c52014986f9afb2c6534281b55c10
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Wed Feb 15 14:56:44 2023 +0100
RED-5277: Moved async processing to its own functions
commit 244a941299dbf75b254adcad8b068b2917c6bf79
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 11:26:00 2023 +0100
Revert "only set git tag on release and master branches"
This reverts commit 9066856d223f0646723fa1c62c444e16a9bb3ce9.
commit adb35db6fa6daf4b79263a918716c34905e8b3bc
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 11:11:07 2023 +0100
increment version
commit 9066856d223f0646723fa1c62c444e16a9bb3ce9
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 11:10:49 2023 +0100
only set git tag on release and master branches
commit ee11e018efdbc63a740008e7fa2415cbb12476ae
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 10:18:08 2023 +0100
configure root logger in `__init__.py`
only set log levels for other loggers, inherit config
commit 776399912ddf1e936138cceb2af981f27d333823
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 10:16:57 2023 +0100
update dependency via `poetry update`
commit 804a8d9fbd1ded3e154fe9b3cafa32428522ca0f
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Wed Feb 15 10:16:25 2023 +0100
increment version
commit cf057daed23d5f5b0f6f3a1a31e956e015e86368
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:59:55 2023 +0100
update
commit 51717d85fce592b8bf38a8b5235faa04379cce1a
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:48:51 2023 +0100
define sonar source
commit ace57c211a61d8e473a700da161806f882b19dc6
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:46:24 2023 +0100
update plan
commit 1fcc00eb18ed692e2646873b4a233a00b5f6d93b
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:46:13 2023 +0100
fix typo
commit 20b59768a68d985e1bf2fe6f93a1e6283bac5cb0
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:43:39 2023 +0100
increment version
commit 8e7b4bf302b5591b2c490ad89c8a01a87c5b4741
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 17:11:59 2023 +0100
get rid of extra logger
commit 3fd3eb255c252d1e208b88b475ec8a07c521619d
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 16:45:56 2023 +0100
increment version
commit b0b5e5ebd94554cdafed6cff333d73a9ba08bea1
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 16:40:22 2023 +0100
update
commit b87b3c351722d6949833c397178bc0354c754d90
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 16:38:41 2023 +0100
fix tag issue from build
commit 73f3dcb280b6f905eeef3c69123b1252e6c934b1
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 14:21:57 2023 +0100
add comments & update logging
commit 72a9e2c51f5bf98fc9f0803183fc1d28aaea9e35
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 12:06:09 2023 +0100
cleanup comments
commit 587814944921f0f148e4d3c4c76d4edffff55bba
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 11:16:17 2023 +0100
use thread executor in a `with` statement
commit 9561a6b447d98d2f0d536f63c0946d7bf1e2ca7d
Author: Francisco Schulz <Francisco.Schulz@iqser.com>
Date: Tue Feb 14 10:42:49 2023 +0100
fix unbound issue `callback_result` & shutdown thread executor
... and 23 more commits
