Merge in RR/pyinfra from RED-5277-backport-REDv3.6 to release/3.6.x
* commit '9f39be70776c486b3387f96b17cd019a46fe2858':
RED-5277: add multi-threading, update config
Merge in RR/pyinfra from feature/MLOPS-23-pyinfra-does-not-use-the-value-of-the-storage_azurecontainername-environment to master
* commit '7a740403bb65db97c8e4cb54de00aac3536b2e4c':
update
update test config
update test config
add pytests to check if a configured bucket can be found
add submodule initialization
load different env vars for the variable depending on the set
Merge in RR/pyinfra from RED-5324 to master
* commit 'ffaa4a668b447fe3f4708d99ec6fccec14f85693':
RED-5324: Added missing storage-region to key to config and to minio-client creation so that storage access works on s3
Merge in RR/pyinfra from RED-5009-fix-ack-bug to master
Squashed commit of the following:
commit 7b00edf6fe1167345e774d658fcd2e60c01d05d5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 24 14:52:57 2022 +0200
RED-5009 pyinfra now truly rejects messages that couldn't be processed by the callback (e.g. inobtainable storage file)
Merge in RR/pyinfra from RED-5009 to master
Squashed commit of the following:
commit 1f8114379bdeb3af8640c71c2edde2a672bb358c
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Aug 22 16:55:04 2022 +0200
RED-5009: Added the possibility for a callback to signal that a message should be declined/dead-lettered
commit be674c2915f6f149c581bc2fe2783217fe424df8
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Aug 19 16:26:38 2022 +0200
RED-5009: Changed callback to not process redelivered messages to prevent endless retries
Merge in RR/pyinfra from add-exists-to-storage to master
Squashed commit of the following:
commit 48d5e1c9e103702bfebfc115e811576514e115c3
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Aug 12 13:32:40 2022 +0200
refactor
commit 711d2c8dbf7c78e26133e3ea3a57670fe829059b
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Aug 12 11:45:42 2022 +0200
add method to check if objects exists for azure and s3
Merge in RR/pyinfra from RED-4653 to master
* commit '7e2cb20040a6be7510f5b06b0a522c1b044d5ee3':
RED-4653: Corrected if-operator
RED-4653: Added value to config to prevent writing the token as a default since that is only useful in a container
RED-4653: Implemented a startup probe for k8s
Merge in RR/pyinfra from RED-4653 to master
* commit '0efbd2c98cecaa1e33991473b1b120827df60ae9':
RED-4653: Removed unnecessary string formatting
RED-4653: Reordered code to prevent errors on application shutdown
RED-4653: Changed code to close only the connection instead of the channel & connection to see if that is sufficient for a clean shutdown
RED-4653: Added some debugging code to test if closing the connection needed
RED-4653: Corrected exception block to not swallow exceptions
RED-4653: Switch to closing channel instead of only cancelling subscription on shutdown.
RED-4653: Corrected signal handler by correctly handling passed params
Changed queue-consumption shutdown to close the channel before closing the connection, since only cancelling the consumers doesn't clean-up the channel correctly, which in turn can cause an error when closing the connection. Also reordered the code so that the connection and channel are only opened when queue-consumption starts.
Merge in RR/pyinfra from tidy_up to master
* commit 'db8f617aa78698760e5aaa198445d349755366a1':
clean up config hygiene; align queue manager and storage signature
Merge in RR/pyinfra from RED-4653 to master
Squashed commit of the following:
commit 14ed6d2ee79f9a6bc4bad187dc775f7476a05d97
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 11:08:16 2022 +0200
RED-4653: Disabled coverage check since there not tests at the moment
commit e926631b167d03e8cc0867db5b5c7d44d6612dcf
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 10:58:50 2022 +0200
RED-4653: Re-added test execution scripts
commit 94648cc449bbc392864197a1796f99f8953b7312
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 10:50:42 2022 +0200
RED-4653: Changed error case for processing messages to not requeue the message since that will be handled in DLQ logic
commit d77982dfedcec49482293d79818283c8d7a17dc7
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 10:46:32 2022 +0200
RED-4653: Removed unnecessary logging message
commit 8c00fd75bf04f8ecc0e9cda654f8e053d4cfb66f
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 10:03:35 2022 +0200
RED-4653: Re-added wrongly removed config
commit 759d72b3fa093b19f97e68d17bf53390cd5453c7
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 09:57:47 2022 +0200
RED-4653: Removed leftover Docker commands
commit 2ff5897ee38e39d6507278b6a82176be2450da16
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 09:48:08 2022 +0200
RED-4653: Removed leftover Docker config
commit 1074167aa98f9f59c0f0f534ba2f1ba09ffb0958
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Tue Jul 26 09:41:21 2022 +0200
RED-4653: Removed Docker build stage since it is not needed for a project that is used as a Python module
commit ec769c6cd74a74097d8ebe4800ea6e2ea86236cc
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Jul 25 16:11:50 2022 +0200
RED-4653: Renamed function for better clarity and consistency
commit 96e8ac4316ac57aac90066f35422d333c532513b
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Jul 25 15:07:40 2022 +0200
RED-4653: Added code to cancel the queue subscription on application exit to queue manager so that it can exit gracefully
commit 64d8e0bd15730898274c08d34f9c34fbac559422
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Jul 25 13:57:06 2022 +0200
RED-4653: Moved queue cancellation to a separate method so that it can be called on application exit
commit aff1d06364f5694c5922f37d961e401c12243221
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Jul 25 11:51:16 2022 +0200
RED-4653: Re-ordered message processing so that ack occurs after publishing the result, to prevent message loss
commit 9339186b86f2fe9653366c22fcdc9f7fc096b138
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 18:07:25 2022 +0200
RED-4653: RED-4653: Reordered code to acknowledge message before publishing a result message
commit 2d6fe1cbd95cd86832b086c6dfbcfa62b3ffa16f
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 17:00:04 2022 +0200
RED-4653: Hopefully corrected storage bucket env var name
commit 8f1ef0dd5532882cb12901721195d9acb336286c
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 16:37:27 2022 +0200
RED-4653: Switched to validating the connection url via a regex since the validators lib parses our endpoints incorrectly
commit 8d0234fcc5ff7ed1ae7695a17856c6af050065bd
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 15:02:54 2022 +0200
RED-4653: Corrected exception creation
commit 098a62335b3b695ee409363d429ac07284de7138
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 14:42:22 2022 +0200
RED-4653: Added a descriptive error message when the storage endpoint is nor a correct url
commit 379685f964a4de641ce6506713f1ea8914a3f5ab
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 22 14:11:48 2022 +0200
RED-4653: Removed variable re-use to make the code clearer
commit 4bf1a023453635568e16b1678ef5ad994c534045
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Thu Jul 21 17:41:55 2022 +0200
RED-4653: Added explicit conversion of the heartbeat config value to an int before passing it to pika
commit 8f2bc4e028aafdef893458d1433a05724f534fce
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Mon Jul 18 16:41:31 2022 +0200
RED-4653: Set heartbeat to lower value so that disconnects are detected more quickly
... and 6 more commits
Merge in RR/pyinfra from RED-4564 to master
Squashed commit of the following:
commit bf4c85a0ab9fed19a44508f2cbef6858cbb32259
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 8 15:46:11 2022 +0200
RED-4564: POC-test to see if cancelling the consumer prevents messages from being stuck
commit 12ebd186b220f263ac2275463b0c124e8f4210fc
Author: Viktor Seifert <viktor.seifert@iqser.com>
Date: Fri Jul 8 14:32:05 2022 +0200
RED-4564: Print full exception with traceback when processing from the queue
Merge in RR/pyinfra from RED-4124-s3-retry to master
Squashed commit of the following:
commit 40e4dc3712fc692115b8274226430a431bf0192f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue May 31 12:59:45 2022 +0200
RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised.
Merge in RR/pyinfra from RED-4049-AMQP-reconnect to master
Squashed commit of the following:
commit 1baafb781c9f623faf4223043e1185fd4ecbbe0e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue May 17 14:36:47 2022 +0200
RED-4049 fix: Pyinfra now retries to reconnect after AMPQ connection loss and eventually throws an error if unsuccessful, forcing container restart.
Merge in RR/pyinfra from storage-data-access-failure-dead-letter-queue-fix to master
Squashed commit of the following:
commit 1f1fcdf0357fe7817b36eac7e369d5ef75ec5af4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon May 2 14:27:29 2022 +0200
fix: storage data access failure now triggers dead letter queue publishing
Merge in RR/pyinfra from aws_region to master
Squashed commit of the following:
commit 29b15be9337fb409f6a0bfbb29026baa4bbe7236
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date: Thu Apr 28 15:08:43 2022 +0200
added aws region to config
Merge in RR/pyinfra from partial_responses to master
Squashed commit of the following:
commit afd67d87a6349c4b97453a12274c6ccf5e976339
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date: Tue Apr 26 12:48:12 2022 +0200
updated test container dockerfile for new location of tests directory
commit 37881da08ebedf0f2d0c6c2b267bdb47818a0da1
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date: Tue Apr 26 12:45:12 2022 +0200
restructuring: moved test out of module scope
Merge in RR/pyinfra from request_as_json_rather_than_data to master
Squashed commit of the following:
commit 6301e00be6d4d76b82b4c153f22271bb3367d3ac
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date: Fri Apr 22 17:02:02 2022 +0200
added test for sending pdf plus metadata in the same request as one json dict
