Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
236 Commits 9 Branches 411 Tags
Commit Graph

236 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Julius Unverfehrt
b2f073e0c5 refactor: IoC for callback, update readme 2024-01-25 10:41:48 +01:00
Julius Unverfehrt
f6f56b8d8c refactoy: simplify storage connection logic 2024-01-25 09:08:51 +01:00
Isaac Riley
8ff637d6ba chore: add opentelemetry subsection to README.md; formatting 2024-01-25 08:25:19 +01:00
Julius Unverfehrt
c18475a77d feat(opentelemetry): improve readability 2024-01-24 17:46:54 +01:00
Julius Unverfehrt
e0b32fa448 feat(opentelemetry): fastAPI tracing 
The tests don't work yet since the webserver has to run in a thread and
the traces don't get exported from the thread with local json exporting.
However, with an export to an external server this should still work.
WIP
 2024-01-24 15:52:42 +01:00
Julius Unverfehrt
da163897c4 feat(opentelemetry): add fastapi instumentation 2024-01-24 14:26:10 +01:00
Julius Unverfehrt
a415666830 feat(opentelemetry): put logic in own module 2024-01-24 14:00:11 +01:00
Julius Unverfehrt
739a7c0731 feat(opentelemetry): add queue instrumenting test 2024-01-24 13:26:01 +01:00
Isaac Riley
936bb4fe80 feat: add opentelemetry on top of newly refactored pyinfra 2024-01-24 08:09:42 +01:00
Julius Unverfehrt
725d6dce45 Update readme 2024-01-23 18:08:57 +01:00
Julius Unverfehrt
be602d8411 Adjust logs 2024-01-23 14:10:56 +01:00
Julius Unverfehrt
429a85b609 Disable automated tests until we found a way to rund docker compose before 2024-01-23 10:26:44 +01:00
Julius Unverfehrt
d6eeb65ccc Update scripts 2024-01-23 10:25:56 +01:00
Julius Unverfehrt
adfbd650e6 Add config tests, add type validation to config loading 2024-01-23 08:51:44 +01:00
Julius Unverfehrt
73eba97ede Add serving example 
TODO: - update readme
      - check if logs are adequate
 2024-01-19 14:53:06 +01:00
Julius Unverfehrt
8cd1d6b283 add retries to queue consuming, so we retray at least a bit if something happens. Eventually the container should crash though since there do exist unfixable problems sadly. 2024-01-19 14:15:00 +01:00
Julius Unverfehrt
87cbf89672 finnish config loading logic 2024-01-19 14:05:05 +01:00
Julius Unverfehrt
9c2f34e694 Put add health check in own function 2024-01-19 13:13:12 +01:00
Julius Unverfehrt
fbbfc553ae fix message encoding for response, rename some functions 2024-01-19 12:46:02 +01:00
Julius Unverfehrt
b7f860f36b WIP: add callback factory and update example scripts 2024-01-18 17:10:04 +01:00
Julius Unverfehrt
6802bf5960 refactor: download and upload file logic, module structure, remove redundant files so far 2024-01-18 15:54:38 +01:00
Julius Unverfehrt
ec5ad09fa8 refactor: multi tenant storage connection 2024-01-18 11:34:21 +01:00
Julius Unverfehrt
17c5eebdf6 finnish prometheus 2024-01-18 08:19:46 +01:00
Julius Unverfehrt
358e227251 fix prometheus tests WIP 2024-01-17 17:39:53 +01:00
Julius Unverfehrt
f31693d36a refactor: adapt prometheus monitoring logic to work with other webservers WIP 2024-01-16 17:24:53 +01:00
Julius Unverfehrt
e5c8a6e9f1 refactor: update storages with dynaconf logic, add validators, repair test 2024-01-16 15:34:56 +01:00
Julius Unverfehrt
27917863c9 refactor: finnish queue manager, queue manager tests, also add validation logic, integrate new settings 2024-01-16 14:35:23 +01:00
Julius Unverfehrt
ebc519ee0d refactor: finnish queue manager, queue manager tests, also add validation logic, integrate new settings 2024-01-16 14:16:27 +01:00
Julius Unverfehrt
b49645cce4 refactor: queue manager and config logic WIP 2024-01-15 16:46:33 +01:00
Julius Unverfehrt
64871bbb62 refactor: add basic queue manager test 2024-01-15 10:30:07 +01:00
Julius Unverfehrt
1f482f2476 fix: storage test 2024-01-09 16:07:48 +01:00
Francisco Schulz
8dfba74682 Merge branch 'RED-7958-logging-issues-of-python-services' into 'master' 
Red 7958 logging issues of python services

See merge request knecon/research/pyinfra!81
1.10.0 		2023-11-28 10:21:37 +01:00
francisco.schulz
570689ed9b increment version 2023-11-28 09:35:46 +01:00
francisco.schulz
5db56d8449 update CI template 2023-11-28 09:20:22 +01:00
francisco.schulz
3a9d34f9c0 add loglevel tests & fix broken exception and error log tests 2023-11-28 09:16:42 +01:00
francisco.schulz
3084d6338c update dependencies for kn-utils@0.2.4.dev112 2023-11-28 09:16:05 +01:00
Julius Unverfehrt
3a3a8e4ce1 Merge branch 'feature/version-upgrade-knutils-logging' into 'master' 
Upgrade python version & change logger

See merge request knecon/research/pyinfra!80
1.9.0 		2023-11-13 15:48:22 +01:00
Julius Unverfehrt
bb00c83a80 Upgrade python version & change logger 
- Upgrades python version to 3.10 and sync packages with isaacs list.
- Changes loguru logger to kn_utlis logger.
- Overrides python version in CI script (temporarily until all services
  are updated and CI template can be adjusted).
 2023-11-13 15:28:49 +01:00
Julius Unverfehrt
b297894505 Merge branch 'feature/stack-trace-for-exeptions' into 'master' 
Add stacktrace to processing failures

See merge request knecon/research/pyinfra!79
1.8.0 		2023-09-05 13:04:00 +02:00
Julius Unverfehrt
261b991049 Add stacktrace to processing failures 
If a processing failure occures in the processing callback, pyinfra now
additionally to the exeption prints the stack trace.

Also removes knutils logging for now, since it still contains bugs and
it should be tested first in a non-production environment if
production-readiness is given.
 2023-09-05 12:59:45 +02:00
Julius Unverfehrt
84c4e7601f Update kn-utils package 
Update kn-utils for missing loglevels fix, which is needed for queue
manager error logging.
1.7.0 		2023-08-30 15:58:29 +02:00
Julius Unverfehrt
201ed5b9a8 Merge branch 'feature/RED-6685-support-absolute-paths' into 'master' 
Add support for absolute file paths

See merge request knecon/research/pyinfra!77
1.6.0 		2023-08-23 14:11:46 +02:00
Julius Unverfehrt
72547201f3 Adjust log levels to reduce log clutter 
Also updates readme and adds pytest execution to CI script.
 2023-08-23 12:38:34 +02:00
Julius Unverfehrt
c09476cfae Update tests 
All components from payload processing downwards are tested.

Tests that depend on docker compose have been disabled by default
because they take too long to use during development. Furthermore, the
queue manager tests are not stable, a refactoring with inversion of
control is urgently needed to make the components properly testable. The
storage tests are stable and should be run once before releasing, this
should be implemented via the CI script.

Also adds, if present, tenant Id and operation kwargs to storage and
queue response.
 2023-08-22 17:33:22 +02:00
Julius Unverfehrt
e580a66347 Refactor storage provider & payload parser 
Applies strategy pattern to payload parsing logic to improve
maintainability and testability.
Renames storage manager to storage provider.
 2023-08-22 10:46:27 +02:00
Julius Unverfehrt
294688ea66 RED-7002 Forward exceptions from thread context 
PyInfra now reports exceptions that happen inside the processing
callback.
Also refactors queue manager logging to fit new logger by
changing "%s", var logic to f string, since this syntax is not supported
with knutlis logging.
 2023-08-22 10:46:27 +02:00
Julius Unverfehrt
7187f0ec0c RES-343 Update logging to knutils logger 2023-08-22 10:46:14 +02:00
Julius Unverfehrt
ef916ee790 Refactor payload processing logic 
Streamlines payload processor class by encapsulating closely dependent
logic, to improve readability and maintainability.
 2023-08-18 12:49:21 +02:00
Julius Unverfehrt
48d74b4307 Add support for absolute file paths 
Introduces new payload parsing logic to be able to process absolute file
paths. The queue message is expected to contain the keys
"targetFilePath" and "responseFilePath".

To ensure backward-compatibility, the legacy "dossierId", "fileId"
messages are still supported.
 2023-08-18 12:45:53 +02:00
Francisco Schulz
692ff204c3 Merge branch 'bugfix/RES-269' into 'master' 
Bugfix/res 269

See merge request knecon/research/pyinfra!75
1.5.10 		2023-08-17 09:55:27 +02:00