61 Commits

Author SHA1 Message Date
Jonathan Kössler
3a04359320 chore: bump pyinfra version 2024-09-25 11:59:52 +02:00
Jonathan Kössler
b46fcbd977 feat: add AioPikaInstrumentor 2024-09-25 11:58:51 +02:00
Francisco Schulz
8ec13502a9 fix: add semaphore to AsyncQueueManager to limit concurrent tasks 2024-09-23 15:19:40 +02:00
francisco.schulz
5b45cae9a0 chore: update package version 2024-09-02 10:53:09 -04:00
francisco.schulz
4c8dc6ccc0 fix(temp): set protobuf version range to >=v3,<v4 so image-classification model keeps working 2024-08-30 08:37:31 -04:00
francisco.schulz
88aef57c5f chore: version increase 2024-08-29 11:18:36 -04:00
Jonathan Kössler
b6a2069a6a feat: add ClientConnectorError 2024-08-28 10:28:12 +02:00
Jonathan Kössler
f27b1fbba1 chore: bump version 2024-08-23 16:56:54 +02:00
Jonathan Kössler
564f2cbb43 chore: bump version 2024-08-21 16:25:17 +02:00
Jonathan Kössler
dba348a621 refactor: tenant queues settings 2024-08-19 14:37:48 +02:00
Jonathan Kössler
2bc332831e fix: use is_initialized instead of is_open 2024-08-16 12:37:28 +02:00
Jonathan Kössler
b3f1529be2 chore: bump version 2024-08-06 09:48:09 +02:00
Jonathan Kössler
aeac1c58f9 chore: bump pyinfra version 2024-07-31 16:05:42 +02:00
Jonathan Kössler
b12b1ce42b refactor: use protoc 4.25.x as compiler to avoid dependency issues 2024-07-31 16:04:43 +02:00
Jonathan Kössler
f3d0f24ea6 Merge branch 'master' into feat/RES-757-protobuffer 2024-07-30 10:40:56 +02:00
Jonathan Kössler
2a2028085e feat: add async retry for tenant server calls 2024-07-25 14:45:19 +02:00
Jonathan Kössler
28451e8f8f chore: bump pyinfra version 2024-07-22 16:54:28 +02:00
Julius Unverfehrt
ceac21c1ef deps: add dvc 2024-07-16 17:35:03 +02:00
Julius Unverfehrt
c5d8a6ed84 feat: add proto requirements and instructions to readme for compiling the schemata 2024-07-16 16:30:32 +02:00
francisco.schulz
aa23894858 chose(dependencies): update 2024-07-11 11:55:17 -04:00
Jonathan Kössler
b2e3ae092f feat: wip for multiple tenants 2024-07-09 18:20:55 +02:00
Jonathan Kössler
2484a5e9f7 chore: bump pyinfra version 2024-06-17 13:53:42 +02:00
Jonathan Kössler
7dbcdf1650 feat: add azure opentelemtry monitoring 2024-06-11 12:00:18 +02:00
Julius Unverfehrt
a1e7b3b565 build: add SBOM and increment package version 2024-04-18 16:39:46 +02:00
Julius Unverfehrt
ed4f912acf build: increment service version 2024-04-16 16:21:57 +02:00
Julius Unverfehrt
1689cd762b fix(CI): fix CI 2024-01-31 12:03:07 +01:00
Julius Unverfehrt
c1301d287f fix(dependencies): move opentel deps to main since groups are not packaged with CI script 2024-01-30 14:31:08 +01:00
Julius Unverfehrt
ec9ab21198 package: increment major version and update kn-utils 2024-01-25 11:08:50 +01:00
Julius Unverfehrt
da163897c4 feat(opentelemetry): add fastapi instumentation 2024-01-24 14:26:10 +01:00
Isaac Riley
936bb4fe80 feat: add opentelemetry on top of newly refactored pyinfra 2024-01-24 08:09:42 +01:00
Julius Unverfehrt
6802bf5960 refactor: download and upload file logic, module structure, remove redundant files so far 2024-01-18 15:54:38 +01:00
Julius Unverfehrt
358e227251 fix prometheus tests WIP 2024-01-17 17:39:53 +01:00
Julius Unverfehrt
f31693d36a refactor: adapt prometheus monitoring logic to work with other webservers WIP 2024-01-16 17:24:53 +01:00
francisco.schulz
570689ed9b increment version 2023-11-28 09:35:46 +01:00
francisco.schulz
3084d6338c update dependencies for kn-utils@0.2.4.dev112 2023-11-28 09:16:05 +01:00
Julius Unverfehrt
bb00c83a80 Upgrade python version & change logger
- Upgrades python version to 3.10 and sync packages with isaacs list.
- Changes loguru logger to kn_utlis logger.
- Overrides python version in CI script (temporarily until all services
  are updated and CI template can be adjusted).
2023-11-13 15:28:49 +01:00
Julius Unverfehrt
261b991049 Add stacktrace to processing failures
If a processing failure occures in the processing callback, pyinfra now
additionally to the exeption prints the stack trace.

Also removes knutils logging for now, since it still contains bugs and
it should be tested first in a non-production environment if
production-readiness is given.
2023-09-05 12:59:45 +02:00
Julius Unverfehrt
84c4e7601f Update kn-utils package
Update kn-utils for missing loglevels fix, which is needed for queue
manager error logging.
2023-08-30 15:58:29 +02:00
Julius Unverfehrt
7187f0ec0c RES-343 Update logging to knutils logger 2023-08-22 10:46:14 +02:00
francisco.schulz
daddec7dc3 increment version 2023-07-18 16:59:50 +02:00
francisco.schulz
370e978fa7 upgrade dependencies, allow python>=3.8 2023-07-18 16:54:29 +02:00
francisco.schulz
4e6b4e2969 update dependencies 2023-06-20 17:13:26 +02:00
Julius Unverfehrt
c4e03d4641 Fix: New tenant storage information endpoint
Parametrize tenant enpoint and publick decryption key as environment
variable and set the default value to new endpoint.
2023-06-15 16:22:30 +02:00
francisco.schulz
5ed41a392a update version number 2023-05-16 14:18:46 +02:00
francisco.schulz
4a0c59b070 update deps 2023-05-16 13:42:53 +02:00
francisco.schulz
309119cb62 update version 2023-04-18 15:48:50 +02:00
Julius Unverfehrt
d838413500 Pull request #70: Bugfix/RED-6273 forward processing kwargs
Merge in RR/pyinfra from bugfix/RED-6273-forward-processing-kwargs to master

Squashed commit of the following:

commit 2f45f7329dc6fd6166e08bad720e022e722737ad
Merge: 0a6d5df 0f4646e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 17:55:24 2023 +0200

    Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into bugfix/RED-6273-forward-processing-kwargs

commit 0a6d5dfc1a6edd8e6d171b50334b812a79f9288d
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 17:51:05 2023 +0200

    update pyinfra version

commit cd417c4b515d2a5d190af883af770bc660e15bb8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 17:48:12 2023 +0200

    Revert poetry update

    - adds strange rust dependency for some reason
2023-03-28 17:57:20 +02:00
Julius Unverfehrt
0f4646e390 Pull request #69: fix monitoring preventing operation kwargs for processing fn getting forwarded
Merge in RR/pyinfra from bugfix/RED-6273-forward-operation-kwargs to master

Squashed commit of the following:

commit 347add07f8ea6e085064660ae79f0df9013dd9d6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 17:16:41 2023 +0200

    update pyinfra version

commit 3c17047377aca666a015eaf0f06190d3dfa28c1c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 17:13:59 2023 +0200

    fix monitoring preventing operation kwargs for processing fn getting forwarded
2023-03-28 17:17:09 +02:00
Julius Unverfehrt
793a427c50 Pull request #68: RED-6273 multi tenant storage
Merge in RR/pyinfra from RED-6273-multi-tenant-storage to master

Squashed commit of the following:

commit 0fead1f8b59c9187330879b4e48d48355885c27c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 28 15:02:22 2023 +0200

    fix typos

commit 892a803726946876f8b8cd7905a0e73c419b2fb1
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date:   Tue Mar 28 14:41:49 2023 +0200

    Refactoring

    Replace custom storage caching logic with LRU decorator

commit eafcd90260731e3360ce960571f07dee8f521327
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Fri Mar 24 12:50:13 2023 +0100

    fix bug in storage connection from endpoint

commit d0c9fb5b7d1c55ae2f90e8faa1efec9f7587c26a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Fri Mar 24 11:49:34 2023 +0100

    add logs to PayloadProcessor

    - set log messages to determine if x-tenant
    storage connection is working

commit 97309fe58037b90469cf7a3de342d4749a0edfde
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Fri Mar 24 10:41:59 2023 +0100

    update PayloadProcessor

    - introduce storage cache to make every unique
    storage connection only once
    - add functionality to pass optional processing
    kwargs in queue message like the operation key to
    the processing function

commit d48e8108fdc0d463c89aaa0d672061ab7dca83a0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Mar 22 13:34:43 2023 +0100

    add multi-tenant storage connection 1st iteration

    - forward x-tenant-id from queue message header to
    payload processor
    - add functions to receive storage infos from an
    endpoint or the config. This enables hashing and
    caching of connections created from these infos
    - add function to initialize storage connections
    from storage infos
    - streamline and refactor tests to make them more
    readable and robust and to make it easier to add
     new tests
    - update payload processor with first iteration
    of multi tenancy storage connection support
    with connection caching and backwards compability

commit 52c047c47b98e62d0b834a9b9b6c0e2bb0db41e5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 21 15:35:57 2023 +0100

    add AES/GCM cipher functions

    - decrypt x-tenant storage connection strings
2023-03-28 15:04:14 +02:00
Julius Unverfehrt
0f24a7f26d Pull request #67: fix prometheus address
Merge in RR/pyinfra from bugfix/RED-6205-prometheus-port to master

Squashed commit of the following:

commit e97d81bebfe34c24d8da4e4392ff7dbd3638e685
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 21 15:48:04 2023 +0100

    increase package version

commit c7e181a462e275c5f2cbf1e6df4c88dfefbe36b7
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 21 15:43:46 2023 +0100

    fix prometheus address

    - change loopback address to all available network interfaces to enable
    external metric scraping
    - disable ENV input for prometheus address and port since they should
    not be set in HELM
2023-03-21 15:54:47 +01:00