26 Commits

Author SHA1 Message Date
Julius Unverfehrt
044ea6cf0a feat: streamline download to always include the filename of the downloaded file 2025-01-16 10:29:50 +01:00
Julius Unverfehrt
5c4400aa8b feat:BREAKING CHANGE: download callback no forwards all files as bytes 2025-01-16 10:29:46 +01:00
Jonathan Kössler
d3fecc518e chore: move integration tests to own subfolder 2024-11-18 17:31:15 +01:00
Jonathan Kössler
c5d53b8665 feat: add file comparison 2024-09-24 11:57:33 +02:00
Jonathan Kössler
09d39930e7 chore: cleanup test 2024-09-23 16:43:59 +02:00
Jonathan Kössler
43881de526 feat: add tests for types of documentreader 2024-09-20 16:42:55 +02:00
Jonathan Kössler
b12b1ce42b refactor: use protoc 4.25.x as compiler to avoid dependency issues 2024-07-31 16:04:43 +02:00
Julius Unverfehrt
70d3a210a1 feat: update data loader tests
We now compare the output proto json conversion to expected json files.
This revealed multiple differences between the file.

FIXED: int64 type was cast into string in python. We now get proper
integers

TODO: Empty fields are omitted by proto, but the jsons have them and the
pyinfra implementing services might expect them. We have to test this
behaviour and adjusts the tests accordingly.
2024-07-18 12:36:29 +02:00
Jonathan Kössler
f935056fa9 refactor: dataloader to not crash on unknown file formats 2024-07-17 13:54:50 +02:00
Julius Unverfehrt
0d232226fd feat: integrate proto data loader in pipeline 2024-07-16 17:34:39 +02:00
Julius Unverfehrt
9d55b3be89 feat: implement proto data loader 2024-07-16 16:32:58 +02:00
Julius Unverfehrt
c16000c774 fix(tracing test): make test work in case azure conntection string is missing 2024-07-15 16:13:41 +02:00
Jonathan Kössler
5328e8de03 refactor: streamline tracing types 2024-06-12 10:41:52 +02:00
Julius Unverfehrt
b810449bba feat: add multiple file download
The download function is now overloaded and additionlly supports a
dict with file paths as values, in addition to the present string as
file path. The data is forwarded as dict of the same structure in the
first case.
2024-04-18 16:35:55 +02:00
Julius Unverfehrt
876253b3fb tests: add test for file not found error 2024-04-16 16:19:45 +02:00
Julius Unverfehrt
fff5be2e50 feat(settings): improve config loading logic
Load settings from .toml files, .env and environment variables. Also ensures a ROOT_PATH environment variable is
set. If ROOT_PATH is not set and no root_path argument is passed, the current working directory is used as root.
Settings paths can be a single .toml file, a folder containing .toml files or a list of .toml files and folders.
If a folder is passed, all .toml files in the folder are loaded. If settings path is None, only .env and
environment variables are loaded. If settings_path are relative paths, they are joined with the root_path argument.
2024-01-30 12:56:58 +01:00
Julius Unverfehrt
f6f56b8d8c refactoy: simplify storage connection logic 2024-01-25 09:08:51 +01:00
Isaac Riley
8ff637d6ba chore: add opentelemetry subsection to README.md; formatting 2024-01-25 08:25:19 +01:00
Julius Unverfehrt
e0b32fa448 feat(opentelemetry): fastAPI tracing
The tests don't work yet since the webserver has to run in a thread and
the traces don't get exported from the thread with local json exporting.
However, with an export to an external server this should still work.
WIP
2024-01-24 15:52:42 +01:00
Julius Unverfehrt
da163897c4 feat(opentelemetry): add fastapi instumentation 2024-01-24 14:26:10 +01:00
Julius Unverfehrt
a415666830 feat(opentelemetry): put logic in own module 2024-01-24 14:00:11 +01:00
Julius Unverfehrt
739a7c0731 feat(opentelemetry): add queue instrumenting test 2024-01-24 13:26:01 +01:00
Julius Unverfehrt
adfbd650e6 Add config tests, add type validation to config loading 2024-01-23 08:51:44 +01:00
Julius Unverfehrt
9c2f34e694 Put add health check in own function 2024-01-19 13:13:12 +01:00
Julius Unverfehrt
b7f860f36b WIP: add callback factory and update example scripts 2024-01-18 17:10:04 +01:00
Julius Unverfehrt
6802bf5960 refactor: download and upload file logic, module structure, remove redundant files so far 2024-01-18 15:54:38 +01:00