Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
330 Commits 9 Branches 411 Tags
Commit Graph

330 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Julius Unverfehrt
e7ee0cda42 add compression for storage item before upload, update script for extraction 2022-06-02 15:49:28 +02:00
Julius Unverfehrt
90f8f9da36 update script for extraction 2022-06-02 15:12:33 +02:00
Julius Unverfehrt
c2d7127a84 add log for Consumer Error, fix page index hash function 2022-06-02 14:15:03 +02:00
Matthias Bisping
bfe8bbb8cb reorganized queue message metadata in request 2022-06-01 16:00:46 +02:00
Matthias Bisping
ecff50ae7c submit endpoint is now 'submit' or 'operation' 2022-06-01 11:46:51 +02:00
Matthias Bisping
7a1b215d69 removed obsolete code 2022-06-01 11:37:47 +02:00
Matthias Bisping
01ce914417 generalizing server setup from operations WIP 2022-06-01 10:58:44 +02:00
Matthias Bisping
2b72174605 Merge branch 'integration_test_in_order_develop_aggregation_stratgey' of ssh://git.iqser.com:2222/rr/pyinfra into integration_test_in_order_develop_aggregation_stratgey 2022-05-31 18:40:47 +02:00
Matthias Bisping
586871a26f added queue message body to analysis input dict 2022-05-31 18:40:40 +02:00
Julius Unverfehrt
187055e5eb adjust mock_client script for conversion 2022-05-31 18:19:41 +02:00
Matthias Bisping
3046b4dc26 misc minor fixes while integrating with pdf2image 2022-05-31 17:58:28 +02:00
Matthias Bisping
dd591bd24b removed obsolete code 2022-05-31 16:23:14 +02:00
Matthias Bisping
12fa52f38c removed obsolete code 2022-05-31 16:22:27 +02:00
Matthias Bisping
043fa1ee53 removed obsolete code 2022-05-31 16:22:11 +02:00
Matthias Bisping
1fa6bbdbc6 removed obsolete code 2022-05-31 16:19:37 +02:00
Matthias Bisping
93747d0f63 test param adjustment 2022-05-31 16:14:48 +02:00
Matthias Bisping
dc4f578e94 reorganized serve-test to use only default-objects instead of test-object 2022-05-31 16:13:47 +02:00
Matthias Bisping
93da0d12bb refactoring: added paramater 'n' to consume_and_publish 2022-05-31 15:30:23 +02:00
Matthias Bisping
a688cbd7bd refactoring 2022-05-31 13:57:20 +02:00
Matthias Bisping
0104395790 removed obsolete code 2022-05-31 13:50:44 +02:00
Matthias Bisping
18a9683ddb operation field in queue message WIP 2022-05-31 13:47:40 +02:00
Julius Unverfehrt
6d31cbe635 Pull request #37: RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised. 
Merge in RR/pyinfra from RED-4124-s3-retry to master

Squashed commit of the following:

commit 40e4dc3712fc692115b8274226430a431bf0192f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue May 31 12:59:45 2022 +0200

    RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised.
master_61 master_62 		2022-05-31 13:13:18 +02:00
Matthias Bisping
ae2509dc59 modified visitor and queue manager for 1 -> n (1 request to n response messages) 2022-05-30 13:04:12 +02:00
Matthias Bisping
bf9f6ba8e2 tweaked response upload related logic and repaired visitor tests that were broken by new visitor code written for accomodating the aggregation storage strategy 2022-05-30 12:10:14 +02:00
Matthias Bisping
868a53b23f response file path depending on response metadata and request page index complete 2022-05-25 17:37:18 +02:00
Matthias Bisping
2d1ec16714 modified serve test to use components from fixtures; response file path depending on response metadata and request page index WIP 2022-05-25 16:56:08 +02:00
Matthias Bisping
9e2ed6a9f9 fix: data was doubly encoded and hence always triggering the immediate upload path 2022-05-24 14:51:24 +02:00
Matthias Bisping
ab56c9a173 added todo comment for modifying acknowledging logic at some point to allow input buffering to take effect 2022-05-24 14:39:39 +02:00
Matthias Bisping
298d8d3e2c metadata as part of storage item test works 2022-05-23 15:59:56 +02:00
Matthias Bisping
0842ec0d91 metadata as part of storage item WIP 2022-05-23 15:36:18 +02:00
Matthias Bisping
c944cdb1a7 refactoring: splitting source data from encoded data in data fixture 2022-05-23 13:53:50 +02:00
Matthias Bisping
7b998cdaf6 refactoring 2022-05-23 13:14:18 +02:00
Matthias Bisping
426967ee46 refactoring 2022-05-23 11:39:30 +02:00
Matthias Bisping
54ca81d577 moved parameter combination based test skipping into operation factory 2022-05-23 11:28:35 +02:00
Matthias Bisping
13888524fb refactoring 2022-05-23 10:40:28 +02:00
Matthias Bisping
a7ffaeb18f changed return value of file name listing function for storages to return strings of filenames rather than tuples of bucket name and file name 2022-05-23 10:19:07 +02:00
Matthias Bisping
c97393f690 skipping undefined combinations for analysis_task 2022-05-23 10:07:28 +02:00
Matthias Bisping
7ff466e0ea added test for empty return-data operation, like classification 2022-05-23 09:58:43 +02:00
Matthias Bisping
02b0009219 data AND metadata is being uploaded instead of data only 2022-05-18 16:35:18 +02:00
Matthias Bisping
cf13f67394 refactoring: serve_test can now be run with input_data_items like image, pdf etc 2022-05-18 10:56:14 +02:00
Matthias Bisping
0ab86206ec fixed bug introduced by overwritng 'body' as explanatory variable within try-block, which resultet in republish() receiving parsed body, instead of body as bytes 2022-05-18 10:32:54 +02:00
Matthias Bisping
35542f994c integration test for lazy pipeline 2022-05-18 09:24:12 +02:00
Matthias Bisping
fb712af7c6 storag aggregation strategy working in first version 2022-05-17 22:30:54 +02:00
Matthias Bisping
6cb13051eb fixed following bugs: 
- upper() did yield instead of return
 - metdadata was not repeated when zipping with results generator
 - since test metadata was empty dict,  target data was therefore empty always, since results were zipped with {}
 - hence added check for target lengths > 0
 - fixed return value of queued stream function dispatcher; only returned first item of 1 -> n results
 2022-05-17 21:48:16 +02:00
Matthias Bisping
456cb4157d refactoring: move 2022-05-17 17:27:58 +02:00
Matthias Bisping
6945760045 refactoring: move 2022-05-17 15:59:04 +02:00
Julius Unverfehrt
4f18dbc60d Pull request #36: RED-4049 fix: Pyinfra now retries to reconnect after AMPQ connection loss and eventually throws an error if unsuccessful, forcing container restart. 
Merge in RR/pyinfra from RED-4049-AMQP-reconnect to master

Squashed commit of the following:

commit 1baafb781c9f623faf4223043e1185fd4ecbbe0e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue May 17 14:36:47 2022 +0200

    RED-4049 fix: Pyinfra now retries to reconnect after AMPQ connection loss and eventually throws an error if unsuccessful, forcing container restart.
1.0.7 		2022-05-17 14:41:03 +02:00
Matthias Bisping
47f1d77c03 renaming 2022-05-17 12:12:43 +02:00
Matthias Bisping
fb325ce43d renaming 2022-05-17 12:10:24 +02:00
Matthias Bisping
5590669939 pipelin laziness test works again 2022-05-17 10:44:43 +02:00