Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
308 Commits 9 Branches 411 Tags
Commit Graph

308 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Matthias Bisping
14ab23b2cc fixed bug in operation wrapper returning a tuple instead of an singleton-iterable with a tuple in one of the return-cases. 2022-06-13 15:36:17 +02:00
Matthias Bisping
8a64e5d868 narrowed down the pytest bug: n_items interacts with s3_backend: when n_items has more than one entry, s3_backend must not be the last decorator 2022-06-13 10:36:26 +02:00
Matthias Bisping
051cea3ded found bug in pytest fixture setup causing serve_test to fail (s3 backend fixture function vs s3 backend decorator fixture) 2022-06-13 10:15:33 +02:00
Matthias Bisping
40bc8c2c2c debugging of queue problem, when queue is not consumed by skipping a test configuration WIP 2022-06-13 09:49:10 +02:00
Matthias Bisping
9962651d88 download strategy WIP: added 1 -> n upload logic 2022-06-10 14:06:19 +02:00
Matthias Bisping
249c6203b2 download strategy WIP 2022-06-10 13:26:43 +02:00
Matthias Bisping
3a3c497383 added logic for uploading response files to a folder, which defaults to the name of the operation used 2022-06-10 13:03:40 +02:00
Matthias Bisping
13b6388e5a fixed default identifier type 2022-06-09 16:24:11 +02:00
Matthias Bisping
1940b974b1 added id setting to operatin mocks and thus removed need for random seed of randomly seeded hash in storage item identifier 2022-06-09 15:18:41 +02:00
Matthias Bisping
7e46a66698 fixed bug in stream_pages: return -> yield 2022-06-09 14:58:42 +02:00
Matthias Bisping
5b45f5fa15 added dependency 2022-06-08 14:27:34 +02:00
Matthias Bisping
e43504f08d commented out consumer tests. something is wrong with the fixtures, leading to the tests failing when run together with other tests. Consumer functionality is covered by serve_test.py, but dedicated tests should be restored at a later point. 2022-06-08 11:03:50 +02:00
Matthias Bisping
bffaa0786e readded correct consumer test code which git messed up 2022-06-07 15:48:54 +02:00
Matthias Bisping
8d209d63c7 Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into fixing_consumer_tests 2022-06-07 15:44:39 +02:00
Matthias Bisping
0dee98b23d consumer test adjustment WIP 2022-06-07 15:11:38 +02:00
Matthias Bisping
91701929e5 adjusted stream buffer test for core-operations taking tuples now 2022-06-07 15:08:58 +02:00
Matthias Bisping
c55e41f2d8 refactoring; tweaked json-blob-parser; added standardization case for decodable strings as storage items 2022-06-07 14:55:40 +02:00
Matthias Bisping
f718b2f7ef parser composer checks for either-type 2022-06-03 16:34:37 +02:00
Matthias Bisping
730bdfb220 refactoring 2022-06-03 16:26:12 +02:00
Matthias Bisping
e48fa85784 replaced trial and error logic for parsing blobs in visitor with parser composer instance 2022-06-03 16:18:56 +02:00
Matthias Bisping
6e5af4092e added parsers and parser composer for clean handling of storage blobs in the context of interpreting downloaded blobs 2022-06-03 16:13:15 +02:00
Matthias Bisping
ea2d3223fb parsing strategy error handling for bytes as not an encoded string 2022-06-03 14:55:04 +02:00
Matthias Bisping
26573eeda3 introduced parsing strategy for storage blobs as part of the queue visitor 2022-06-03 14:49:19 +02:00
Matthias Bisping
7730950b50 cleaning up standardization method for downloaded storage items (WIP) 2022-06-03 14:08:40 +02:00
Matthias Bisping
9a47388017 adjusted param fixtures for serve test 2022-06-03 13:48:33 +02:00
Matthias Bisping
8a2b60a8f5 applied black 2022-06-03 13:42:52 +02:00
Matthias Bisping
9232385dea modified core operations to return metadata for better classification mock test 2022-06-03 13:40:59 +02:00
Matthias Bisping
eb81e96400 added extraction test case (with page index) to serving test 2022-06-03 13:13:45 +02:00
Julius Unverfehrt
e7ee0cda42 add compression for storage item before upload, update script for extraction 2022-06-02 15:49:28 +02:00
Julius Unverfehrt
90f8f9da36 update script for extraction 2022-06-02 15:12:33 +02:00
Julius Unverfehrt
c2d7127a84 add log for Consumer Error, fix page index hash function 2022-06-02 14:15:03 +02:00
Matthias Bisping
bfe8bbb8cb reorganized queue message metadata in request 2022-06-01 16:00:46 +02:00
Matthias Bisping
ecff50ae7c submit endpoint is now 'submit' or 'operation' 2022-06-01 11:46:51 +02:00
Matthias Bisping
7a1b215d69 removed obsolete code 2022-06-01 11:37:47 +02:00
Matthias Bisping
01ce914417 generalizing server setup from operations WIP 2022-06-01 10:58:44 +02:00
Matthias Bisping
2b72174605 Merge branch 'integration_test_in_order_develop_aggregation_stratgey' of ssh://git.iqser.com:2222/rr/pyinfra into integration_test_in_order_develop_aggregation_stratgey 2022-05-31 18:40:47 +02:00
Matthias Bisping
586871a26f added queue message body to analysis input dict 2022-05-31 18:40:40 +02:00
Julius Unverfehrt
187055e5eb adjust mock_client script for conversion 2022-05-31 18:19:41 +02:00
Matthias Bisping
3046b4dc26 misc minor fixes while integrating with pdf2image 2022-05-31 17:58:28 +02:00
Matthias Bisping
dd591bd24b removed obsolete code 2022-05-31 16:23:14 +02:00
Matthias Bisping
12fa52f38c removed obsolete code 2022-05-31 16:22:27 +02:00
Matthias Bisping
043fa1ee53 removed obsolete code 2022-05-31 16:22:11 +02:00
Matthias Bisping
1fa6bbdbc6 removed obsolete code 2022-05-31 16:19:37 +02:00
Matthias Bisping
93747d0f63 test param adjustment 2022-05-31 16:14:48 +02:00
Matthias Bisping
dc4f578e94 reorganized serve-test to use only default-objects instead of test-object 2022-05-31 16:13:47 +02:00
Matthias Bisping
93da0d12bb refactoring: added paramater 'n' to consume_and_publish 2022-05-31 15:30:23 +02:00
Matthias Bisping
a688cbd7bd refactoring 2022-05-31 13:57:20 +02:00
Matthias Bisping
0104395790 removed obsolete code 2022-05-31 13:50:44 +02:00
Matthias Bisping
18a9683ddb operation field in queue message WIP 2022-05-31 13:47:40 +02:00
Julius Unverfehrt
6d31cbe635 Pull request #37: RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised. 
Merge in RR/pyinfra from RED-4124-s3-retry to master

Squashed commit of the following:

commit 40e4dc3712fc692115b8274226430a431bf0192f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue May 31 12:59:45 2022 +0200

    RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised.
master_61 master_62 		2022-05-31 13:13:18 +02:00