Matthias Bisping
14ab23b2cc
fixed bug in operation wrapper returning a tuple instead of an singleton-iterable with a tuple in one of the return-cases.
2022-06-13 15:36:17 +02:00
Matthias Bisping
8a64e5d868
narrowed down the pytest bug: n_items interacts with s3_backend: when n_items has more than one entry, s3_backend must not be the last decorator
2022-06-13 10:36:26 +02:00
Matthias Bisping
051cea3ded
found bug in pytest fixture setup causing serve_test to fail (s3 backend fixture function vs s3 backend decorator fixture)
2022-06-13 10:15:33 +02:00
Matthias Bisping
40bc8c2c2c
debugging of queue problem, when queue is not consumed by skipping a test configuration WIP
2022-06-13 09:49:10 +02:00
Matthias Bisping
9962651d88
download strategy WIP: added 1 -> n upload logic
2022-06-10 14:06:19 +02:00
Matthias Bisping
1940b974b1
added id setting to operatin mocks and thus removed need for random seed of randomly seeded hash in storage item identifier
2022-06-09 15:18:41 +02:00
Matthias Bisping
7e46a66698
fixed bug in stream_pages: return -> yield
2022-06-09 14:58:42 +02:00
Matthias Bisping
e43504f08d
commented out consumer tests. something is wrong with the fixtures, leading to the tests failing when run together with other tests. Consumer functionality is covered by serve_test.py, but dedicated tests should be restored at a later point.
2022-06-08 11:03:50 +02:00
Matthias Bisping
bffaa0786e
readded correct consumer test code which git messed up
2022-06-07 15:48:54 +02:00
Matthias Bisping
8d209d63c7
Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into fixing_consumer_tests
2022-06-07 15:44:39 +02:00
Matthias Bisping
0dee98b23d
consumer test adjustment WIP
2022-06-07 15:11:38 +02:00
Matthias Bisping
91701929e5
adjusted stream buffer test for core-operations taking tuples now
2022-06-07 15:08:58 +02:00
Matthias Bisping
c55e41f2d8
refactoring; tweaked json-blob-parser; added standardization case for decodable strings as storage items
2022-06-07 14:55:40 +02:00
Matthias Bisping
f718b2f7ef
parser composer checks for either-type
2022-06-03 16:34:37 +02:00
Matthias Bisping
e48fa85784
replaced trial and error logic for parsing blobs in visitor with parser composer instance
2022-06-03 16:18:56 +02:00
Matthias Bisping
6e5af4092e
added parsers and parser composer for clean handling of storage blobs in the context of interpreting downloaded blobs
2022-06-03 16:13:15 +02:00
Matthias Bisping
7730950b50
cleaning up standardization method for downloaded storage items (WIP)
2022-06-03 14:08:40 +02:00
Matthias Bisping
9a47388017
adjusted param fixtures for serve test
2022-06-03 13:48:33 +02:00
Matthias Bisping
8a2b60a8f5
applied black
2022-06-03 13:42:52 +02:00
Matthias Bisping
9232385dea
modified core operations to return metadata for better classification mock test
2022-06-03 13:40:59 +02:00
Matthias Bisping
eb81e96400
added extraction test case (with page index) to serving test
2022-06-03 13:13:45 +02:00
Julius Unverfehrt
e7ee0cda42
add compression for storage item before upload, update script for extraction
2022-06-02 15:49:28 +02:00
Matthias Bisping
3046b4dc26
misc minor fixes while integrating with pdf2image
2022-05-31 17:58:28 +02:00
Matthias Bisping
dd591bd24b
removed obsolete code
2022-05-31 16:23:14 +02:00
Matthias Bisping
12fa52f38c
removed obsolete code
2022-05-31 16:22:27 +02:00
Matthias Bisping
043fa1ee53
removed obsolete code
2022-05-31 16:22:11 +02:00
Matthias Bisping
1fa6bbdbc6
removed obsolete code
2022-05-31 16:19:37 +02:00
Matthias Bisping
93747d0f63
test param adjustment
2022-05-31 16:14:48 +02:00
Matthias Bisping
dc4f578e94
reorganized serve-test to use only default-objects instead of test-object
2022-05-31 16:13:47 +02:00
Matthias Bisping
93da0d12bb
refactoring: added paramater 'n' to consume_and_publish
2022-05-31 15:30:23 +02:00
Matthias Bisping
a688cbd7bd
refactoring
2022-05-31 13:57:20 +02:00
Matthias Bisping
ae2509dc59
modified visitor and queue manager for 1 -> n (1 request to n response messages)
2022-05-30 13:04:12 +02:00
Matthias Bisping
bf9f6ba8e2
tweaked response upload related logic and repaired visitor tests that were broken by new visitor code written for accomodating the aggregation storage strategy
2022-05-30 12:10:14 +02:00
Matthias Bisping
2d1ec16714
modified serve test to use components from fixtures; response file path depending on response metadata and request page index WIP
2022-05-25 16:56:08 +02:00
Matthias Bisping
9e2ed6a9f9
fix: data was doubly encoded and hence always triggering the immediate upload path
2022-05-24 14:51:24 +02:00
Matthias Bisping
298d8d3e2c
metadata as part of storage item test works
2022-05-23 15:59:56 +02:00
Matthias Bisping
0842ec0d91
metadata as part of storage item WIP
2022-05-23 15:36:18 +02:00
Matthias Bisping
c944cdb1a7
refactoring: splitting source data from encoded data in data fixture
2022-05-23 13:53:50 +02:00
Matthias Bisping
7b998cdaf6
refactoring
2022-05-23 13:14:18 +02:00
Matthias Bisping
426967ee46
refactoring
2022-05-23 11:39:30 +02:00
Matthias Bisping
54ca81d577
moved parameter combination based test skipping into operation factory
2022-05-23 11:28:35 +02:00
Matthias Bisping
13888524fb
refactoring
2022-05-23 10:40:28 +02:00
Matthias Bisping
a7ffaeb18f
changed return value of file name listing function for storages to return strings of filenames rather than tuples of bucket name and file name
2022-05-23 10:19:07 +02:00
Matthias Bisping
c97393f690
skipping undefined combinations for analysis_task
2022-05-23 10:07:28 +02:00
Matthias Bisping
7ff466e0ea
added test for empty return-data operation, like classification
2022-05-23 09:58:43 +02:00
Matthias Bisping
02b0009219
data AND metadata is being uploaded instead of data only
2022-05-18 16:35:18 +02:00
Matthias Bisping
cf13f67394
refactoring: serve_test can now be run with input_data_items like image, pdf etc
2022-05-18 10:56:14 +02:00
Matthias Bisping
0ab86206ec
fixed bug introduced by overwritng 'body' as explanatory variable within try-block, which resultet in republish() receiving parsed body, instead of body as bytes
2022-05-18 10:32:54 +02:00
Matthias Bisping
35542f994c
integration test for lazy pipeline
2022-05-18 09:24:12 +02:00
Matthias Bisping
6cb13051eb
fixed following bugs:
...
- upper() did yield instead of return
- metdadata was not repeated when zipping with results generator
- since test metadata was empty dict, target data was therefore empty always, since results were zipped with {}
- hence added check for target lengths > 0
- fixed return value of queued stream function dispatcher; only returned first item of 1 -> n results
2022-05-17 21:48:16 +02:00
