Matthias Bisping
|
c85800aefc
|
refactoring; removed obsolete config entries and code
|
2022-06-23 19:22:51 +02:00 |
|
Matthias Bisping
|
efd36d0fc4
|
refactoring: multi dl strat -> downloader, rm single dl strat
|
2022-06-23 18:33:51 +02:00 |
|
Matthias Bisping
|
afffdeb993
|
operations section in config cleaned up
|
2022-06-23 16:39:22 +02:00 |
|
Matthias Bisping
|
932a3e314b
|
cleaned up file descr mngr
|
2022-06-23 14:43:23 +02:00 |
|
Matthias Bisping
|
2c80d7cec0
|
refactoring: introduced input- and output-file specific methods to file descr mngr
|
2022-06-23 10:59:57 +02:00 |
|
Matthias Bisping
|
fb0b64f8d5
|
added hash method to config, so lru_cache can distinguish between different configs. but something is broken again in te fixture setup. using many_to_n = [True, False] hangs, while independently both pass. the consumer does not get requests in the former case, probably.
|
2022-06-22 15:21:14 +02:00 |
|
Matthias Bisping
|
1adeb4038a
|
removed direct download strategy usages in test setups
|
2022-06-22 13:56:12 +02:00 |
|
Matthias Bisping
|
bb6b28bb4e
|
refactoring; single dl strat now uses file descriptor manager
|
2022-06-22 12:30:36 +02:00 |
|
Matthias Bisping
|
23876501dc
|
refactoring: de-hardcorded operation by refactoring queue message metadata fixture
|
2022-06-22 12:00:42 +02:00 |
|
Matthias Bisping
|
af88e2d7d4
|
file pattern matching WIP: using of different operations to select file pattern config section -- semi-hardcoded multi input file operation. next: refactor fixtures in order to de-hardcode
|
2022-06-22 11:18:55 +02:00 |
|
Matthias Bisping
|
5728961b26
|
file pattern matching WIP: pulled out file descriptor logic from multi dl strat and put into dedicated class; added single dl strat as default for dl strat in visitor
|
2022-06-22 10:18:41 +02:00 |
|
Matthias Bisping
|
473362b86f
|
file pattern matching WIP: reimplememyed logic in a way that allows unifying upload (for tests) and download file descriptors.
|
2022-06-21 18:11:52 +02:00 |
|
Matthias Bisping
|
3f03e25bd3
|
refactoring; added operations to test config + overwriting main config in components fixture
|
2022-06-21 16:59:43 +02:00 |
|
Matthias Bisping
|
21daab2404
|
introduced component factory to wrap compnent getters into a config parametrized clojure
|
2022-06-21 14:21:56 +02:00 |
|
Matthias Bisping
|
562e383bdd
|
file pattern matching WIP: added first bit of file pattern matching logic to multi dl strat
|
2022-06-21 14:02:51 +02:00 |
|
Matthias Bisping
|
2362619bef
|
fixed bug introduced by aggr strat refac: singleton lists are now unpacked before upload via default merge function
|
2022-06-20 16:36:40 +02:00 |
|
Matthias Bisping
|
343637aaf7
|
- modified aggr strat to produce only one response message, which now contains a list of response files instead of a single response file
- introduced response formatter
|
2022-06-20 15:22:59 +02:00 |
|
Matthias Bisping
|
5392d03295
|
refactoring; removed dispatch callback from aggregation strategy
|
2022-06-20 12:45:59 +02:00 |
|
Matthias Bisping
|
2c902f3129
|
refactored visitor init
|
2022-06-19 00:06:24 +02:00 |
|
Matthias Bisping
|
02da828268
|
wrapped gzip.(de)compress
|
2022-06-18 23:15:22 +02:00 |
|
Matthias Bisping
|
f7e4953a4e
|
refactored visitor module: split into various submodules
|
2022-06-18 22:41:09 +02:00 |
|
Matthias Bisping
|
1558398c56
|
made object name construction logic part of download strategies
|
2022-06-15 12:25:27 +02:00 |
|
Matthias Bisping
|
771df7c78d
|
make bucket before running test; rabbitmq 3.9 again
|
2022-06-14 16:58:27 +02:00 |
|
Matthias Bisping
|
4fcc89f938
|
s3 backend fixture no longer needs to not come last
|
2022-06-14 15:40:23 +02:00 |
|
Matthias Bisping
|
a69f613fe6
|
completed multi download to single response logic. but broke pipeline test again, maybe?
|
2022-06-14 11:08:03 +02:00 |
|
Matthias Bisping
|
8a64e5d868
|
narrowed down the pytest bug: n_items interacts with s3_backend: when n_items has more than one entry, s3_backend must not be the last decorator
|
2022-06-13 10:36:26 +02:00 |
|
Matthias Bisping
|
051cea3ded
|
found bug in pytest fixture setup causing serve_test to fail (s3 backend fixture function vs s3 backend decorator fixture)
|
2022-06-13 10:15:33 +02:00 |
|
Matthias Bisping
|
40bc8c2c2c
|
debugging of queue problem, when queue is not consumed by skipping a test configuration WIP
|
2022-06-13 09:49:10 +02:00 |
|
Matthias Bisping
|
9962651d88
|
download strategy WIP: added 1 -> n upload logic
|
2022-06-10 14:06:19 +02:00 |
|
Matthias Bisping
|
7e46a66698
|
fixed bug in stream_pages: return -> yield
|
2022-06-09 14:58:42 +02:00 |
|
Matthias Bisping
|
91701929e5
|
adjusted stream buffer test for core-operations taking tuples now
|
2022-06-07 15:08:58 +02:00 |
|
Matthias Bisping
|
9a47388017
|
adjusted param fixtures for serve test
|
2022-06-03 13:48:33 +02:00 |
|
Matthias Bisping
|
9232385dea
|
modified core operations to return metadata for better classification mock test
|
2022-06-03 13:40:59 +02:00 |
|
Matthias Bisping
|
eb81e96400
|
added extraction test case (with page index) to serving test
|
2022-06-03 13:13:45 +02:00 |
|
Julius Unverfehrt
|
e7ee0cda42
|
add compression for storage item before upload, update script for extraction
|
2022-06-02 15:49:28 +02:00 |
|
Matthias Bisping
|
3046b4dc26
|
misc minor fixes while integrating with pdf2image
|
2022-05-31 17:58:28 +02:00 |
|
Matthias Bisping
|
dd591bd24b
|
removed obsolete code
|
2022-05-31 16:23:14 +02:00 |
|
Matthias Bisping
|
12fa52f38c
|
removed obsolete code
|
2022-05-31 16:22:27 +02:00 |
|
Matthias Bisping
|
043fa1ee53
|
removed obsolete code
|
2022-05-31 16:22:11 +02:00 |
|
Matthias Bisping
|
1fa6bbdbc6
|
removed obsolete code
|
2022-05-31 16:19:37 +02:00 |
|
Matthias Bisping
|
93747d0f63
|
test param adjustment
|
2022-05-31 16:14:48 +02:00 |
|
Matthias Bisping
|
dc4f578e94
|
reorganized serve-test to use only default-objects instead of test-object
|
2022-05-31 16:13:47 +02:00 |
|
Matthias Bisping
|
93da0d12bb
|
refactoring: added paramater 'n' to consume_and_publish
|
2022-05-31 15:30:23 +02:00 |
|
Matthias Bisping
|
a688cbd7bd
|
refactoring
|
2022-05-31 13:57:20 +02:00 |
|
Matthias Bisping
|
ae2509dc59
|
modified visitor and queue manager for 1 -> n (1 request to n response messages)
|
2022-05-30 13:04:12 +02:00 |
|
Matthias Bisping
|
bf9f6ba8e2
|
tweaked response upload related logic and repaired visitor tests that were broken by new visitor code written for accomodating the aggregation storage strategy
|
2022-05-30 12:10:14 +02:00 |
|
Matthias Bisping
|
2d1ec16714
|
modified serve test to use components from fixtures; response file path depending on response metadata and request page index WIP
|
2022-05-25 16:56:08 +02:00 |
|
Matthias Bisping
|
9e2ed6a9f9
|
fix: data was doubly encoded and hence always triggering the immediate upload path
|
2022-05-24 14:51:24 +02:00 |
|
Matthias Bisping
|
298d8d3e2c
|
metadata as part of storage item test works
|
2022-05-23 15:59:56 +02:00 |
|
Matthias Bisping
|
0842ec0d91
|
metadata as part of storage item WIP
|
2022-05-23 15:36:18 +02:00 |
|