Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
348 Commits 9 Branches 411 Tags
Commit Graph

62 Commits

Author SHA1 Message Date
Matthias Bisping
94254e1681 Pull request #38: 2.0.0 input output file pattern for download strategy 
Merge in RR/pyinfra from 2.0.0-input-output-file-pattern-for-download-strategy to 2.0.0

Squashed commit of the following:

commit c7ce79ebbeace6a8cb7925ed69eda2d7cd2a4783
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Fri Jun 24 12:35:29 2022 +0200

    refactor

commit 80f04e544962760adb2dc60c9dd03ccca22167d6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Jun 24 11:06:10 2022 +0200

    refactoring of component factory, callback and client-pipeline getter

commit 6c024e1a789e1d55f0739c6846e5c02e8b7c943d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 20:04:10 2022 +0200

    operations section in config cleaned up; added upload formatter

commit c85800aefc224967cea591c1ec4cf1aaa3ac8215
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 19:22:51 2022 +0200

    refactoring; removed obsolete config entries and code

commit 4be125952d82dc868935c8c73ad87fd8f0bd1d6c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 19:14:47 2022 +0200

    removed obsolete code

commit ac69a5c8e3f1e2fd7e828a17eeab97984f4f9746
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 18:58:41 2022 +0200

    refactoring: rm dl strat module

commit efd36d0fc4f8f36d267bfa9d35415811fe723ccc
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 18:33:51 2022 +0200

    refactoring: multi dl strat -> downloader, rm single dl strat

commit afffdeb993500a6abdb6fe85a549e3d6e97e9ee7
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 16:39:22 2022 +0200

    operations section in config cleaned up

commit 671129af3e343490e0fb277a2b0329aa3027fd73
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Jun 23 16:09:16 2022 +0200

    rename prometheus metric name to include service name

commit 932a3e314b382315492aecab95b1f02f2916f8a6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 14:43:23 2022 +0200

    cleaned up file descr mngr

commit 79350b4ce71fcd095ed6a5e1d3a598ea246fae53
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 12:26:15 2022 +0200

    refactoring WIP: moving response stratgey logic into storage strategy (needs to be refactored as well, later) and file descr mngr. Here the moved code needs to be cleaned up.

commit 7e48c66f0c378b25a433a4034eefdc8a0957e775
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 12:00:48 2022 +0200

    refactoring; removed operation / response folder from output path

commit 8e6cbdaf23c48f6eeb52512b7f382d5727e206d6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 11:08:37 2022 +0200

    refactoring; added operation -> file pattern mapping to file descr mngr (mainly for self-documentaton purposes)

commit 2c80d7cec0cc171e099e5b13aadd2ae0f9bf4f02
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:59:57 2022 +0200

    refactoring: introduced input- and output-file specific methods to file descr mngr

commit ecced37150eaac3008cc1b01b235e5f7135e504b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:43:26 2022 +0200

    refactoring

commit 3828341e98861ff8d63035ee983309ad5064bb30
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:42:46 2022 +0200

    refactoring

commit 9a7c412523d467af40feb6924823ca89e28aadfe
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 17:04:54 2022 +0200

    add prometheus metric name for default operation

commit d207b2e274ba53b2a21a18c367bb130fb05ee1cd
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 17:02:55 2022 +0200

    Merge config

commit d3fdf36b12d8def18810454765e731599b833bfc
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 22 17:01:12 2022 +0200

    added fixmes / todos

commit f49d0b9cb7764473ef9d127bc5d88525a4a16a23
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 16:28:25 2022 +0200

    update script

... and 47 more commits
 2022-06-24 12:59:26 +02:00
Julius Unverfehrt
bb7e631f91 introduce flag to distinguish between server side tests and complete integration tests 2022-06-14 11:56:47 +02:00
Matthias Bisping
a69f613fe6 completed multi download to single response logic. but broke pipeline test again, maybe? 2022-06-14 11:08:03 +02:00
Matthias Bisping
14ab23b2cc fixed bug in operation wrapper returning a tuple instead of an singleton-iterable with a tuple in one of the return-cases. 2022-06-13 15:36:17 +02:00
Matthias Bisping
40bc8c2c2c debugging of queue problem, when queue is not consumed by skipping a test configuration WIP 2022-06-13 09:49:10 +02:00
Matthias Bisping
1940b974b1 added id setting to operatin mocks and thus removed need for random seed of randomly seeded hash in storage item identifier 2022-06-09 15:18:41 +02:00
Matthias Bisping
7e46a66698 fixed bug in stream_pages: return -> yield 2022-06-09 14:58:42 +02:00
Matthias Bisping
9a47388017 adjusted param fixtures for serve test 2022-06-03 13:48:33 +02:00
Matthias Bisping
8a2b60a8f5 applied black 2022-06-03 13:42:52 +02:00
Matthias Bisping
9232385dea modified core operations to return metadata for better classification mock test 2022-06-03 13:40:59 +02:00
Matthias Bisping
eb81e96400 added extraction test case (with page index) to serving test 2022-06-03 13:13:45 +02:00
Matthias Bisping
2d1ec16714 modified serve test to use components from fixtures; response file path depending on response metadata and request page index WIP 2022-05-25 16:56:08 +02:00
Matthias Bisping
7b998cdaf6 refactoring 2022-05-23 13:14:18 +02:00
Matthias Bisping
54ca81d577 moved parameter combination based test skipping into operation factory 2022-05-23 11:28:35 +02:00
Matthias Bisping
c97393f690 skipping undefined combinations for analysis_task 2022-05-23 10:07:28 +02:00
Matthias Bisping
7ff466e0ea added test for empty return-data operation, like classification 2022-05-23 09:58:43 +02:00
Matthias Bisping
6cb13051eb fixed following bugs: 
- upper() did yield instead of return
 - metdadata was not repeated when zipping with results generator
 - since test metadata was empty dict,  target data was therefore empty always, since results were zipped with {}
 - hence added check for target lengths > 0
 - fixed return value of queued stream function dispatcher; only returned first item of 1 -> n results
 2022-05-17 21:48:16 +02:00
Matthias Bisping
9c262e7138 non-rest pipeline works again 2022-05-17 10:27:32 +02:00
Matthias Bisping
96bf831b00 refactoring: move 2022-05-16 13:31:11 +02:00
Matthias Bisping
5d2b71d647 target data fixture and test for flat stream buffer on different data 2022-05-16 13:21:31 +02:00
Matthias Bisping
7adbdefb4e refactoring: skipping invalid parameter combinations 2022-05-16 12:18:04 +02:00
Matthias Bisping
a1c292a485 refactoring: pulled core operation taking only data out from operation taking data and metadata 2022-05-16 11:53:52 +02:00
Matthias Bisping
948575d199 renaming 2022-05-16 11:43:48 +02:00
Matthias Bisping
2070f300c9 refactoring: queued stream function returns first of generator 2022-05-16 10:28:53 +02:00
Matthias Bisping
092a0e2964 renaming 2022-05-13 17:16:27 +02:00
Matthias Bisping
40777ae609 refactoring: simplyfing lazy processor to queued function WIP 2022-05-13 16:59:50 +02:00
Matthias Bisping
9870aa38d1 renaming 2022-05-13 15:02:05 +02:00
Matthias Bisping
c09e5df23e refactoring: introduced flat stream buffer class 2022-05-13 12:13:59 +02:00
Matthias Bisping
bfdce62ccf refactoring; renaming 2022-05-12 19:46:29 +02:00
Matthias Bisping
8b0c2d4e07 refactoring: further simplified queue consuming function; added one -> many test fixture param 2022-05-12 17:55:45 +02:00
Matthias Bisping
e3793e5c7c refactoring: split stream processor into two functions; moved queue streaming Nothing check from caller to stream function 2022-05-12 17:12:26 +02:00
Matthias Bisping
f29bd7d4d3 removed need for bufferize wrapper by composing with first(chunks(...)) and applying to on-demand processor execution chain; broke mock pipeline, fixing next 2022-05-09 11:06:14 +02:00
Matthias Bisping
f99d779c29 refactoring 2022-05-06 23:39:29 +02:00
Matthias Bisping
dca3eaaa54 fixed buffering and result streaming: all items are yielded individually and computed on demand now 2022-05-06 12:14:09 +02:00
Matthias Bisping
b58a9d11c3 refactoring: added processor class 2022-05-05 16:08:59 +02:00
Matthias Bisping
531ff8d3e0 refactoring; fixed sender 2022-05-04 16:57:08 +02:00
Matthias Bisping
cf0a877569 refactoring 2022-05-04 10:37:46 +02:00
Matthias Bisping
29fb0dda30 improved 1 -> n test and explanation 2022-05-03 15:43:30 +02:00
Matthias Bisping
ea9b405d2a signature harminization for 1 -> 1 and 1 -> n completed 2022-05-02 15:50:14 +02:00
Matthias Bisping
3172a00aaa refactoring 2022-04-29 17:08:34 +02:00
Matthias Bisping
501f0bd5fc signature harminization for 1 -> 1 and 1 -> n WIP: batched and not batched working again 2022-04-29 16:46:25 +02:00
Matthias Bisping
fd57261631 signature harminization for 1 -> 1 and 1 -> n WIP 2022-04-29 15:43:20 +02:00
Matthias Bisping
00276dbcc7 added batching wrapper for internally batching functions 2022-04-29 10:35:55 +02:00
Matthias Bisping
bc0e9ed643 renaming 2022-04-29 09:51:41 +02:00
Matthias Bisping
940bc3a689 changed buffering function behaviour: applies function to buffer. function needs to be lifted from the outside if single items are to be processed. 2022-04-28 21:58:38 +02:00
Matthias Bisping
a999ce2c3b refactoring 2022-04-28 21:36:42 +02:00
Matthias Bisping
e8315ffea9 signature of process_fn changed. Now returns {'data': ..., 'metadata'} instead of {'data': ...} 2022-04-28 21:12:25 +02:00
Matthias Bisping
3b0d0868b9 refactoring: method dispatch via peekable rather than special empty data request 2022-04-28 13:01:01 +02:00
Matthias Bisping
da84ff5112 buffer size constrained by assertion 2022-04-27 18:45:14 +02:00
Matthias Bisping
c9f26000d7 test parametrization for number of input items and buffer size 2022-04-27 18:38:43 +02:00