Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
350 Commits 9 Branches 411 Tags
Commit Graph

82 Commits

Author SHA1 Message Date
Julius Unverfehrt
a1bfec765c Pull request #43: Image prediction v2 support 
Merge in RR/pyinfra from image-prediction-v2-support to 2.0.0

Squashed commit of the following:

commit 37c536324e847357e86dd9b72d1e07ad792ed90f
Merge: 77d1db8 01bfb1d
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Jul 11 13:53:56 2022 +0200

    Merge branch '2.0.0' of ssh://git.iqser.com:2222/rr/pyinfra into image-prediction-v2-support

commit 77d1db8e8630de8822c124eb39f4cd817ed1d3e1
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Jul 11 13:07:41 2022 +0200

    add operation assignment via config if operation is not defined by caller

commit 36c8ca48a8c6151f713c093a23de110901ba6b02
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Jul 11 10:33:34 2022 +0200

    refactor nothing part 2

commit f6cd0ef986802554dd544b9b7a24073d3b3f05b5
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Jul 11 10:28:49 2022 +0200

    refactor nothing

commit 1e70d49531e89613c70903be49290b94ee014f65
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jul 6 17:42:12 2022 +0200

    enable docker-compose fixture

commit 9fee32cecdd120cfac3e065fb8ad2b4f37b49226
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jul 6 17:40:35 2022 +0200

    added 'multi' key to actual operation configurations

commit 4287f6d9878dd361489b8490eafd06f81df472ce
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jul 6 16:56:12 2022 +0200

    removed debug prints

commit 23a533e8f99222c7e598fb0864f65e9aa3508a3b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jul 6 16:31:50 2022 +0200

    completed correcting / cleaning upload and download logic with regard to operations and ids. next: remove debug code

commit 33246d1ff94989d2ea70242c7ae2e58afa4d35c1
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jul 6 14:37:17 2022 +0200

    corrected / cleaned upload and download logic with regard to operations and ids

commit 7f2b4e882022c6843cb2f80df202caa495c54ee9
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Jul 5 18:41:07 2022 +0200

    partially decomplected file descriptor manager from concrete and non-generic descriptor code

commit 40b892da17670dae3b8eba1700877c1dcf219852
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Jul 5 09:53:46 2022 +0200

    typo

commit ec4fa8e6f4551ff1f8d4f78c484b7a260f274898
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Jul 5 09:52:41 2022 +0200

    typo

commit 701b43403c328161fd96a73ce388a66035cca348
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 17:26:53 2022 +0200

    made adjustments for image classification with pyinfra 2.x; added related fixmes

commit 7a794bdcc987631cdc4d89b5620359464e2e018e
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 13:05:26 2022 +0200

    removed obsolete imports

commit 3fc6a7ef5d0172dbce1c4292d245eced2f378b5a
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 11:47:12 2022 +0200

    enable docker-compose fixture

commit 36d8d3bc851b06d94cf12a73048a00a67ef79c42
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 11:46:53 2022 +0200

    renaming

commit 3bf00d11cd041dff325b66f13fcd00d3ce96b8b5
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 30 12:47:57 2022 +0200

    refactoring: added cached pipeline factory

commit 90e735852af2f86e35be845fabf28494de952edb
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 13:47:08 2022 +0200

    renaming

commit 93b3d4b202b41183ed8cabe193a4bfa03f520787
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 13:25:03 2022 +0200

    further refactored server setup code: moving and decomplecting

commit 8b2ed83c7ade5bd811cb045d56fbfb0353fa385e
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 12:53:09 2022 +0200

    refactored server setup code: factored out and decoupled operation registry and prometheus summary registry

... and 6 more commits
 2022-07-11 14:17:59 +02:00
Julius Unverfehrt
01bfb1d668 Pull request #40: 2.0.0 documentation 
Merge in RR/pyinfra from 2.0.0-documentation to 2.0.0

Squashed commit of the following:

commit 7a794bdcc987631cdc4d89b5620359464e2e018e
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 13:05:26 2022 +0200

    removed obsolete imports

commit 3fc6a7ef5d0172dbce1c4292d245eced2f378b5a
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 11:47:12 2022 +0200

    enable docker-compose fixture

commit 36d8d3bc851b06d94cf12a73048a00a67ef79c42
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jul 4 11:46:53 2022 +0200

    renaming

commit 3bf00d11cd041dff325b66f13fcd00d3ce96b8b5
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 30 12:47:57 2022 +0200

    refactoring: added cached pipeline factory

commit 90e735852af2f86e35be845fabf28494de952edb
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 13:47:08 2022 +0200

    renaming

commit 93b3d4b202b41183ed8cabe193a4bfa03f520787
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 13:25:03 2022 +0200

    further refactored server setup code: moving and decomplecting

commit 8b2ed83c7ade5bd811cb045d56fbfb0353fa385e
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 29 12:53:09 2022 +0200

    refactored server setup code: factored out and decoupled operation registry and prometheus summary registry

commit da2dce762bdd6889165fbb320dc9ee8a0bd089b2
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Jun 28 19:40:04 2022 +0200

    adjusted test target

commit 70df7911b9b92f4b72afd7d4b33ca2bbf136295e
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Jun 28 19:32:38 2022 +0200

    minor refactoring

commit 0937b63dc000346559bde353381304b273244109
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jun 27 13:59:59 2022 +0200

    support for empty operation suffix

commit 5e56917970962a2e69bbd66a324bdb4618c040bd
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jun 27 12:52:36 2022 +0200

    minor refactoring

commit 40665a7815ae5927b3877bda14fb77deef37d667
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jun 27 10:57:04 2022 +0200

    optimization: prefix filtering via storage API for get_all_object_names

commit af0892a899d09023eb0e61eecb63e03dc2fd3b60
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Jun 27 10:55:47 2022 +0200

    topological sorting of definitions by caller hierarchy
 2022-07-11 09:09:44 +02:00
Matthias Bisping
94254e1681 Pull request #38: 2.0.0 input output file pattern for download strategy 
Merge in RR/pyinfra from 2.0.0-input-output-file-pattern-for-download-strategy to 2.0.0

Squashed commit of the following:

commit c7ce79ebbeace6a8cb7925ed69eda2d7cd2a4783
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Fri Jun 24 12:35:29 2022 +0200

    refactor

commit 80f04e544962760adb2dc60c9dd03ccca22167d6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Jun 24 11:06:10 2022 +0200

    refactoring of component factory, callback and client-pipeline getter

commit 6c024e1a789e1d55f0739c6846e5c02e8b7c943d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 20:04:10 2022 +0200

    operations section in config cleaned up; added upload formatter

commit c85800aefc224967cea591c1ec4cf1aaa3ac8215
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 19:22:51 2022 +0200

    refactoring; removed obsolete config entries and code

commit 4be125952d82dc868935c8c73ad87fd8f0bd1d6c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 19:14:47 2022 +0200

    removed obsolete code

commit ac69a5c8e3f1e2fd7e828a17eeab97984f4f9746
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 18:58:41 2022 +0200

    refactoring: rm dl strat module

commit efd36d0fc4f8f36d267bfa9d35415811fe723ccc
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 18:33:51 2022 +0200

    refactoring: multi dl strat -> downloader, rm single dl strat

commit afffdeb993500a6abdb6fe85a549e3d6e97e9ee7
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 16:39:22 2022 +0200

    operations section in config cleaned up

commit 671129af3e343490e0fb277a2b0329aa3027fd73
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Jun 23 16:09:16 2022 +0200

    rename prometheus metric name to include service name

commit 932a3e314b382315492aecab95b1f02f2916f8a6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 14:43:23 2022 +0200

    cleaned up file descr mngr

commit 79350b4ce71fcd095ed6a5e1d3a598ea246fae53
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 12:26:15 2022 +0200

    refactoring WIP: moving response stratgey logic into storage strategy (needs to be refactored as well, later) and file descr mngr. Here the moved code needs to be cleaned up.

commit 7e48c66f0c378b25a433a4034eefdc8a0957e775
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 12:00:48 2022 +0200

    refactoring; removed operation / response folder from output path

commit 8e6cbdaf23c48f6eeb52512b7f382d5727e206d6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 11:08:37 2022 +0200

    refactoring; added operation -> file pattern mapping to file descr mngr (mainly for self-documentaton purposes)

commit 2c80d7cec0cc171e099e5b13aadd2ae0f9bf4f02
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:59:57 2022 +0200

    refactoring: introduced input- and output-file specific methods to file descr mngr

commit ecced37150eaac3008cc1b01b235e5f7135e504b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:43:26 2022 +0200

    refactoring

commit 3828341e98861ff8d63035ee983309ad5064bb30
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Jun 23 10:42:46 2022 +0200

    refactoring

commit 9a7c412523d467af40feb6924823ca89e28aadfe
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 17:04:54 2022 +0200

    add prometheus metric name for default operation

commit d207b2e274ba53b2a21a18c367bb130fb05ee1cd
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 17:02:55 2022 +0200

    Merge config

commit d3fdf36b12d8def18810454765e731599b833bfc
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Jun 22 17:01:12 2022 +0200

    added fixmes / todos

commit f49d0b9cb7764473ef9d127bc5d88525a4a16a23
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Jun 22 16:28:25 2022 +0200

    update script

... and 47 more commits
 2022-06-24 12:59:26 +02:00
Matthias Bisping
1558398c56 made object name construction logic part of download strategies 2022-06-15 12:25:27 +02:00
Matthias Bisping
e64ade3135 added comments to new config params 2022-06-14 15:33:11 +02:00
Matthias Bisping
d179fdede6 consumer test runs again...? 2022-06-14 14:17:08 +02:00
Julius Unverfehrt
9b975b759b set xfail for broken tests 2022-06-14 14:06:06 +02:00
Julius Unverfehrt
c033d98acd adjust test for new return type of visitor, add download strategy parameter to config 2022-06-14 12:33:56 +02:00
Julius Unverfehrt
bb7e631f91 introduce flag to distinguish between server side tests and complete integration tests 2022-06-14 11:56:47 +02:00
Matthias Bisping
14ab23b2cc fixed bug in operation wrapper returning a tuple instead of an singleton-iterable with a tuple in one of the return-cases. 2022-06-13 15:36:17 +02:00
Matthias Bisping
e43504f08d commented out consumer tests. something is wrong with the fixtures, leading to the tests failing when run together with other tests. Consumer functionality is covered by serve_test.py, but dedicated tests should be restored at a later point. 2022-06-08 11:03:50 +02:00
Matthias Bisping
bffaa0786e readded correct consumer test code which git messed up 2022-06-07 15:48:54 +02:00
Matthias Bisping
8d209d63c7 Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into fixing_consumer_tests 2022-06-07 15:44:39 +02:00
Matthias Bisping
0dee98b23d consumer test adjustment WIP 2022-06-07 15:11:38 +02:00
Matthias Bisping
91701929e5 adjusted stream buffer test for core-operations taking tuples now 2022-06-07 15:08:58 +02:00
Matthias Bisping
c55e41f2d8 refactoring; tweaked json-blob-parser; added standardization case for decodable strings as storage items 2022-06-07 14:55:40 +02:00
Matthias Bisping
7730950b50 cleaning up standardization method for downloaded storage items (WIP) 2022-06-03 14:08:40 +02:00
Matthias Bisping
bf9f6ba8e2 tweaked response upload related logic and repaired visitor tests that were broken by new visitor code written for accomodating the aggregation storage strategy 2022-05-30 12:10:14 +02:00
Matthias Bisping
54ca81d577 moved parameter combination based test skipping into operation factory 2022-05-23 11:28:35 +02:00
Matthias Bisping
13888524fb refactoring 2022-05-23 10:40:28 +02:00
Matthias Bisping
a7ffaeb18f changed return value of file name listing function for storages to return strings of filenames rather than tuples of bucket name and file name 2022-05-23 10:19:07 +02:00
Matthias Bisping
0ab86206ec fixed bug introduced by overwritng 'body' as explanatory variable within try-block, which resultet in republish() receiving parsed body, instead of body as bytes 2022-05-18 10:32:54 +02:00
Matthias Bisping
6cb13051eb fixed following bugs: 
- upper() did yield instead of return
 - metdadata was not repeated when zipping with results generator
 - since test metadata was empty dict,  target data was therefore empty always, since results were zipped with {}
 - hence added check for target lengths > 0
 - fixed return value of queued stream function dispatcher; only returned first item of 1 -> n results
 2022-05-17 21:48:16 +02:00
Matthias Bisping
456cb4157d refactoring: move 2022-05-17 17:27:58 +02:00
Matthias Bisping
6945760045 refactoring: move 2022-05-17 15:59:04 +02:00
Matthias Bisping
fb325ce43d renaming 2022-05-17 12:10:24 +02:00
Matthias Bisping
5590669939 pipelin laziness test works again 2022-05-17 10:44:43 +02:00
Matthias Bisping
9c262e7138 non-rest pipeline works again 2022-05-17 10:27:32 +02:00
Matthias Bisping
e5a4e7e994 applied black 2022-05-16 15:00:09 +02:00
Matthias Bisping
89f562aa71 refactoring: move 2022-05-16 14:58:19 +02:00
Matthias Bisping
1074f44b30 no buffer capacity test; commented out probably dead codde -- removing next 2022-05-16 14:34:37 +02:00
Matthias Bisping
96bf831b00 refactoring: move 2022-05-16 13:31:11 +02:00
Matthias Bisping
5d2b71d647 target data fixture and test for flat stream buffer on different data 2022-05-16 13:21:31 +02:00
Matthias Bisping
d12124b2d5 refactoring 2022-05-16 13:04:37 +02:00
Matthias Bisping
7adbdefb4e refactoring: skipping invalid parameter combinations 2022-05-16 12:18:04 +02:00
Matthias Bisping
948575d199 renaming 2022-05-16 11:43:48 +02:00
Matthias Bisping
2070f300c9 refactoring: queued stream function returns first of generator 2022-05-16 10:28:53 +02:00
Matthias Bisping
092a0e2964 renaming 2022-05-13 17:16:27 +02:00
Matthias Bisping
40777ae609 refactoring: simplyfing lazy processor to queued function WIP 2022-05-13 16:59:50 +02:00
Matthias Bisping
2434e0ea55 refactoring; stream buffer tests 2022-05-13 16:38:20 +02:00
Matthias Bisping
e3793e5c7c refactoring: split stream processor into two functions; moved queue streaming Nothing check from caller to stream function 2022-05-12 17:12:26 +02:00
Matthias Bisping
de0deaa2f4 renaming 2022-05-10 12:19:42 +02:00
Matthias Bisping
83ce7692e6 renaming; adjusted and added tests for lazy bufferize (formerlay on-demand processor) 2022-05-10 12:18:20 +02:00
Matthias Bisping
949413af4a fixed bug in compute_next of on demand processor that skipped all but the first return value of 1 -> n functions 2022-05-10 11:07:58 +02:00
Matthias Bisping
5b913983eb refactoring: move; added null value param to bufferize 2022-05-09 14:50:54 +02:00
Matthias Bisping
f29bd7d4d3 removed need for bufferize wrapper by composing with first(chunks(...)) and applying to on-demand processor execution chain; broke mock pipeline, fixing next 2022-05-09 11:06:14 +02:00
Matthias Bisping
ec620abf54 lazy pipeline test 2022-05-09 01:17:53 +02:00
Matthias Bisping
5b30a32fff added client pipeline without webserver backend 2022-05-09 00:43:21 +02:00
Matthias Bisping
c092e7bcab refactoring; server pipeline WIP 2022-05-08 18:36:15 +02:00
Matthias Bisping
8c1ad64464 refactoring; additional buffer test 2022-05-07 00:29:58 +02:00