Marmelator/pyinfra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 36 Wiki Activity
411 Commits 9 Branches 411 Tags
Commit Graph

22 Commits

Author SHA1 Message Date
Matthias Bisping
c85800aefc refactoring; removed obsolete config entries and code 2022-06-23 19:22:51 +02:00
Julius Unverfehrt
92b4416f21 refactor processing server set up to also work with mappings of operations to functions 2022-06-21 11:27:38 +02:00
Matthias Bisping
8d209d63c7 Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into fixing_consumer_tests 2022-06-07 15:44:39 +02:00
Julius Unverfehrt
e7ee0cda42 add compression for storage item before upload, update script for extraction 2022-06-02 15:49:28 +02:00
Julius Unverfehrt
90f8f9da36 update script for extraction 2022-06-02 15:12:33 +02:00
Julius Unverfehrt
c2d7127a84 add log for Consumer Error, fix page index hash function 2022-06-02 14:15:03 +02:00
Julius Unverfehrt
187055e5eb adjust mock_client script for conversion 2022-05-31 18:19:41 +02:00
Matthias Bisping
3046b4dc26 misc minor fixes while integrating with pdf2image 2022-05-31 17:58:28 +02:00
Julius Unverfehrt
6d31cbe635 Pull request #37: RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised. 
Merge in RR/pyinfra from RED-4124-s3-retry to master

Squashed commit of the following:

commit 40e4dc3712fc692115b8274226430a431bf0192f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue May 31 12:59:45 2022 +0200

    RED-4124 PyInfra now tries three times to download an unobtainable object before republishing message, removed unreadable stack trace print if message is put on dead letter queue since allready logged where the exeption is raised.
 2022-05-31 13:13:18 +02:00
Matthias Bisping
535be0413d Pull request #30: Multiple consumers fix 
Merge in RR/pyinfra from multiple_consumer_fix to master

Squashed commit of the following:

commit b2892d2f21c90e4ebc9a48718ab0a834bf65fc32
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Apr 21 17:38:11 2022 +0200

    formatting

commit 57b3be627caf3c6a87ec248036b7258e9063709d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Apr 21 17:36:05 2022 +0200

    removed debug print

commit 41db74b9103b55fd6f6d79b27a654042f86c86ea
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Apr 21 17:35:05 2022 +0200

    reintroduced usage for basic_consume. Normal consume was blocking other consumers, but basic_consume is not easily testable.
 2022-04-21 17:45:14 +02:00
Kresnadi Budisantoso
1363030b95 Pull request #27: Add parser for bucket/container env vars 
Merge in RR/pyinfra from kbudisantoso/configyaml-1650538128334 to master

Squashed commit of the following:

commit 6103b7720315aaef3d98aea8f3c817477bbf500b
Merge: 69ac65a 3b91185
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Apr 21 14:45:10 2022 +0200

    Merge remote-tracking branch 'origin' into kbudisantoso/configyaml-1650538128334

commit 69ac65ae1bd4095c797112c6f9530f0b1705277e
Merge: 9a1cd07 a00ceae
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Apr 21 14:37:34 2022 +0200

    Merge remote-tracking branch 'origin' into kbudisantoso/configyaml-1650538128334

commit 9a1cd07c09e5ee2618f2c1a3c27b69c67b1eaeb0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Apr 21 14:35:49 2022 +0200

    test done

commit e7127e8af937fe067f1f92eb688187ebbe609478
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Apr 21 14:32:25 2022 +0200

    test

commit 262957e33d19dbafb3f10b5a32c438460b966a88
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Apr 21 14:16:33 2022 +0200

    add parser for env var storage_buckets/containers

commit 3535002b4aac9f297bdbe112b04f537cef25f5c2
Author: Kresnadi Budisantoso <kresnadi.budisantoso@iqser.com>
Date:   Thu Apr 21 12:48:52 2022 +0200

    config.yaml online editiert mit Bitbucket
 2022-04-21 14:55:01 +02:00
Julius Unverfehrt
11c9f8a073 Pull request #25: Fixes 
Merge in RR/pyinfra from fixes to master

Squashed commit of the following:

commit e3eff12ccdea52e041cc7a14cda72d3e32aa2144
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 22 15:42:35 2022 +0100

    black

commit 2bc520c849ea4e833cb60b2c97626da6636d3155
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 22 15:42:08 2022 +0100

    adjust mock script

commit 429b6b8f3a3fc8aa35515395712057d1c7bec13e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 22 15:41:42 2022 +0100

    change scope for retry consume

commit 7488394e313270fe7ba356c40d810e7cb3c706ee
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 22 15:39:39 2022 +0100

    add heartbeat to AMQP connection

commit 004c5fa805bfb982f55de533bc109fa21bacfbc8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 22 15:38:15 2022 +0100

    Adjust error handling for missing prometheus endpoint: error is logged not raised
 2022-03-22 15:48:27 +01:00
Julius Unverfehrt
1b1da50faf Pull request #24: Response dict & Prometheus forwarding 
Merge in RR/pyinfra from response-dict to master

Squashed commit of the following:

commit aed09653aafe7b9558c542ea0c36a5f072c6ab9e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Mar 21 15:20:24 2022 +0100

    BLACK

commit d1b76673a22373c32815466c37dcb006fa0d6fb4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Mar 21 15:19:55 2022 +0100

    change default response dict

commit 82bd48a7545b9b244bdcdebf7ce976947eecf580
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Mon Mar 21 15:19:15 2022 +0100

    fix handling of missing prometheus endpoint
 2022-03-21 15:22:38 +01:00
Julius Unverfehrt
5a382f22c3 Pull request #22: Change queue durability 
Merge in RR/pyinfra from change-queue-durability to master

Squashed commit of the following:

commit b01e5e05e27776cbb629ba3c2e757a64934ace30
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 17 12:50:40 2022 +0100

    refactor

commit 549b89952e626ef7b1029c97ec7fb527112cc2b2
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 17 12:47:13 2022 +0100

    black

commit 6a36f56a3e6308f4954a28c1bf4797a838986371
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 17 12:46:51 2022 +0100

    adjust mock scriptis,add different file format option for IO

commit 3b50f88482d74ea39240cdd41cc7dd62076c87b7
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Thu Mar 17 12:45:49 2022 +0100

    change request_queue to durable=True
 2022-03-17 12:52:42 +01:00
Julius Unverfehrt
391bcc482f Pull request #21: refactoring of prometheus tunneling 
Merge in RR/pyinfra from add-prometheus-metrics to master

Squashed commit of the following:

commit 3736e867bfb105f2c2601f6d25343c996027cc5f
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Mar 16 11:35:25 2022 +0100

    removed obsolete config entry

commit dc191b17d863ec4f8009fb130c2c3a78d4116969
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Mar 16 11:34:12 2022 +0100

    removed obsolete dependency

commit 5ba9765e88da7dd15700b211794f433a6f7ea0df
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Mar 16 11:32:37 2022 +0100

    changed error handling for prometheus endpoint

commit 894a6b5d4c7026b9a703a8b2cd70641e7ed7323b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Wed Mar 16 11:16:39 2022 +0100

    fixed definition order broken by auto-refac; reduced prometheus code to only forwarding to analysis endpoint

commit 9f3c884c75289c7b558e8cc8fb0154b5ddd3a323
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Mar 16 08:59:45 2022 +0100

    black is back

commit 5950799e03f3578ff58f19430494c6b0c223c0f6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Wed Mar 16 08:59:11 2022 +0100

    add prometheus memory peak monitoring, combine report with analysis report
 2022-03-16 11:37:37 +01:00
Julius Unverfehrt
f5b7203778 Pull request #20: Pika encapsulation 
Merge in RR/pyinfra from pika_encapsulation to master

Squashed commit of the following:

commit 251fea4094062a72f5f0b1f8a54f959a1d7309ec
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 15:02:25 2022 +0100

    Adjusted readme and config to the actual changes.

commit d00a0aadc02be8ab1343dbaa2a8df82418e77673
Merge: ded29bb c34a1ce
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 14:06:21 2022 +0100

    Merge branch 'master' of ssh://git.iqser.com:2222/rr/pyinfra into pika_encapsulation

commit ded29bb8d0a78c1d5fd172edb74f0b120da05d5f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 14:02:37 2022 +0100

    blackkkkkkkkkkkkkkkk

commit de3899c69f4e83fa5b8dc2702a18be66661201e9
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:59:43 2022 +0100

    Judgement Day - The Great Refactoring

commit 48eb2bb792dd45539e7db20949fedae03316c2b3
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:30:41 2022 +0100

    blacky

commit 38fb073291c6989fcc5560ef9f97e7dafd9570be
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:29:46 2022 +0100

    update mock client/publish

commit 8744cdeb6d21adab81fb82c7850a4ada31a1a3f9
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date:   Tue Mar 15 13:29:19 2022 +0100

    quickfix consumer.consume_and_publish bug

commit 0ce30cafbf15142e3f773d0e510b7d22927ee86d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 11:42:04 2022 +0100

    updated serving logic for new queue, visior and storage logic

commit 5ac5fbf50bf9d8a6b6466cd04d82d21525250b4c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 11:41:18 2022 +0100

    changed callback signature to expect a dict with a key 'data'

commit 4c7c4b466e2b5228a6098c9130cd6c3334315153
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Tue Mar 15 09:46:39 2022 +0100

    set dead letter queue related params

commit f8b60aad16eb0f5c91bed857ac3b352c4bcb7468
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 20:22:57 2022 +0100

    added republishing logic

commit a40f07139af9121d8b72efa52b6b2588e7d233e8
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:10:08 2022 +0100

    removed obsolete import

commit d3605b0913f8b155961b8c8e90e7c1d442a1da4f
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:09:50 2022 +0100

    removed obsolete code

commit 9130b9fc753c63bfe4abd3cc26b31caf5c5bf2cb
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 19:05:59 2022 +0100

    test refac

commit 1205adecdbd42a46dcc264a46086aab94c285b94
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:59 2022 +0100

    removed obsolete import

commit 59d0de97e39646b3fa6c6ca91644dbf8adbd0de3
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:21 2022 +0100

    applied black

commit f78a6894bc99b42d53a7c265ffd8c7869e9d606d
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:49:03 2022 +0100

    applied black

commit 32740a914f8be15eed954389f6ddbb565e1567e6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:47:47 2022 +0100

    removed obsolete code

commit f21cbc141efc10d0e779f683dcc452e6c889dd6b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 14:47:27 2022 +0100

    consumer with visitor test fixed (added connection.close() in case of pika queue manager)

commit c415dcd8e6375e015b959a9c3cdee61b8ffe7d60
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Mon Mar 14 13:56:54 2022 +0100

    consumer with visitor test WIP 2

... and 15 more commits
 2022-03-15 15:05:14 +01:00
Matthias Bisping
b82ae951a4 Pull request #12: Config refactoring 
Merge in RR/pyinfra from config_refactoring to master

Squashed commit of the following:

commit 22636e5e5df1148004598a268348673537454e36
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 14:57:57 2022 +0100

    applied black

commit 5d244c3f67fb9d6bd7cb78cbe92fc8035b6cf9b7
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 14:56:22 2022 +0100

    restructured config and made correspondig changes in referencing code

commit 4dc64124e16e0f490e6785324b88751ee32dc49c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 13:12:41 2022 +0100

    test config restructuring

commit b0bd9aebdf58f3f4f43a1d4cdaf451bd6036d135
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 13:10:35 2022 +0100

    factored out test section of config
 2022-02-25 15:12:12 +01:00
Matthias Bisping
dca206cda5 Pull request #11: storage refactoring 
Merge in RR/pyinfra from s3_refactoring to master

Squashed commit of the following:

commit d9224b879d47c6f1574d78b7239a209f800972dc
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 12:52:57 2022 +0100

    fixed brokem reference

commit 4189ebb0481c85b5984adb8c6008bbc9bae6d71b
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 12:51:11 2022 +0100

    renaming

commit 945f2397c5301409be19229e11bb50a733f6e2ff
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 12:48:13 2022 +0100

    renaming

commit 3d86500b4b703a56cb32d998c1c12e8fd06e3b2c
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 12:45:32 2022 +0100

    renaming

commit 4b26acca978ad49bb0b6e785addcb601d90214db
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 12:39:08 2022 +0100

    renaming

commit 81be5e55b6d70ccb9f8b07b5860f89da7231122f
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 11:44:46 2022 +0100

    renaming

commit 23b35b1f91def3a6185315772c1249826637e627
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Fri Feb 25 11:31:15 2022 +0100

    added teardown step of clearing test bucke to storage tests; added tests for nested paths

commit 6e26c1a6962bfe867827f80a38c091f63fa395f0
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Fri Feb 25 10:56:32 2022 +0100

    black & removed käsebrot

commit 519c88b39933fa699e6eff5d29bfea7ed1ca8719
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Fri Feb 25 10:55:18 2022 +0100

    data loading/publishing adjusted to new storage logic

commit c530b5929a6f892663009eef4b92f04b4f7f533c
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 17:24:47 2022 +0100

    removing old storage logic WIP

commit 880e22832cd203dc0d0471729996cbd0b5f455f1
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 17:12:44 2022 +0100

    manage minio script refactor

commit 01149770750bdf88094376b1f731ace343f32d76
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 16:32:23 2022 +0100

    renaming

commit 7d6dbabf30d928cb699564e2e7a668253ec6990b
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 16:26:10 2022 +0100

    added list bucket files & refactor serve.py to take new storage logic

commit b20d6fd26b4523538fd02acc5889b932de2e3f03
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Feb 24 15:42:23 2022 +0100

    added compose file referecen to pytest.ini; added coveragerc

commit ff1c7e319bd229bc5dd905ddbc04c211b78f88cd
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Feb 24 15:41:13 2022 +0100

    added pytest.ini

commit d32828ad917a20cf16785a31008d90b4f2ed4442
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 15:37:30 2022 +0100

    black

commit 36931850c3d720b76d39a9f503abeaa326132531
Author: Julius Unverfehrt <Julius.Unverfehrt@iqser.com>
Date:   Thu Feb 24 15:35:55 2022 +0100

    get storage test added

commit 0eb7aaf795217bb34f8f73b5e1e90576b55e9374
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Feb 24 15:08:29 2022 +0100

    added fixture magic for testing differen S3 backends

commit f3f1c42e574365702be632a416b07f025746e989
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Feb 24 14:47:03 2022 +0100

    defined test sections in config for enpoints of storages

commit 16e34052da2cc8c61473d21c1929023445e619d6
Author: Matthias Bisping <matthias.bisping@iqser.com>
Date:   Thu Feb 24 14:11:53 2022 +0100

    refactoring; implemented S3 handle

... and 18 more commits
 2022-02-25 12:58:55 +01:00
Julius Unverfehrt
ae73420955 refactoring 2022-02-17 08:20:59 +01:00
Matthias Bisping
61e03f9c47 refactorig WIP 3: parametrized storage; added azure BS; updated reqs 2022-02-16 16:28:40 +01:00
Matthias Bisping
5cd52928ce refactorig WIP 2022-02-16 15:49:38 +01:00
Matthias Bisping
ca020ee64f renaming, restructuring 2022-02-16 14:43:47 +01:00