Merge in RR/image-prediction from fschulz/update-to-new-pyinfra-version to master
* commit 'd63f8c4eaf39ef7346188b585fb9d968de72db87':
update dependencies
Merge in RR/image-prediction from RED-5324 to master
* commit 'c72ef26a6caac8d87cdc08dd19dbe235247129d4':
RED-5324: Update pyinfra to include storage-region fix
Merge in RR/image-prediction from RED-4206-fix-unwanted-restart-bug to master
Squashed commit of the following:
commit 3dfe7b861816ef9019103e16a23efd97a08fb617
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Sep 22 13:53:32 2022 +0200
RED-4206 wrap queue callback in process to manage memory allocation with the operating system and force deallocation after processing.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit 9674901235264de6b74d679fd39a52775ac4aee1
Merge: ec2ab89 9763d2c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:55:58 2022 +0200
Merge remote-tracking branch 'origin' into RED-5202-port-hotfixes
commit ec2ab890b8307942d147d6b8b236f6a3c1d0aebc
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:49:17 2022 +0200
swap case when the log is printed for env var parsing
commit aaa02ea35e9c1b3b307116d7e3e32c93fd79ef5d
Merge: 5d87066 521222e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:28:39 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into RED-5202-port-hotfixes
commit 5d87066b40b28f919b1346f5e5396b46445b4e00
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:25:01 2022 +0200
remove warning log for non existent non default env var
commit 23c61ef49ef918b29952150d4a6e61b99d60ac64
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:14:19 2022 +0200
make env var parser discrete
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit aaa02ea35e9c1b3b307116d7e3e32c93fd79ef5d
Merge: 5d87066 521222e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:28:39 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into RED-5202-port-hotfixes
commit 5d87066b40b28f919b1346f5e5396b46445b4e00
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:25:01 2022 +0200
remove warning log for non existent non default env var
commit 23c61ef49ef918b29952150d4a6e61b99d60ac64
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:14:19 2022 +0200
make env var parser discrete
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5009-update-pyinfra to master
* commit 'e54819e687b4515c0031df431e26bee033359099':
RED-5009 update pyinfra to support message rejection on unobtainable files
Merge in RR/image-prediction from RED-5009-extend-callback to master
Squashed commit of the following:
commit 5ed02af09812783c46c2fb47832fe3a02344aa03
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Aug 23 10:56:37 2022 +0200
queue callback: add storage lookup for input file, add should_publish flag to signal processing success to queue manager
Merge in RR/image-prediction from update-build-scripts to master
Squashed commit of the following:
commit 4a5b21d6e6e0d76091443ba3faaad15953855bad
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 15:08:13 2022 +0200
add pdf2image & pyinfra installation
Merge in RR/image-prediction from integrate-image-extraction-new-pyinfra to master
Squashed commit of the following:
commit 8470c065c71ea2a985aadfc399fb32c693e3a90f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 09:19:52 2022 +0200
add key script
commit 8f6eb1e79083fb32fb7bedac640c10b6fd411899
Merge: 27fd7de c1b9629
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 09:17:50 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into integrate-image-extraction-new-pyinfra
commit 27fd7de39a59d0d88fbddb471dd7797b61223ece
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 17 13:15:09 2022 +0200
update pyinfra
commit ca58f85642598dc15e286074982e7cedae9a1355
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Aug 16 16:16:10 2022 +0200
update pdf2image-service
commit f43795cee0e211e14ac5f9296b01d440ae759c55
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Aug 15 10:32:02 2022 +0200
update pipeline script to also work with figure detection metadata
commit 2b2da1b60ce56fb006cf2f6b65aeda9774391b2a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Aug 12 13:37:48 2022 +0200
add new pyinfra, add optional image classifcation under key dataCV if figure metadata is present on storage
commit bae25bedbd3a262a9d00e18a1b19f4ee6f1eb924
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 13:27:41 2022 +0200
tidy-up
commit 287b0ebc8a952e506185d13508eaa386d0420704
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 12:57:35 2022 +0200
update server logic for new pyinfra, add extraction from scanned PDF with figure detection logic
commit 3225cefaa25e4559b105397bc06c867a22806ba8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 10:37:31 2022 +0200
integrate new pyinfra logic
commit 46926078342b0680a7416560bb69bec037cf8038
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 3 13:15:27 2022 +0200
add image extraction for scanned PDFs WIP
commit 1b3b11b6f9044d44cb9a822a78197a2ebc6f306a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 3 09:41:06 2022 +0200
add pyinfra and pdf2image as git submodule
Merge in RR/image-prediction from restore-master to master
Squashed commit of the following:
commit 937968241f08281859be4304bdb0d8eff49f3678
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Jun 21 15:30:41 2022 +0200
refactor
commit b8b84548ef187bfcb88ef97fda74508e37dfb967
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Jun 21 15:21:37 2022 +0200
restore master
Merge in RR/image-prediction from RED-4329-add-prometheus to master
Squashed commit of the following:
commit 2b8446e703617c6897b6149846f2548ec292a9a1
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Jun 21 14:40:44 2022 +0200
RED-4329 add prometheus endpoint with summary metric
Merge in RR/image-prediction from tests to master
Squashed commit of the following:
commit 1776e3083c97025e699d579f936dd0cc6e1fe152
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 13:54:27 2022 +0100
blacckkkyykykykyk
commit 4c9e6c38bdcea7d81008bf9dfcfcdd19d199da6a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 13:53:40 2022 +0100
add predicting as subprocess, add workaround for keras not working if the model was loaded in different process
commit 530de2ff8979c70aa22f06edf297864787e0cc79
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 13:36:23 2022 +0100
refactor
commit 130d0e8b23e0375a6fd240ac8aa00492c341a716
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 13:34:54 2022 +0100
add minimal not working example for keras bug in multiprocess process
commit 2589598b052f680fd702df4f60d56a55778474a9
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 11:13:45 2022 +0100
test
commit eb6f211f02bc184e7f92d6b4d53c91da34ab9f2f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 11:07:32 2022 +0100
hardcoded test
commit 3e9bfac5cf9b2e09340e2c2c5b24a800925bcd60
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 11:01:21 2022 +0100
test
commit 3d9c4d8856522cc2a22b2a7b9ea64d34629eb2c1
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 10:57:03 2022 +0100
change test
commit 58ca784d6c56fd63734062d0c40b6b39550cf7d7
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 10:21:38 2022 +0100
fix test
commit 6faad5ad5b6ef59bb5ef701b57d4c4addd17de0e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Mar 21 10:00:28 2022 +0100
add predictor test
commit 3fbca0ac23821568a8afa904a8fb33ab0679f129
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 18 13:04:13 2022 +0100
refactor folder structure
commit 90e3058c7124394a9f229d50278e57194f3d875d
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 18 12:58:02 2022 +0100
add response test
commit 2a2deffd0b461ec5161009b3923623152f4c8f44
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Mar 18 12:56:32 2022 +0100
add test infrastructure
Merge in RR/image-prediction from integrate_image_service to master
* commit '8a1df76078cb05ded0152ae4a8d8af275a25bc4e':
blacked
refactor
tidy up repo
RED-3501: adapt service-container to image-service-v2
adapt service-container to image-service-v2