- revert functional refactoring changes to be able
to determine where the error described in the ticket comes from
- change array normalization to dimensionally
sparse arrays to reduce memory consumption
Merge in RR/image-prediction from adjust-falsy-loglevel to master
Squashed commit of the following:
commit 66794acb1a64be6341f98c7c0ce0bc202634a9f4
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Feb 10 10:15:41 2023 +0100
replace trace log level by debug
- trace method is not supported by buld-in logging module
- disable scanned page filter, since dropping these disables the
computation of the images hash and the frontend OCR hint, which are both
wanted
- optimize image extraction by using arrays instead of byte streams for
the conversion to PIL images
- reduce code duplication by adapting functions of the module
- use the modules enums for image metadata
- improve readabilty of the scanned page detection heuristic
Merge in RR/image-prediction from fschulz/update-to-new-pyinfra-version to master
* commit 'd63f8c4eaf39ef7346188b585fb9d968de72db87':
update dependencies
Merge in RR/image-prediction from RED-5324 to master
* commit 'c72ef26a6caac8d87cdc08dd19dbe235247129d4':
RED-5324: Update pyinfra to include storage-region fix
Merge in RR/image-prediction from RED-4206-fix-unwanted-restart-bug to master
Squashed commit of the following:
commit 3dfe7b861816ef9019103e16a23efd97a08fb617
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Sep 22 13:53:32 2022 +0200
RED-4206 wrap queue callback in process to manage memory allocation with the operating system and force deallocation after processing.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit 9674901235264de6b74d679fd39a52775ac4aee1
Merge: ec2ab89 9763d2c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:55:58 2022 +0200
Merge remote-tracking branch 'origin' into RED-5202-port-hotfixes
commit ec2ab890b8307942d147d6b8b236f6a3c1d0aebc
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:49:17 2022 +0200
swap case when the log is printed for env var parsing
commit aaa02ea35e9c1b3b307116d7e3e32c93fd79ef5d
Merge: 5d87066 521222e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:28:39 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into RED-5202-port-hotfixes
commit 5d87066b40b28f919b1346f5e5396b46445b4e00
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:25:01 2022 +0200
remove warning log for non existent non default env var
commit 23c61ef49ef918b29952150d4a6e61b99d60ac64
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:14:19 2022 +0200
make env var parser discrete
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit aaa02ea35e9c1b3b307116d7e3e32c93fd79ef5d
Merge: 5d87066 521222e
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:28:39 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into RED-5202-port-hotfixes
commit 5d87066b40b28f919b1346f5e5396b46445b4e00
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:25:01 2022 +0200
remove warning log for non existent non default env var
commit 23c61ef49ef918b29952150d4a6e61b99d60ac64
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 15:14:19 2022 +0200
make env var parser discrete
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5202-port-hotfixes to master
Squashed commit of the following:
commit c1b92270354c764861da0f7782348e9cd0725d76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:28:44 2022 +0200
fixed statefulness issue with os.environ in tests
commit ad9c5657fe93079d5646ba2b70fa091e8d2daf76
Author: Matthias Bisping <matthias.bisping@axbit.com>
Date: Mon Sep 12 13:04:55 2022 +0200
- Adapted response formatting logic for threshold maps passed via env vars.
- Added test for reading threshold maps and values from env vars.
commit c60e8cd6781b8e0c3ec69ccd0a25375803de26f0
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 11:38:01 2022 +0200
add parser for environment variables WIP
commit 101b71726c697f30ec9298ba62d2203bd7da2efb
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:52:33 2022 +0200
Add typehints, make custom page quotient breach function private since the intention of outsourcing it from build_image_info is to make it testable seperately
commit 04aee4e62781e78cd54c6d20e961dcd7bf1fc081
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 09:25:59 2022 +0200
DotIndexable default get method exception made more specific
commit 4584e7ba66400033dc5f1a38473b644eeb11e67c
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:55:05 2022 +0200
RED-5202 port temporary broken image handling so the hotfix won't be lost by upgrading the service. A proper solution is still desirable (see RED-5148)
commit 5f99622646b3f6d3a842aebef91ff8e082072cd6
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Sep 12 08:47:02 2022 +0200
RED-5202 add per class customizable max image to page quotient setting for signatures, default is 0.4. Can be overwritten by , set to null to use default value or set to value that should be used.
Merge in RR/image-prediction from RED-5009-update-pyinfra to master
* commit 'e54819e687b4515c0031df431e26bee033359099':
RED-5009 update pyinfra to support message rejection on unobtainable files
Merge in RR/image-prediction from RED-5009-extend-callback to master
Squashed commit of the following:
commit 5ed02af09812783c46c2fb47832fe3a02344aa03
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Aug 23 10:56:37 2022 +0200
queue callback: add storage lookup for input file, add should_publish flag to signal processing success to queue manager
Merge in RR/image-prediction from update-build-scripts to master
Squashed commit of the following:
commit 4a5b21d6e6e0d76091443ba3faaad15953855bad
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 15:08:13 2022 +0200
add pdf2image & pyinfra installation
Merge in RR/image-prediction from integrate-image-extraction-new-pyinfra to master
Squashed commit of the following:
commit 8470c065c71ea2a985aadfc399fb32c693e3a90f
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 09:19:52 2022 +0200
add key script
commit 8f6eb1e79083fb32fb7bedac640c10b6fd411899
Merge: 27fd7de c1b9629
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Thu Aug 18 09:17:50 2022 +0200
Merge branch 'master' of ssh://git.iqser.com:2222/rr/image-prediction into integrate-image-extraction-new-pyinfra
commit 27fd7de39a59d0d88fbddb471dd7797b61223ece
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 17 13:15:09 2022 +0200
update pyinfra
commit ca58f85642598dc15e286074982e7cedae9a1355
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Tue Aug 16 16:16:10 2022 +0200
update pdf2image-service
commit f43795cee0e211e14ac5f9296b01d440ae759c55
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Mon Aug 15 10:32:02 2022 +0200
update pipeline script to also work with figure detection metadata
commit 2b2da1b60ce56fb006cf2f6b65aeda9774391b2a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Fri Aug 12 13:37:48 2022 +0200
add new pyinfra, add optional image classifcation under key dataCV if figure metadata is present on storage
commit bae25bedbd3a262a9d00e18a1b19f4ee6f1eb924
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 13:27:41 2022 +0200
tidy-up
commit 287b0ebc8a952e506185d13508eaa386d0420704
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 12:57:35 2022 +0200
update server logic for new pyinfra, add extraction from scanned PDF with figure detection logic
commit 3225cefaa25e4559b105397bc06c867a22806ba8
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 10 10:37:31 2022 +0200
integrate new pyinfra logic
commit 46926078342b0680a7416560bb69bec037cf8038
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 3 13:15:27 2022 +0200
add image extraction for scanned PDFs WIP
commit 1b3b11b6f9044d44cb9a822a78197a2ebc6f306a
Author: Julius Unverfehrt <julius.unverfehrt@iqser.com>
Date: Wed Aug 3 09:41:06 2022 +0200
add pyinfra and pdf2image as git submodule
