Compare commits

...

29 Commits

Author SHA1 Message Date
Kevin Tumma
b54deb95ec Adjust to new CI 2023-06-22 09:33:40 +02:00
Kevin Tumma
a46bc32ea0 Adjust to new CI 2023-06-22 09:31:11 +02:00
Kevin Tumma
4b441cad59 Adjust to new CI 2023-06-22 09:29:39 +02:00
Dominique Eifländer
0b23694a09 Merge branch 'RED-6912-backport' into 'release/1.363.9.x'
RED-6912 - Entries not sorted correctly in the dossier dictionary

See merge request redactmanager/persistence-service!17
2023-06-22 08:57:22 +02:00
Corina Olariu
b0ad7af9c6 RED-6912 - Entries not sorted correctly in the dossier dictionary
- sort entries and falsePositives and falseRecommendations in endpoint getDictionaryForType
- add junit tests
2023-06-22 09:35:38 +03:00
deiflaender
928d30f98c RED-6860: changed to multipart file upload for s3 2023-05-31 16:14:01 +02:00
deiflaender
402f0ccb77 hotfix: Removed long running transaction at download 2023-05-31 11:23:01 +02:00
Viktor Seifert
9742afe175 Pull request #652: RED-6497
Merge in RED/persistence-service from RED-6497 to release/1.363.x

* commit '3fc9dc5132b5eaeb1d8bd609acb57bacc62237d5':
  RED-6497: Removed alternate file size check because it offers less error reporting capabilities
  RED-6497: Fixed issue where the file-size of download was saved incorrectly.
  RED-6497: Changed code to use a different method of getting a file size to have better error reporting
  RED-6497: Removed explicit variable initialization because its marked as a checkstyle violation.
  RED-6497: Corrected handling of the stream, temp-file and exceptions.
2023-03-31 09:47:47 +02:00
Viktor Seifert
3fc9dc5132 Merge branch 'release/1.363.x' into RED-6497 2023-03-30 17:45:05 +02:00
Viktor Seifert
6a3db677ef RED-6497: Removed alternate file size check because it offers less error reporting capabilities 2023-03-30 17:37:18 +02:00
Viktor Seifert
6f203f07e1 RED-6497: Fixed issue where the file-size of download was saved incorrectly.
* Changed the update to the DownloadStatusEntity to use the entity object instead of using a query on JPA-repo.  This prevents values from the object being incorrectly inserted otherwise.
* Extended the DownloadPreparationTest to check the resulting download state and file size.
2023-03-30 17:30:16 +02:00
Corina Olariu
4051d6b3e1 Pull request #651: RED-6265 - Bulk dossier stats endpoint does not return dossier stats of dossiers, for which the current dossier does not have access permissions
Merge in RED/persistence-service from bugfix/RED-6265-3.6.0 to release/1.363.x

* commit '08965675999b4309147a216d362c433d4c08381b':
  RED-6265 - Bulk dossier stats endpoint does not return dossier stats of dossiers, for which the current dossier does not have access permissions
2023-03-30 15:47:44 +02:00
Viktor Seifert
284738b59e RED-6497: Changed code to use a different method of getting a file size to have better error reporting 2023-03-30 15:43:07 +02:00
devplant
0896567599 RED-6265 - Bulk dossier stats endpoint does not return dossier stats of dossiers, for which the current dossier does not have access permissions
- remove deprecated
2023-03-30 16:12:54 +03:00
Viktor Seifert
65884ce42b RED-6497: Removed explicit variable initialization because its marked as a checkstyle violation. 2023-03-30 14:20:55 +02:00
Viktor Seifert
d74b8bf4ba RED-6497: Corrected handling of the stream, temp-file and exceptions.
* Corrected the order of closing the steam and deleting the file.
* Switched the output of exception to the WARN level, since the hint at potential problems.
* Corrected the tests so that they clean-up resources.
* Changed the tests to use an archiver that re-throws exceptions, so that problems are not swallowed in testing.
2023-03-29 18:36:48 +02:00
Dominique Eiflaender
9930430bcd Pull request #641: RED-6503: Do not override ocred pages on ocr successful
Merge in RED/persistence-service from RED-6503-3.6 to release/1.363.x

* commit '12d7eae288b87bd88ecbe66bf08e31a9c6953461':
  RED-6503: Do not override ocred pages on ocr successful
2023-03-28 09:16:58 +02:00
deiflaender
12d7eae288 RED-6503: Do not override ocred pages on ocr successful 2023-03-27 13:49:28 +02:00
Viktor Seifert
cfa9dfcc6f Pull request #624: RED-6310 3.6
Merge in RED/persistence-service from RED-6310-3.6 to release/1.363.x

* commit '18ee468ca15ac07ff8ec6adc2712ccf42129df91':
  RED-6310: Changed element-collection fetch to eager because lazy loading runs into timing based errors
  RED-6310: Updated test for download preparation test to execute all preparation steps.
2023-03-13 16:56:01 +01:00
Viktor Seifert
18ee468ca1 RED-6310: Changed element-collection fetch to eager because lazy loading runs into timing based errors 2023-03-13 16:07:04 +01:00
Viktor Seifert
b9345110aa RED-6310: Updated test for download preparation test to execute all preparation steps.
Previously the last step was not executed.
2023-03-10 17:39:53 +01:00
Viktor Seifert
c2cf764b56 Pull request #618: RED-6310 3.6
Merge in RED/persistence-service from RED-6310-3.6 to release/1.363.x

* commit 'c698bf3a685dc906ce0e811f157122cb62caa93b':
  RED-6310: Backported test to Junit 4
  RED-6310: Moved code to create user-preferences to a separate class so that the calling code can handle a persistence exception
  RED-6310: Corrected services so that they use the user id instead of wrongly using the entity id
  RED-6310: Moved code for multithreaded tests to a helper class
  RED-6310: Removed not needed code from test
  RED-6310: Added setup of tenant id to fix the service test
  RED-6310: Added tests to check if concurrent access to notification-preferences works
2023-03-10 09:58:33 +01:00
Viktor Seifert
c698bf3a68 RED-6310: Backported test to Junit 4 2023-03-09 15:56:32 +01:00
Viktor Seifert
222e5915ed RED-6310: Moved code to create user-preferences to a separate class so that the calling code can handle a persistence exception
(cherry picked from commit 45bd8e600328ce85c8457525b605da2c16dd70c8)
2023-03-09 15:42:44 +01:00
Viktor Seifert
657a616093 RED-6310: Corrected services so that they use the user id instead of wrongly using the entity id
(cherry picked from commit 643ffc8723d923a7ede8ee52e25ea40e25045148)
2023-03-09 15:42:43 +01:00
Viktor Seifert
551840a46f RED-6310: Moved code for multithreaded tests to a helper class
(cherry picked from commit 40d6961742436fd4f7c5e07616850d8b1eb31a43)
2023-03-09 15:42:42 +01:00
Viktor Seifert
a0e0aadb61 RED-6310: Removed not needed code from test
(cherry picked from commit fd7f39bc7e4b4cc67ed55d5e77ef84d93779b342)
2023-03-09 15:42:41 +01:00
Viktor Seifert
d42d56e7f1 RED-6310: Added setup of tenant id to fix the service test
(cherry picked from commit b1d79970c033822ddffb47be199c379cada2ce40)
2023-03-09 15:42:39 +01:00
Viktor Seifert
bbc58379f4 RED-6310: Added tests to check if concurrent access to notification-preferences works
(cherry picked from commit ef36f5c10f21042ce74621dd8e48a754b6a7258f)
2023-03-09 15:42:38 +01:00
31 changed files with 539 additions and 452 deletions

6
.gitlab-ci.yml Normal file
View File

@ -0,0 +1,6 @@
variables:
SONAR_PROJECT_KEY: 'RED_persistence-service'
include:
- project: 'gitlab/gitlab'
ref: 'main'
file: 'ci-templates/maven_java.yml'

View File

@ -1,37 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.atlassian.bamboo</groupId>
<artifactId>bamboo-specs-parent</artifactId>
<version>8.1.3</version>
<relativePath/>
</parent>
<artifactId>bamboo-specs</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>com.atlassian.bamboo</groupId>
<artifactId>bamboo-specs-api</artifactId>
</dependency>
<dependency>
<groupId>com.atlassian.bamboo</groupId>
<artifactId>bamboo-specs</artifactId>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<!-- run 'mvn test' to perform offline validation of the plan -->
<!-- run 'mvn -Ppublish-specs' to upload the plan to your Bamboo server -->
</project>

View File

@ -1,126 +0,0 @@
package buildjob;
import static com.atlassian.bamboo.specs.builders.task.TestParserTask.createJUnitParserTask;
import java.time.LocalTime;
import com.atlassian.bamboo.specs.api.BambooSpec;
import com.atlassian.bamboo.specs.api.builders.BambooKey;
import com.atlassian.bamboo.specs.api.builders.Variable;
import com.atlassian.bamboo.specs.api.builders.docker.DockerConfiguration;
import com.atlassian.bamboo.specs.api.builders.permission.PermissionType;
import com.atlassian.bamboo.specs.api.builders.permission.Permissions;
import com.atlassian.bamboo.specs.api.builders.permission.PlanPermissions;
import com.atlassian.bamboo.specs.api.builders.plan.Job;
import com.atlassian.bamboo.specs.api.builders.plan.Plan;
import com.atlassian.bamboo.specs.api.builders.plan.PlanIdentifier;
import com.atlassian.bamboo.specs.api.builders.plan.Stage;
import com.atlassian.bamboo.specs.api.builders.plan.branches.BranchCleanup;
import com.atlassian.bamboo.specs.api.builders.plan.branches.PlanBranchManagement;
import com.atlassian.bamboo.specs.api.builders.project.Project;
import com.atlassian.bamboo.specs.builders.task.CheckoutItem;
import com.atlassian.bamboo.specs.builders.task.InjectVariablesTask;
import com.atlassian.bamboo.specs.builders.task.ScriptTask;
import com.atlassian.bamboo.specs.builders.task.VcsCheckoutTask;
import com.atlassian.bamboo.specs.builders.task.VcsTagTask;
import com.atlassian.bamboo.specs.builders.trigger.BitbucketServerTrigger;
import com.atlassian.bamboo.specs.builders.trigger.ScheduledTrigger;
import com.atlassian.bamboo.specs.model.task.InjectVariablesScope;
import com.atlassian.bamboo.specs.model.task.ScriptTaskProperties.Location;
import com.atlassian.bamboo.specs.util.BambooServer;
/**
* Plan configuration for Bamboo.
* Learn more on: <a href="https://confluence.atlassian.com/display/BAMBOO/Bamboo+Specs">https://confluence.atlassian.com/display/BAMBOO/Bamboo+Specs</a>
*/
@BambooSpec
public class PlanSpec {
private static final String SERVICE_NAME = "persistence-service";
private static final String SERVICE_KEY = SERVICE_NAME.toUpperCase().replaceAll("-", "");
/**
* Run main to publish plan on Bamboo
*/
public static void main(final String[] args) throws Exception {
//By default, credentials are read from the '.credentials' file.
BambooServer bambooServer = new BambooServer("http://localhost:8085");
Plan buildPlan = new PlanSpec().createPlanBuild();
bambooServer.publish(buildPlan);
PlanPermissions buildPlanPermission = new PlanSpec().createPlanPermission(buildPlan.getIdentifier());
bambooServer.publish(buildPlanPermission);
Plan secPlan = new PlanSpec().createSecBuild();
bambooServer.publish(secPlan);
PlanPermissions secPlanPermission = new PlanSpec().createPlanPermission(secPlan.getIdentifier());
bambooServer.publish(secPlanPermission);
}
private PlanPermissions createPlanPermission(PlanIdentifier planIdentifier) {
Permissions permission = new Permissions().userPermissions("atlbamboo",
PermissionType.EDIT,
PermissionType.VIEW,
PermissionType.ADMIN,
PermissionType.CLONE,
PermissionType.BUILD)
.groupPermissions("development", PermissionType.EDIT, PermissionType.VIEW, PermissionType.CLONE, PermissionType.BUILD)
.groupPermissions("devplant", PermissionType.EDIT, PermissionType.VIEW, PermissionType.CLONE, PermissionType.BUILD)
.loggedInUserPermissions(PermissionType.VIEW)
.anonymousUserPermissionView();
return new PlanPermissions(planIdentifier.getProjectKey(), planIdentifier.getPlanKey()).permissions(permission);
}
private Project project() {
return new Project().name("RED").key(new BambooKey("RED"));
}
public Plan createPlanBuild() {
return new Plan(project(), SERVICE_NAME, new BambooKey(SERVICE_KEY)).description("Build Plan for Persitence Service")
.variables(new Variable("maven_add_param", ""))
.stages(new Stage("Default Stage").jobs(new Job("Default Job", new BambooKey("JOB1")).tasks(new ScriptTask().description("Clean")
.inlineBody("#!/bin/bash\n" + "set -e\n" + "rm -rf ./*"),
new VcsCheckoutTask().description("Checkout Default Repository").cleanCheckout(true).checkoutItems(new CheckoutItem().defaultRepository()),
new ScriptTask().description("Build").location(Location.FILE).fileFromPath("bamboo-specs/src/main/resources/scripts/build-java.sh").argument(SERVICE_NAME),
createJUnitParserTask().description("Resultparser")
.resultDirectories("**/test-reports/*.xml, **/target/surefire-reports/*.xml, **/target/failsafe-reports/*.xml")
.enabled(true),
new InjectVariablesTask().description("Inject git Tag").path("git.tag").namespace("g").scope(InjectVariablesScope.LOCAL),
new VcsTagTask().description("${bamboo.g.gitTag}").tagName("${bamboo.g.gitTag}").defaultRepository())
.dockerConfiguration(new DockerConfiguration().image("nexus.iqser.com:5001/infra/maven:3.8.4-openjdk-17-slim")
.dockerRunArguments("--net=host")
.volume("/etc/maven/settings.xml", "/usr/share/maven/ref/settings.xml")
.volume("/var/run/docker.sock", "/var/run/docker.sock"))))
.linkedRepositories("RED / " + SERVICE_NAME)
.triggers(new BitbucketServerTrigger())
.planBranchManagement(new PlanBranchManagement().createForVcsBranch()
.delete(new BranchCleanup().whenInactiveInRepositoryAfterDays(14))
.notificationForCommitters());
}
public Plan createSecBuild() {
return new Plan(project(), SERVICE_NAME + "-Sec", new BambooKey(SERVICE_KEY + "SEC")).description("Security Analysis Plan")
.stages(new Stage("Default Stage").jobs(new Job("Default Job", new BambooKey("JOB1")).tasks(new ScriptTask().description("Clean")
.inlineBody("#!/bin/bash\n" + "set -e\n" + "rm -rf ./*"),
new VcsCheckoutTask().description("Checkout Default Repository").cleanCheckout(true).checkoutItems(new CheckoutItem().defaultRepository()),
new ScriptTask().description("Sonar").location(Location.FILE).fileFromPath("bamboo-specs/src/main/resources/scripts/sonar-java.sh").argument(SERVICE_NAME))
.dockerConfiguration(new DockerConfiguration().image("nexus.iqser.com:5001/infra/maven:3.8.4-openjdk-17-slim")
.dockerRunArguments("--net=host")
.volume("/etc/maven/settings.xml", "/usr/share/maven/conf/settings.xml")
.volume("/var/run/docker.sock", "/var/run/docker.sock"))))
.linkedRepositories("RED / " + SERVICE_NAME)
.triggers(new ScheduledTrigger().scheduleOnceDaily(LocalTime.of(23, 00)))
.planBranchManagement(new PlanBranchManagement().createForVcsBranchMatching("release.*").notificationForCommitters());
}
}

View File

@ -1,61 +0,0 @@
#!/bin/bash
set -e
SERVICE_NAME=$1
set SERVER_PORT=$(shuf -i 20000-65000 -n 1)
if [[ "$bamboo_planRepository_branchName" == "master" ]]
then
branchVersion=$(cat pom.xml | grep -Eo " <version>.*-SNAPSHOT</version>" | sed -s 's|<version>\(.*\)\..*\(-*.*\)</version>|\1|' | tr -d ' ')
latestVersion=$( semver $(git tag -l "${branchVersion}.*" ) | tail -n1 )
newVersion="$(semver $latestVersion -p -i minor)"
elif [[ "$bamboo_planRepository_branchName" == release* ]]
then
branchVersion=$(echo $bamboo_planRepository_branchName | sed -s 's|release\/\([0-9]\+\.[0-9]\+\)\.x|\1|')
latestVersion=$( semver $(git tag -l "${branchVersion}.*" ) | tail -n1 )
newVersion="$(semver $latestVersion -p -i patch)"
elif [[ "${bamboo_version_tag}" != "dev" ]]
then
newVersion="${bamboo_version_tag}"
else
mvn -f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
--no-transfer-progress \
${bamboo_maven_add_param} \
clean install \
-Djava.security.egd=file:/dev/./urandomelse
echo "gitTag=${bamboo_planRepository_1_branch}_${bamboo_buildNumber}" > git.tag
exit 0
fi
echo "gitTag=${newVersion}" > git.tag
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
versions:set \
-DnewVersion=${newVersion}
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-image-v1/pom.xml \
versions:set \
-DnewVersion=${newVersion}
mvn -f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
--no-transfer-progress \
clean deploy \
${bamboo_maven_add_param} \
-e \
-DdeployAtEnd=true \
-Dmaven.wagon.http.ssl.insecure=true \
-Dmaven.wagon.http.ssl.allowall=true \
-Dmaven.wagon.http.ssl.ignore.validity.dates=true \
-DaltDeploymentRepository=iqser_release::default::https://nexus.iqser.com/repository/red-platform-releases
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-image-v1/pom.xml \
package
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-image-v1/pom.xml \
docker:push

View File

@ -1,45 +0,0 @@
#!/bin/bash
set -e
SERVICE_NAME=$1
set SERVER_PORT=$(shuf -i 20000-65000 -n 1)
echo "build jar binaries"
mvn -f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
--no-transfer-progress \
clean install \
-Djava.security.egd=file:/dev/./urandomelse
echo "dependency-check:aggregate"
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
org.owasp:dependency-check-maven:aggregate
if [[ -z "${bamboo_repository_pr_key}" ]]
then
echo "Sonar Scan for branch: ${bamboo_planRepository_1_branch}"
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
sonar:sonar \
-Dsonar.projectKey=RED_$SERVICE_NAME \
-Dsonar.host.url=https://sonarqube.iqser.com \
-Dsonar.login=${bamboo_sonarqube_api_token_secret} \
-Dsonar.branch.name=${bamboo_planRepository_1_branch} \
-Dsonar.dependencyCheck.jsonReportPath=target/dependency-check-report.json \
-Dsonar.dependencyCheck.xmlReportPath=target/dependency-check-report.xml \
-Dsonar.dependencyCheck.htmlReportPath=target/dependency-check-report.html
else
echo "Sonar Scan for PR with key1: ${bamboo_repository_pr_key}"
mvn --no-transfer-progress \
-f ${bamboo_build_working_directory}/$SERVICE_NAME-v1/pom.xml \
sonar:sonar \
-Dsonar.projectKey=RED_$SERVICE_NAME \
-Dsonar.host.url=https://sonarqube.iqser.com \
-Dsonar.login=${bamboo_sonarqube_api_token_secret} \
-Dsonar.pullrequest.key=${bamboo_repository_pr_key} \
-Dsonar.pullrequest.branch=${bamboo_repository_pr_sourceBranch} \
-Dsonar.pullrequest.base=${bamboo_repository_pr_targetBranch} \
-Dsonar.dependencyCheck.jsonReportPath=target/dependency-check-report.json \
-Dsonar.dependencyCheck.xmlReportPath=target/dependency-check-report.xml \
-Dsonar.dependencyCheck.htmlReportPath=target/dependency-check-report.html
fi

View File

@ -1,21 +0,0 @@
package buildjob;
import org.junit.Test;
import com.atlassian.bamboo.specs.api.builders.plan.Plan;
import com.atlassian.bamboo.specs.api.exceptions.PropertiesValidationException;
import com.atlassian.bamboo.specs.api.util.EntityPropertiesBuilders;
public class PlanSpecTest {
@Test
public void checkYourPlanOffline() throws PropertiesValidationException {
Plan buildPlan = new PlanSpec().createPlanBuild();
EntityPropertiesBuilders.build(buildPlan);
Plan secPlan = new PlanSpec().createSecBuild();
EntityPropertiesBuilders.build(secPlan);
}
}

View File

@ -3,9 +3,9 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>com.iqser.red</groupId>
<groupId>com.knecon.fforesight</groupId>
<artifactId>platform-docker-dependency</artifactId>
<version>1.2.0</version>
<version>0.1.0</version>
<relativePath/>
</parent>
<modelVersion>4.0.0</modelVersion>

View File

@ -19,12 +19,10 @@ public interface DossierStatsResource {
String DOSSIER_ID_PATH_PARAM = "/{" + DOSSIER_ID_PARAM + "}";
@Deprecated
@GetMapping(value = REST_PATH + DOSSIER_ID_PATH_PARAM, produces = MediaType.APPLICATION_JSON_VALUE)
DossierStats getDossierStats(@PathVariable(DOSSIER_ID_PARAM) String dossierId);
@Deprecated
@PostMapping(value = REST_PATH, produces = MediaType.APPLICATION_JSON_VALUE)
List<DossierStats> getDossierStats(@RequestBody Set<String> dossierIds);

View File

@ -61,10 +61,10 @@ public class DownloadStatusEntity {
@Column
long fileSize;
@ManyToOne(fetch = FetchType.LAZY)
@ManyToOne(fetch = FetchType.EAGER)
DossierEntity dossier;
@ManyToMany
@ManyToMany(fetch = FetchType.EAGER)
@Fetch(FetchMode.SUBSELECT)
List<FileEntity> files = new ArrayList<>();
@ -73,7 +73,7 @@ public class DownloadStatusEntity {
@Convert(converter = JSONDownloadFileTypeConverter.class)
Set<DownloadFileType> downloadFileTypes = new HashSet<>();
@ManyToMany(fetch = FetchType.LAZY)
@ManyToMany(fetch = FetchType.EAGER)
@Fetch(FetchMode.SUBSELECT)
List<ReportTemplateEntity> reports = new ArrayList<>();

View File

@ -6,6 +6,7 @@ import java.util.List;
import javax.persistence.Column;
import javax.persistence.ElementCollection;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.Id;
import javax.persistence.Table;
@ -39,11 +40,11 @@ public class NotificationPreferencesEntity {
@Column
private EmailNotificationType emailNotificationType;
@ElementCollection
@ElementCollection(fetch = FetchType.EAGER)
@Fetch(FetchMode.SUBSELECT)
private List<String> emailNotifications = new ArrayList<>();
@ElementCollection
@ElementCollection(fetch = FetchType.EAGER)
@Fetch(FetchMode.SUBSELECT)
private List<String> inAppNotifications = new ArrayList<>();

View File

@ -73,9 +73,11 @@ public class DownloadStatusPersistenceService {
@Transactional
public void updateStatus(String storageId, DownloadStatusValue status, long fileSize) {
public void updateStatus(DownloadStatusEntity entity, DownloadStatusValue statusValue, long fileSize) {
downloadStatusRepository.updateStatus(storageId, status, fileSize);
entity.setStatus(statusValue);
entity.setFileSize(fileSize);
downloadStatusRepository.save(entity);
}

View File

@ -37,6 +37,7 @@ public class NotificationPersistenceService {
@SneakyThrows
@Transactional
public void insertNotification(AddNotificationRequest addNotificationRequest) {
var notification = new NotificationEntity();

View File

@ -10,6 +10,8 @@ import java.util.stream.Collectors;
import javax.transaction.Transactional;
import org.springframework.beans.BeanUtils;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;
import com.iqser.red.service.persistence.management.v1.processor.entity.notification.NotificationPreferencesEntity;
@ -18,15 +20,20 @@ import com.iqser.red.service.persistence.management.v1.processor.service.persist
import com.iqser.red.service.persistence.service.v1.api.model.notification.NotificationPreferences;
import com.iqser.red.service.persistence.service.v1.api.model.notification.NotificationType;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.experimental.FieldDefaults;
@Service
@RequiredArgsConstructor
@FieldDefaults(makeFinal = true, level = AccessLevel.PRIVATE)
public class NotificationPreferencesPersistenceService {
private final NotificationPreferencesRepository notificationPreferencesRepository;
NotificationPreferencesRepository notificationPreferencesRepository;
private final NotificationRepository notificationRepository;
NonThreadSafeNotificationPreferencesRepositoryWrapper notificationPreferencesRepositoryWrapper;
NotificationRepository notificationRepository;
@Transactional
@ -60,31 +67,28 @@ public class NotificationPreferencesPersistenceService {
@Transactional
public void deleteNotificationPreferences(String userId) {
notificationPreferencesRepository.deleteById(userId);
notificationPreferencesRepository.deleteByUserId(userId);
}
@Transactional
// This method intentionally does not have a @Transactional annotation, since it needs to handle an underlying transaction exception.
public NotificationPreferencesEntity getOrCreateNotificationPreferences(String userId) {
return notificationPreferencesRepository.findById(userId).orElseGet(() -> {
var notificationPreference = new NotificationPreferencesEntity();
notificationPreference.setUserId(userId);
notificationPreference.setEmailNotificationsEnabled(false);
notificationPreference.setInAppNotificationsEnabled(true);
notificationPreference.setInAppNotifications(Arrays.stream(NotificationType.values()).map(Enum::name).collect(Collectors.toList()));
return notificationPreferencesRepository.save(notificationPreference);
});
try {
// The method called here will fail if it is called concurrently (more than 1 thread), since it will always try to create
// the desired entity. But the exception only means, that the entity has been created by another thread.
// In that case we can just fetch the data from the db.
return notificationPreferencesRepositoryWrapper.getOrCreateNotificationPreferences(userId);
} catch (DataIntegrityViolationException ex) {
return notificationPreferencesRepository.getByUserId(userId);
}
}
@Transactional
public void initializePreferencesIfNotExists(String userId) {
if (!notificationPreferencesRepository.existsByUserId(userId)) {
getOrCreateNotificationPreferences(userId);
}
getOrCreateNotificationPreferences(userId);
}
@ -93,4 +97,29 @@ public class NotificationPreferencesPersistenceService {
return notificationPreferencesRepository.findAll();
}
@Component
@RequiredArgsConstructor
@FieldDefaults(makeFinal = true, level = AccessLevel.PRIVATE)
private static class NonThreadSafeNotificationPreferencesRepositoryWrapper {
NotificationPreferencesRepository notificationPreferencesRepository;
@Transactional(Transactional.TxType.REQUIRES_NEW)
public NotificationPreferencesEntity getOrCreateNotificationPreferences(String userId) {
return notificationPreferencesRepository.findByUserId(userId).orElseGet(() -> {
var notificationPreference = new NotificationPreferencesEntity();
notificationPreference.setUserId(userId);
notificationPreference.setEmailNotificationsEnabled(false);
notificationPreference.setInAppNotificationsEnabled(true);
notificationPreference.setInAppNotifications(Arrays.stream(NotificationType.values()).map(Enum::name).collect(Collectors.toList()));
return notificationPreferencesRepository.save(notificationPreference);
});
}
}
}

View File

@ -18,12 +18,7 @@ public interface DownloadStatusRepository extends JpaRepository<DownloadStatusEn
@Modifying
@Query("update DownloadStatusEntity ds set ds.status = :status where ds.storageId = :storageId")
void updateStatus(String storageId, DownloadStatusValue status);
@Modifying
@Query("update DownloadStatusEntity ds set ds.status = :status, ds.fileSize = :fileSize where ds.storageId = :storageId")
void updateStatus(String storageId, DownloadStatusValue status, long fileSize);
@Modifying
@Query("update DownloadStatusEntity ds set ds.lastDownload = :lastDownload where ds.storageId = :storageId")

View File

@ -1,11 +1,19 @@
package com.iqser.red.service.persistence.management.v1.processor.service.persistence.repository;
import java.util.Optional;
import org.springframework.data.jpa.repository.JpaRepository;
import com.iqser.red.service.persistence.management.v1.processor.entity.notification.NotificationPreferencesEntity;
public interface NotificationPreferencesRepository extends JpaRepository<NotificationPreferencesEntity, String> {
boolean existsByUserId(String userId);
Optional<NotificationPreferencesEntity> findByUserId(String userId);
NotificationPreferencesEntity getByUserId(String userId);
void deleteByUserId(String userId);
}

View File

@ -72,6 +72,7 @@
<dependency>
<groupId>com.iqser.red.commons</groupId>
<artifactId>storage-commons</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>com.iqser.red.service</groupId>

View File

@ -2,7 +2,9 @@ package com.iqser.red.service.peristence.v1.server.controller;
import static com.iqser.red.service.persistence.management.v1.processor.utils.MagicConverter.convert;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
import javax.transaction.Transactional;
@ -98,9 +100,12 @@ public class DictionaryController implements DictionaryResource {
var entity = dictionaryPersistenceService.getType(typeId);
var target = convert(entity, Type.class);
target.setEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.ENTRY, fromVersion), DictionaryEntry.class));
target.setFalsePositiveEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.FALSE_POSITIVE, fromVersion), DictionaryEntry.class));
target.setFalseRecommendationEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.FALSE_RECOMMENDATION, fromVersion), DictionaryEntry.class));
target.setEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.ENTRY, fromVersion), DictionaryEntry.class)
.stream().sorted(Comparator.comparing(DictionaryEntry::getValue)).collect(Collectors.toList()));
target.setFalsePositiveEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.FALSE_POSITIVE, fromVersion), DictionaryEntry.class)
.stream().sorted(Comparator.comparing(DictionaryEntry::getValue)).collect(Collectors.toList()));
target.setFalseRecommendationEntries(convert(entryPersistenceService.getEntries(typeId, DictionaryEntryType.FALSE_RECOMMENDATION, fromVersion), DictionaryEntry.class)
.stream().sorted(Comparator.comparing(DictionaryEntry::getValue)).collect(Collectors.toList()));
return target;
}

View File

@ -19,7 +19,6 @@ public class DossierStatsController implements DossierStatsResource {
private final DossierStatsService dossierStatsService;
@Deprecated
@Override
public DossierStats getDossierStats(String dossierId) {
@ -27,7 +26,6 @@ public class DossierStatsController implements DossierStatsResource {
}
@Deprecated
@Override
public List<DossierStats> getDossierStats(Set<String> dossierIds) {

View File

@ -6,7 +6,6 @@ import org.springframework.web.bind.annotation.RestController;
import com.iqser.red.service.pdftron.redaction.v1.api.model.UntouchedDocumentResponse;
import com.iqser.red.service.peristence.v1.server.settings.FileManagementServiceSettings;
import com.iqser.red.service.persistence.management.v1.processor.exception.ConflictException;
import com.iqser.red.service.persistence.management.v1.processor.model.OCRStatusUpdateResponse;
import com.iqser.red.service.persistence.management.v1.processor.service.persistence.DossierPersistenceService;
import com.iqser.red.service.persistence.management.v1.processor.service.persistence.FileStatusPersistenceService;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.dossier.file.ProcessingStatus;
@ -131,7 +130,6 @@ public class FileStatusProcessingUpdateService {
retryTemplate.execute(retryContext -> {
log.info("OCR Successful for dossier {} and file {}, Attempt to update status: {}", dossierId, fileId, retryContext.getRetryCount());
fileStatusService.updateOCRStatus(OCRStatusUpdateResponse.builder().fileId(fileId).ocrFinished(true).build());
fileStatusService.setStatusFullReprocess(dossierId, fileId, false, true);
return null;

View File

@ -128,7 +128,6 @@ public class DownloadPreparationService {
}
@Transactional
public void createDownload(RedactionResultMessage reportResultMessage) {
DownloadStatusEntity downloadStatus = downloadStatusPersistenceService.getStatus(reportResultMessage.getDownloadId());
@ -141,7 +140,7 @@ public class DownloadPreparationService {
addReports(reportResultMessage.getDownloadId(), storedFileInformations, fileSystemBackedArchiver);
storeZipFile(downloadStatus, fileSystemBackedArchiver);
downloadStatusPersistenceService.updateStatus(downloadStatus.getStorageId(), DownloadStatusValue.READY, fileSystemBackedArchiver.getContentLength());
updateStatusToReady(downloadStatus, fileSystemBackedArchiver);
notificationPersistenceService.insertNotification(AddNotificationRequest.builder()
.userId(downloadStatus.getUserId())
@ -161,6 +160,12 @@ public class DownloadPreparationService {
}
private void updateStatusToReady(DownloadStatusEntity downloadStatus, FileSystemBackedArchiver fileSystemBackedArchiver) {
downloadStatusPersistenceService.updateStatus(downloadStatus, DownloadStatusValue.READY, fileSystemBackedArchiver.getContentLength());
}
private void generateAndAddFiles(DownloadStatusEntity downloadStatus, RedactionResultMessage reportResultMessage, FileSystemBackedArchiver fileSystemBackedArchiver) {
int i = 1;

View File

@ -40,6 +40,12 @@ public class RedactionResultMessageReceiver {
redactionResultMessage.getDownloadId()));
}
receive(redactionResultMessage);
}
public void receive(RedactionResultMessage redactionResultMessage) {
log.info("Received redaction results for downloadId:{}", redactionResultMessage.getDownloadId());
downloadPreparationService.createDownload(redactionResultMessage);

View File

@ -208,7 +208,7 @@ public class DossierTemplateExportService {
}
storeZipFile(downloadStatus.getStorageId(), fileSystemBackedArchiver);
downloadStatusPersistenceService.updateStatus(downloadStatus.getStorageId(), DownloadStatusValue.READY, fileSystemBackedArchiver.getContentLength());
downloadStatusPersistenceService.updateStatus(downloadStatus, DownloadStatusValue.READY, fileSystemBackedArchiver.getContentLength());
} catch (JsonProcessingException e) {
log.debug("fail ", e);

View File

@ -6,6 +6,8 @@ import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
@ -21,14 +23,31 @@ import lombok.extern.slf4j.Slf4j;
@Slf4j
public class FileSystemBackedArchiver implements AutoCloseable {
private final boolean rethrowExceptions;
private final Set<String> createdFolders = new HashSet<>();
private final File tempFile;
private final ZipOutputStream zipOutputStream;
private long tempFileLength;
@SneakyThrows
public FileSystemBackedArchiver() {
this(false);
}
/**
* Controls whether exceptions are re-thrown. Mostly meant for testing.
*
* @param rethrowExceptions If true exceptions caught when handling streams and files will be re-thrown.
*/
@SneakyThrows
FileSystemBackedArchiver(boolean rethrowExceptions) {
this.rethrowExceptions = rethrowExceptions;
tempFile = FileUtils.createTempFile("archive", ".zip");
zipOutputStream = new ZipOutputStream(new FileOutputStream(tempFile));
}
@ -50,11 +69,7 @@ public class FileSystemBackedArchiver implements AutoCloseable {
@SneakyThrows
public InputStream toInputStream() {
try {
zipOutputStream.close();
} catch (IOException e) {
log.debug(e.getMessage());
}
closeStreamAndStoreTempFileLength();
return new BufferedInputStream(new FileInputStream(tempFile));
}
@ -80,26 +95,50 @@ public class FileSystemBackedArchiver implements AutoCloseable {
@Override
public void close() {
closeStreamAndStoreTempFileLength();
try {
boolean res = tempFile.delete();
if (!res) {
log.warn("Failed to delete temp file");
Files.delete(tempFile.toPath());
} catch (IOException e) {
log.warn("Failed to delete temp-file", e);
if (rethrowExceptions) {
throw new RuntimeException(e);
}
zipOutputStream.close();
} catch (Exception e) {
log.debug("Failed to close FileSystemBackedArchiver");
}
}
public long getContentLength() {
closeStreamAndStoreTempFileLength();
return tempFileLength;
}
private void closeStreamAndStoreTempFileLength() {
try {
zipOutputStream.close();
} catch (IOException e) {
log.debug(e.getMessage());
log.warn("Failed to close temp-file stream", e);
if (rethrowExceptions) {
throw new RuntimeException(e);
}
}
if (tempFile.exists()) {
try {
var basicFileAttributes = Files.readAttributes(tempFile.toPath(), BasicFileAttributes.class);
tempFileLength = basicFileAttributes.size();
} catch (IOException e) {
if (rethrowExceptions) {
throw new RuntimeException(e);
}
}
} else {
log.warn("The temp file {} was deleted before it was completely processed", tempFile);
}
return tempFile.length();
}

View File

@ -376,6 +376,15 @@ public class DictionaryTest extends AbstractPersistenceServerServiceTest {
var existingEntries = dictionaryClient.getEntriesForType(createdType.getTypeId(), 0L, DictionaryEntryType.ENTRY);
assertThat(existingEntries.stream().filter(f -> !f.isDeleted()).count()).isEqualTo(5);
var dictionary = dictionaryClient.getDictionaryForType(createdType.getTypeId(), 0L);
var dictEntries = dictionary.getEntries();
assertThat(dictEntries).hasSize(5);
assertThat(dictEntries.get(0).getValue()).isEqualTo(word2);
assertThat(dictEntries.get(1).getValue()).isEqualTo(word1);
assertThat(dictEntries.get(2).getValue()).isEqualTo(word5);
assertThat(dictEntries.get(3).getValue()).isEqualTo(word4);
assertThat(dictEntries.get(4).getValue()).isEqualTo(word3);
}
}

View File

@ -5,11 +5,15 @@ import static org.assertj.core.api.Assertions.assertThat;
import java.io.ByteArrayInputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import org.junit.Before;
import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.iqser.red.service.pdftron.redaction.v1.api.model.RedactionResultMessage;
import com.iqser.red.service.peristence.v1.server.integration.client.DossierClient;
import com.iqser.red.service.peristence.v1.server.integration.client.DownloadClient;
import com.iqser.red.service.peristence.v1.server.integration.client.FileClient;
@ -19,22 +23,35 @@ import com.iqser.red.service.peristence.v1.server.integration.service.DossierTes
import com.iqser.red.service.peristence.v1.server.integration.service.FileTesterAndProvider;
import com.iqser.red.service.peristence.v1.server.integration.utils.AbstractPersistenceServerServiceTest;
import com.iqser.red.service.peristence.v1.server.service.download.DownloadReportMessageReceiver;
import com.iqser.red.service.peristence.v1.server.service.download.RedactionResultMessageReceiver;
import com.iqser.red.service.persistence.management.v1.processor.utils.multitenancy.TenantContext;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.DossierTemplate;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.ReportTemplate;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.ReportTemplateUploadRequest;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.dossier.CreateOrUpdateDossierRequest;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.dossier.Dossier;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.dossier.file.FileModel;
import com.iqser.red.service.persistence.service.v1.api.model.dossiertemplate.dossier.file.WorkflowStatus;
import com.iqser.red.service.persistence.service.v1.api.model.download.DownloadStatus;
import com.iqser.red.service.persistence.service.v1.api.model.download.DownloadStatusValue;
import com.iqser.red.service.persistence.service.v1.api.model.download.DownloadWithOptionRequest;
import com.iqser.red.service.redaction.report.v1.api.model.ReportResultMessage;
import com.iqser.red.service.redaction.report.v1.api.model.StoredFileInformation;
import com.iqser.red.storage.commons.service.StorageService;
import lombok.AccessLevel;
import lombok.SneakyThrows;
import lombok.experimental.FieldDefaults;
public class DownloadPreparationTest extends AbstractPersistenceServerServiceTest {
public static final String USER_ID = "1";
@Autowired
private DownloadReportMessageReceiver downloadReportMessageReceiver;
@Autowired
private RedactionResultMessageReceiver redactionResultMessageReceiver;
@Autowired
private StorageService storageService;
@ -60,76 +77,120 @@ public class DownloadPreparationTest extends AbstractPersistenceServerServiceTes
private FileClient fileClient;
@Before
public void before() {
TenantContext.setTenantId("redaction");
}
@Test
@SneakyThrows
public void testReceiveDownloadPackage() {
var dossierTemplate = dossierTemplateTesterAndProvider.provideTestTemplate();
var testData = new TestData();
var dossier = dossierTesterAndProvider.provideTestDossier(dossierTemplate);
var file = fileTesterAndProvider.testAndProvideFile(dossier);
fileClient.setStatusApproved(dossier.getId(), file.getId(), file.getUploader());
var file11 = fileClient.getFileStatus(dossier.getId(), file.getId());
fileClient.setStatusApproved(testData.dossier.getId(), testData.file.getId(), testData.file.getUploader());
var file11 = fileClient.getFileStatus(testData.dossier.getId(), testData.file.getId());
assertThat(file11.getWorkflowStatus()).isEqualTo(WorkflowStatus.APPROVED);
reportTemplateClient.uploadTemplate(ReportTemplateUploadRequest.builder()
.activeByDefault(true)
.dossierTemplateId(dossierTemplate.getId())
.multiFileReport(true)
.fileName("test.docx")
.template(new byte[]{1, 2, 3, 4})
.build());
var availableTemplates = reportTemplateClient.getAvailableReportTemplates(dossierTemplate.getId());
assertThat(availableTemplates).isNotEmpty();
dossierClient.updateDossier(CreateOrUpdateDossierRequest.builder()
.dossierName(dossier.getDossierName())
.description(dossier.getDescription())
.ownerId(dossier.getOwnerId())
.memberIds(dossier.getMemberIds())
.approverIds(dossier.getApproverIds())
.downloadFileTypes(dossier.getDownloadFileTypes())
.watermarkId(dossier.getWatermarkId())
.dueDate(dossier.getDueDate())
.dossierTemplateId(dossier.getDossierTemplateId())
.reportTemplateIds(availableTemplates.stream().map(ReportTemplate::getTemplateId).collect(Collectors.toList()))
.build(), dossier.getId());
var updatedDossier = dossierClient.getDossierById(dossier.getId(), false, false);
var updatedDossier = dossierClient.getDossierById(testData.dossier.getId(), false, false);
assertThat(updatedDossier.getReportTemplateIds()).isNotEmpty();
downloadClient.prepareDownload(DownloadWithOptionRequest.builder()
.userId("1")
.dossierId(dossier.getId())
.fileIds(Collections.singletonList(file.getId()))
.userId(USER_ID)
.dossierId(testData.dossier.getId())
.fileIds(Collections.singletonList(testData.file.getId()))
.redactionPreviewColor("#aaaaaa")
.build());
var statuses = downloadClient.getDownloadStatus("1");
var statuses = downloadClient.getDownloadStatus(USER_ID);
assertThat(statuses).isNotEmpty();
assertThat(statuses.iterator().next().getLastDownload()).isNull();
DownloadStatus firstStatus = statuses.get(0);
assertThat(firstStatus.getLastDownload()).isNull();
String downloadId = firstStatus.getStorageId();
addStoredFileInformationToStorage(testData.file, testData.availableTemplates, downloadId);
ReportResultMessage reportResultMessage = new ReportResultMessage();
reportResultMessage.setUserId(USER_ID);
reportResultMessage.setDownloadId(downloadId);
downloadReportMessageReceiver.receive(reportResultMessage);
redactionResultMessageReceiver.receive(RedactionResultMessage.builder()
.downloadId(downloadId)
.dossierId(testData.dossier.getId())
.redactionResultDetails(Collections.emptyList())
.build());
List<DownloadStatus> finalDownloadStatuses = downloadClient.getDownloadStatus(USER_ID);
assertThat(finalDownloadStatuses).hasSize(1);
DownloadStatus finalDownloadStatus = finalDownloadStatuses.get(0);
assertThat(finalDownloadStatus.getStatus()).isEqualTo(DownloadStatusValue.READY);
assertThat(finalDownloadStatus.getFileSize()).isGreaterThan(0);
}
@SneakyThrows
private void addStoredFileInformationToStorage(FileModel file, List<ReportTemplate> availableTemplates, String downloadId) {
var storedFileInformationstorageId = downloadId.substring(0, downloadId.length() - 3) + "/REPORT_INFO.json";
String reportStorageId = "XYZ";
// FIXME Check if this is still needed.
// This variable seems to do nothing, if it is not needed it can be removed.
var sivList = new ArrayList<StoredFileInformation>();
var siv = new StoredFileInformation();
siv.setFileId(file.getId());
siv.setStorageId("XYZ");
siv.setStorageId(reportStorageId);
siv.setTemplateId(availableTemplates.iterator().next().getTemplateId());
sivList.add(siv);
// FIXME Check if this is still needed.
storageService.storeObject("XYZ", new ByteArrayInputStream(new byte[]{1, 2, 3, 4}));
storageService.storeObject(storedFileInformationstorageId, new ByteArrayInputStream(new ObjectMapper().writeValueAsBytes(sivList)));
storageService.storeObject(reportStorageId, new ByteArrayInputStream(new byte[]{1, 2, 3, 4}));
}
ReportResultMessage reportResultMessage = new ReportResultMessage();
reportResultMessage.setUserId("1");
reportResultMessage.setDownloadId(statuses.iterator().next().getStorageId());
downloadReportMessageReceiver.receive(reportResultMessage);
@FieldDefaults(makeFinal = true, level = AccessLevel.PRIVATE)
private class TestData {
DossierTemplate dossierTemplate = dossierTemplateTesterAndProvider.provideTestTemplate();
Dossier dossier = dossierTesterAndProvider.provideTestDossier(dossierTemplate);
FileModel file = fileTesterAndProvider.testAndProvideFile(dossier);
List<ReportTemplate> availableTemplates;
private TestData() {
reportTemplateClient.uploadTemplate(ReportTemplateUploadRequest.builder()
.activeByDefault(true)
.dossierTemplateId(dossierTemplate.getId())
.multiFileReport(true)
.fileName("test.docx")
.template(new byte[]{1, 2, 3, 4})
.build());
availableTemplates = reportTemplateClient.getAvailableReportTemplates(dossierTemplate.getId());
assertThat(availableTemplates).isNotEmpty();
dossierClient.updateDossier(CreateOrUpdateDossierRequest.builder()
.dossierName(dossier.getDossierName())
.description(dossier.getDescription())
.ownerId(dossier.getOwnerId())
.memberIds(dossier.getMemberIds())
.approverIds(dossier.getApproverIds())
.downloadFileTypes(dossier.getDownloadFileTypes())
.watermarkId(dossier.getWatermarkId())
.dueDate(dossier.getDueDate())
.dossierTemplateId(dossier.getDossierTemplateId())
.reportTemplateIds(availableTemplates.stream().map(ReportTemplate::getTemplateId).collect(Collectors.toList()))
.build(), dossier.getId());
}
}
}

View File

@ -0,0 +1,52 @@
package com.iqser.red.service.peristence.v1.server.integration.tests;
import static org.assertj.core.api.Assertions.assertThat;
import org.junit.Before;
import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;
import com.iqser.red.service.peristence.v1.server.integration.utils.AbstractPersistenceServerServiceTest;
import com.iqser.red.service.peristence.v1.server.integration.utils.MultithreadedTestRunner;
import com.iqser.red.service.persistence.management.v1.processor.service.persistence.NotificationPreferencesPersistenceService;
import com.iqser.red.service.persistence.management.v1.processor.utils.multitenancy.TenantContext;
import lombok.AccessLevel;
import lombok.SneakyThrows;
import lombok.experimental.FieldDefaults;
import lombok.extern.slf4j.Slf4j;
@Slf4j
@FieldDefaults(level = AccessLevel.PRIVATE)
public class NotificationPreferencesServiceTest extends AbstractPersistenceServerServiceTest {
@Autowired
NotificationPreferencesPersistenceService notificationPreferencesPersistenceService;
final MultithreadedTestRunner multithreadedTestRunner = new MultithreadedTestRunner(2, 1000);
@Before
public void setup() {
TenantContext.setTenantId("redaction");
}
@Test
@SneakyThrows
public void testNotificationPreferencesConcurrent() {
final String userId = "1";
Runnable test = () -> notificationPreferencesPersistenceService.getOrCreateNotificationPreferences(userId);
Runnable afterTest = () -> notificationPreferencesPersistenceService.deleteNotificationPreferences(userId);
var exceptions = multithreadedTestRunner.runMutlithreadedCollectingExceptions(test, afterTest);
for (Exception ex : exceptions) {
log.error("Exception during notification creation", ex);
}
assertThat(exceptions).isEmpty();
}
}

View File

@ -12,11 +12,17 @@ import org.springframework.beans.factory.annotation.Autowired;
import com.iqser.red.service.peristence.v1.server.integration.client.NotificationClient;
import com.iqser.red.service.peristence.v1.server.integration.client.NotificationPreferencesClient;
import com.iqser.red.service.peristence.v1.server.integration.utils.AbstractPersistenceServerServiceTest;
import com.iqser.red.service.peristence.v1.server.integration.utils.MultithreadedTestRunner;
import com.iqser.red.service.persistence.service.v1.api.model.audit.AddNotificationRequest;
import com.iqser.red.service.persistence.service.v1.api.model.common.JSONPrimitive;
import com.iqser.red.service.persistence.service.v1.api.model.notification.Notification;
import com.iqser.red.service.persistence.service.v1.api.model.notification.NotificationType;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
@SuppressWarnings("SpringJavaInjectionPointsAutowiringInspection")
@Slf4j
public class NotificationTest extends AbstractPersistenceServerServiceTest {
@Autowired
@ -25,6 +31,8 @@ public class NotificationTest extends AbstractPersistenceServerServiceTest {
@Autowired
private NotificationPreferencesClient notificationPreferencesClient;
private final MultithreadedTestRunner multithreadedTestRunner = new MultithreadedTestRunner(2, 1000);
@Test
public void testNotificationPreferences() {
@ -106,4 +114,38 @@ public class NotificationTest extends AbstractPersistenceServerServiceTest {
return currentNotifications.iterator().next();
}
@Test
@SneakyThrows
public void testNotificationPreferencesConcurrent() {
final String userId = "1";
Runnable test = () -> notificationPreferencesClient.getNotificationPreferences(userId);
Runnable afterTest = () -> notificationPreferencesClient.deleteNotificationPreferences(userId);
var exceptions = multithreadedTestRunner.runMutlithreadedCollectingExceptions(test, afterTest);
for (Exception ex : exceptions) {
log.error("Exception during notification creation", ex);
}
assertThat(exceptions).isEmpty();
}
@Test
@SneakyThrows
public void testNotificationsConcurrent() {
final String userId = "1";
Runnable test = () -> notificationClient.getNotifications(userId, false);
Runnable afterTest = () -> notificationPreferencesClient.deleteNotificationPreferences(userId);
var exceptions = multithreadedTestRunner.runMutlithreadedCollectingExceptions(test, afterTest);
for (Exception ex : exceptions) {
log.error("Exception during notification creation", ex);
}
assertThat(exceptions).isEmpty();
}
}

View File

@ -0,0 +1,71 @@
package com.iqser.red.service.peristence.v1.server.integration.utils;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import lombok.AccessLevel;
import lombok.RequiredArgsConstructor;
import lombok.experimental.FieldDefaults;
@RequiredArgsConstructor
@FieldDefaults(makeFinal = true, level = AccessLevel.PRIVATE)
public class MultithreadedTestRunner {
int numberOfThreads;
int numberOfExecutions;
public List<Exception> runMutlithreadedCollectingExceptions(boolean stopOnFirstRunWithExceptions, Runnable test, Runnable afterTest) {
List<Exception> allExceptions = new ArrayList<>();
for (int execution = 1; execution <= numberOfExecutions; execution++) {
var threads = new ArrayList<Thread>(numberOfThreads);
var exceptions = Collections.synchronizedList(new ArrayList<Exception>());
for (int threadNumber = 1; threadNumber <= numberOfThreads; threadNumber++) {
Thread t = new Thread(() -> {
try {
test.run();
} catch (Exception e) {
exceptions.add(e);
}
});
threads.add(t);
}
for (Thread t : threads) {
t.start();
}
for (Thread t : threads) {
try {
t.join();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
afterTest.run();
if (stopOnFirstRunWithExceptions) {
if (!exceptions.isEmpty()) {
return exceptions;
}
} else {
allExceptions.addAll(exceptions);
}
}
return allExceptions;
}
public List<Exception> runMutlithreadedCollectingExceptions(Runnable test, Runnable afterTest) {
return runMutlithreadedCollectingExceptions(true, test, afterTest);
}
}

View File

@ -1,57 +0,0 @@
package com.iqser.red.service.peristence.v1.server.utils;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.ObjectOutputStream;
import java.util.SplittableRandom;
import org.apache.commons.io.IOUtils;
import org.junit.Test;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class FileSystemBackArchiverTest {
@Test
@SneakyThrows
public void testFileSystemBackedArchiver() {
try (var fsba = new FileSystemBackedArchiver()) {
SplittableRandom sr = new SplittableRandom();
var data = sr.doubles().limit(1024 * 1024).toArray();
for (int i = 0; i < 10; i++) {
log.info("At entry: {}, using {}MB of memory", i, (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) / (1024 * 1024));
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(data);
byte[] bytes = bos.toByteArray();
bos.close();
var entry = new FileSystemBackedArchiver.ArchiveModel("folder-" + i, "file-" + i, bytes);
fsba.addEntry(entry);
}
File f = File.createTempFile("test", ".zip");
var contentSize = fsba.getContentLength();
try (FileOutputStream fos = new FileOutputStream(f)) {
IOUtils.copy(fsba.toInputStream(), fos);
log.info("File: {}", f.getAbsolutePath());
assertThat(f.length()).isEqualTo(contentSize);
}
log.info("Total File Size: {}MB", f.length() / (1024 * 1024));
}
}
}

View File

@ -0,0 +1,108 @@
package com.iqser.red.service.peristence.v1.server.utils;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.InputStream;
import java.io.ObjectOutputStream;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.util.SplittableRandom;
import org.junit.jupiter.api.Test;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class FileSystemBackedArchiverTest {
private final static byte[] dummyFileContent = new byte[]{1, 2};
@Test
@SneakyThrows
public void testFileSystemBackedArchiver() {
try (var fileSystemBackedArchiver = new FileSystemBackedArchiver(true)) {
SplittableRandom sr = new SplittableRandom();
var data = sr.doubles().limit(1024 * 1024).toArray();
for (int i = 0; i < 10; i++) {
log.info("At entry: {}, using {}MB of memory", i, (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) / (1024 * 1024));
try (ByteArrayOutputStream bos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(bos)) {
oos.writeObject(data);
byte[] bytes = bos.toByteArray();
var entry = new FileSystemBackedArchiver.ArchiveModel("folder-" + i, "file-" + i, bytes);
fileSystemBackedArchiver.addEntry(entry);
}
}
File tempFile = File.createTempFile("test", ".zip");
var contentSize = fileSystemBackedArchiver.getContentLength();
try (InputStream inputStream = fileSystemBackedArchiver.toInputStream()) {
Files.copy(inputStream, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
log.info("File: {}", tempFile.getAbsolutePath());
}
assertThat(tempFile.length()).isEqualTo(contentSize);
log.info("Total File Size: {}MB", tempFile.length() / (1024 * 1024));
Files.delete(tempFile.toPath());
}
}
@Test
public void testContentLengthForTwoEntries() {
try (var fileSystemBackedArchiver = new FileSystemBackedArchiver(true)) {
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Original", "original", dummyFileContent));
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Preview", "preview", dummyFileContent));
assertThat(fileSystemBackedArchiver.getContentLength()).isGreaterThan(0);
}
}
@Test
@SneakyThrows
public void testContentLengthForTwoEntriesAndStream() {
try (var fileSystemBackedArchiver = new FileSystemBackedArchiver(true)) {
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Original", "original", dummyFileContent));
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Preview", "preview", dummyFileContent));
try (InputStream stream = fileSystemBackedArchiver.toInputStream()) {
// Dummy statement to just have the code do something with the stream
//noinspection ResultOfMethodCallIgnored
stream.getClass();
}
assertThat(fileSystemBackedArchiver.getContentLength()).isGreaterThan(0);
}
}
@Test
public void testContentLengthForTwoEntriesWithClosing() {
// deliberately do not use try-with-resources to see if the content-length is available after temp file deletion
var fileSystemBackedArchiver = new FileSystemBackedArchiver(true);
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Original", "original", dummyFileContent));
fileSystemBackedArchiver.addEntry(new FileSystemBackedArchiver.ArchiveModel("Preview", "preview", dummyFileContent));
fileSystemBackedArchiver.close();
assertThat(fileSystemBackedArchiver.getContentLength()).isGreaterThan(0);
}
}

View File

@ -7,13 +7,12 @@
<artifactId>persistence-service</artifactId>
<groupId>com.iqser.red.service</groupId>
<version>1.0-SNAPSHOT</version>
<version>2.0-SNAPSHOT</version>
<packaging>pom</packaging>
<modules>
<module>bamboo-specs</module>
<module>persistence-service-v1</module>
<module>persistence-service-image-v1</module>
</modules>