Centralized Contract Store
I worked at an insurance company where the various divisions relied on multiple source systems, each with unique data formats, to manage insurance contracts. This diversity posed significant challenges for software projects that required access to these contracts. Integrating each source system individually for every project was inefficient and unnecessarily complex.
To resolve this, the company introduced the Centralized Contracts Store (CCS), designed as an Operational Data Store to standardize contract data across all divisions. The CCS provided a unified format, streamlining access and integration for various applications. I was actively involved from the inception of this initiative and would like to share a brief overview of the project.
Project Overview
The following diagram illustrates a simplified system context of the Centralized Contracts Store:
Several source systems, each managing different insurance divisions, provide the CCS with contracts in various formats. The CCS transforms these formats into a unified standard and stores them in its own database. Specifically, we adopted the BiPRO format, as it is the standard for data transmission between insurance companies and external parties.
Outcome
The CCS project was a tremendous success! Over time, more and more contract consumers benefited from the unified format. Additionally, we gradually integrated more source systems, enabling all contract consumers to benefit from the expanded data coverage.
The source systems were no longer limited to the company’s internal systems. Through partnerships with other insurance companies, we also integrated their source systems, further enhancing the value of the CCS for all consumers.
With the CCS’s ease of use, stakeholders began to propose additional requirements. One of the most notable feature requests we implemented was a “Google-like” full-text search. For example, a call center employee can simply enter a customer’s name, their birthdate, or even the city they live in into a single search field. The system then returns a list of matching contracts, making it easier and faster to retrieve relevant information.
Key Technologies and Tools
We implemented the CCS using several Spring Boot services that provide REST endpoints. Access to these endpoints is secured with OAuth2, ensuring that only authenticated and authorized users can access sensitive contract data.
Given the deeply nested structure of the BiPRO format, we chose MongoDB as the document database. For format transformation, we utilized MapStruct to simplify and streamline the mapping process.
For seamless development and integration testing, we used Docker Compose.
To support the “Google-like” full-text search functionality, we integrated Elasticsearch, enabling efficient and intuitive search capabilities.
Learnings
When we started the project, the hype around microservices was at its peak, and we implemented many of them. Over time, we realized we had created too many. To simplify our development and deployment processes, we gradually merged several microservices.
Maintaining our arc42 documentation was incredibly helpful in guiding these decisions. It facilitated discussions about which components should implement specific features, and which microservices should be merged or kept separate. This is something I highly recommend for every software project!
We valued MongoDB for its advantages, such as its flexible schema-free data structure. However, we also faced challenges, particularly the extra effort required to ensure smooth and reliable data migrations. Despite these hurdles, choosing MongoDB proved to be the right decision for the CCS.
Some early REST API design decisions caused pain points later in the project. Fortunately, we planned for REST API versioning early on, which made it straightforward to transition. We supported both the old and new APIs during the migration, ensuring a smooth process.
We were amazed by the power of Elasticsearch! To be fair, its setup is substantial, and the resource requirements scale with the size of the data. Since we manage a large volume of contract data, this was significant. However, the investment was worthwhile. The “Google-like” full-text search functionality turned out to be highly impressive.
Additionally, the Elasticsearch infrastructure enabled us to implement more complex requirements, such as querying contracts based on a lot of attributes, without needing to create an overwhelming number of indices in MongoDB.