Building World-Class Data & AI/ML Products in the Cloud

Challenge

Figure Eight aimed to implement a cloud microservice architecture in order to:

• Mitigate scalability and agility issues caused by the nature of their monolithic application
• Simplify and eliminate ambiguity in the build and deploy processes on the developer’s side
• Accelerate the speed of delivery of data tasks for their clients

Figure Eight approached Provectus to assist their engineering team to transform their old monolithic architecture into a new cloud microservice architecture.

Solution

Provectus worked closely with the Figure Eight engineering team to design, standardize, and implement a new cloud microservice architecture to ensure its agility, scalability, and performance.

As a first step, the legacy applications were redesigned, dockerized, and deployed on Amazon Elasticsearch Service. To support Figure Eight’s on-premise clients, their applications were deployed on Kubernetes. This ensured reliable delivery of the company’s platform to every client.

Redis and RabbitMQ services were replaced with ElasticCache, SQS, and Lambda implementations. To enable efficient static content distribution, Amazon CloudFront was used, with content stored in Amazon S3.

Amazon Cloud Formation was used to automate infrastructure distribution (i.e. addition of new microservices). AWS Lambda was used to automate the distribution of notifications in Slack and other messengers. It was also used in the platform’s business logic (i.e. UX automation, users
profiles processing, etc.).

Amazon Elastic MapReduce ​was implemented to enable batch processing of the application logs. To process log statistics, Hive was used. Based on the logged number of errors, refund amounts could be calculated and sent to the clients.

The proposed solution standardized microservices infrastructure and enabled Figure Eight to rapidly decompose legacy applications into 50+ microservices.

Outcome

Figure Eight transitioned their old monolithic architecture to a new microservice one, which allowed to rapidly scale every specific microservice based on its load.

The new architecture allowed to optimize the structure of engineering teams to increase their performance and productivity, ensuring rapid delivery of high-quality training data to Figure Eight’s clients.

The entire release cycle was optimized and shortened. Due to the nature of the microservice applications, independent teams were then able to work on independent releases.

The transition to microservices eradicated the exponentially growing amount of inefficiencies and allowed Figure Eight to reduce the number of dependencies in the source code and cut technical debt.

Figure Eight managed to deliver a new microservice in just a day, including self-serve deployment, monitoring, service discovery, sandbox and production environments. The service uptime of the new architecture reached 99.99% and scaled the Figure Eight platform horizontally. The AWS costs were reduced by over 10%.