AWS Migration & Re-Architecture
IMVU generates critical insights on customer lifetime behavior through data streaming and advanced analytics on AWS
400 Spark Jobs
Project Duration: 5 months
1PB Processed Daily
IMVU was looking to enhance their platform’s analytics capabilities by enabling advanced real-time analytics and data streaming. The platform’s aging and outdated infrastructure was not scalable, however, and it could not be used as a foundation for IMVU’s cloud-based products. It also prevented IMVU from generating a range of critical reports on customer in-game behavior, as well as did not allow for AI-powered applications.
Lacking modern data architecture (no CI/CD), the platform could not rapidly deploy new jobs, which made the IMVU team to run analytics jobs once a day, at night. Not only did that make analytics more complex and complicated, but it also created multiple bottlenecks. For instance, late reports resulted in inaccurate assumptions on customers’ in-game purchases, which caused slower sales and loss of profit. On top of that, the analytics team did not have a test environment to efficiently check analytics assumptions.
Technology-wise, the IMVU platform was powered by a 90-node on-prem Hadoop cluster. Compute was coupled with storage, with 300-400 Hive/Spark SQL-like jobs used to reprocess data from scratch all the time, which was cost-inefficient. IMVU needed a new solution to run the cluster only when required.
- Tightly coupled compute and storage required buying excess capacity
- Hadoop cluster was over-utilized during peak hours and under-utilized at other times
- Solution as such resulted in high costs and low efficiency
Having tried to enhance their data architecture collaborating with other consultancies, IMVU teamed up with Provectus to drive innovation, implement data streaming, enable advanced analytics, and build a few AI-powered apps (recommendations, abuse detection, etc.) to improve customer retention.
Provectus took on a challenge to re-architect IMVU’s platform to enable data streaming and advanced analytics, as well as developed a comprehensive data strategy covering IMVU’s next-steps.
To ensure the effectiveness and cost-efficiency of data analysis on the IMVU platform, their Hadoop cluster was migrated to Amazon EMR. IMVU’s Hadoop Distributed File System (over 300 TB of data) was moved to Amazon S3 using multiple AWS Snowballs.
Along the way, Provectus optimized Hive/Spark SQL-like jobs to introduce a streaming-first architecture with key techniques, such as change data capture, streams enrichment, and a metadata-driven data lake in Amazon S3 and AWS Glue. Apache Airflow was used for offline orchestration of data pipelines. To ensure smooth migration and rolling upgrades, Kafka data streams were mirrored from the on-prem environment to the cloud.
As part of every deployment, Provectus implemented continuous delivery using Terraform, Kubernetes, Jenkins and Prometheus, OpsGenie integrated with Bitbucket, Jira, and Slack.
IMVU has received a scalable, cloud-native infrastructure to enable data streaming and advanced analytics. With those tools in place, IMVU can now take advantage of vast amounts of data generated by users to look into customer lifetime behavior, enabling a range of reports including but not limited to in-game purchases, customer engagements, and avatar preferences. IMVU’s analysts can also generate reports much faster and whenever they need to.
Infrastructure-wise, IMVU has overcome significant technological barriers to fast and seamless code deployments. By having their platform re-architected for AWS, IMVU has created a solid foundation for their cloud-based products, as well as introduced the potential for using Artificial Intelligence and Machine Learning for customer retention.
IMVU seeks to keep cooperating with Provectus to further improve their platform’s analytics capabilities through Machine Learning. The company aims to increase its real-time analytics potential to closely monitor customer-to-customer engagements (e.g. detect abuse in chat rooms) using AI.
Looking to explore the solution? Contact Us!