Providing Music Fans with the Best Possible Concert Experiences Online with AI

Tackling Growth Seasonality and User Preferences with AI Recommendations and Analytics

Nugs.net is a popular music streaming platform that offers a unique experience of live performances, with exclusive content and coverage of concerts for a variety of live performers.

The platform offers a wide selection of high-quality audio and video recordings of performances from a variety of genres, including rock, jazz, country, and more. The recordings are available for streaming and download on their website and mobile application, where subscribers can access exclusive content and features, such as live webcasts, archives of past shows, and artist interviews.

Nugs.net is known for partnering with musicians and bands, providing them with a platform to sell and distribute their live recordings directly to fans. With its focus on live music and its community-driven approach, Nugs.net has become a go-to destination for music lovers looking for an authentic and immersive concert experience.

Provectus is honored to be a part of Nugs.net’s journey, supporting them in achieving their diverse goals for more than eight years.

As a strategic partner, Provectus has helped Nugs.net to develop and realize various successful strategies, spearheading the company’s product development and growth hacking efforts.

Notably, our end-to-end product development services include:

• Consulting on product and engineering strategies
• Implementation of data-driven growth strategies
• Building a robust foundation for product and service analytics
• Product design, including mobile, web, and backend development
• AI adoption and ML engineering

The partnership between Provectus and Nugs.net has been a remarkable success. Provectus has consistently provided not only engineering deliverables, but also met high- and low-level KPIs. Our strategic collaboration has been instrumental in the successful achievement of Nugs.net’s goals and objectives.

Major Business Problems and KPIs

Unlike other music streaming platforms, Nugs.net is greatly impacted by growth seasonality. Users typically stick to newly performed and released concerts and recordings. But those are highly dependent on artists’ schedules, which can be disrupted by various factors, including illness, cancellations and venue shutdowns, making for an unpredictable growth strategy for Nugs.net.

To mitigate the impact of seasonality, Provectus suggested rolling out a user retention optimization strategy that included different growth hacking tactics and product development programs. User experience personalization through AI-powered content recommendations was identified as a potential solution.

Another challenge that Provectus identified was a user tendency to stick to one or two favorite artists or genres, making Nugs.net even more dependent on growth seasonality for retention.

To break this pattern, Provectus suggested enhancing the discoverability of existing content in the Nugs.net catalog, and motivating users to listen to it.

Other suggestions included:

• Improving listening diversity
• Motivating users to discover new artists and genres
• Enhancing catalog breadth observability, especially for new users
• Encouraging users to follow a broader diversity of artists

All of these goals were supported by in-depth research by the Provectus team. Specifically, we proved that:

• Users who listen to a diverse range of artists have dramatically higher retention

• Users who listen to older shows have much higher retention

• Trial users who followed multiple artists during a trial period have dramatically higher retention

To prove these hypotheses in practice, Provectus broke down project KPIs into high-, middle-, and low-level KPI categories, using them as beacons to navigate Nugs.net’s digital and AI transformation.

The high-level KPIs included trial and paid user retention and activation conversion. The middle-level KPIs influencing high-level KPIs were monthly average playback duration per user* and playback stickiness (how many days per week a user retains listening). The low-level KPIs included the average followed artists per user, diversity of playbacks (average number of different artists played per month), proportion of old vs new shows played, and content discoverability (what it takes to discover shows from the bottom of the catalog).

Note: Monthly average playback duration per user was identified as the most important metric, because it measures the level of engagement and satisfaction of users. The longer users listen to music on the platform, the more valuable they are to the business in terms of generating revenue through subscriptions and advertising. When users listen to music for longer periods of time, they are more likely to discover new artists and songs, and are more likely to become loyal users. This metric can also provide insights into the quality and relevance of content being offered, as well as the effectiveness of personalized recommendations and user experience. This information can be used to improve the platform’s offerings and tailor its marketing efforts, resulting in higher user retention rates and increased revenue over time.

Behavioral Analytics with Amplitude

Provectus is a certified partner of Amplitude, a leading product analytics platform that helps businesses understand user behavior and improve product experiences. Provectus used Amplitude for advanced behavioral analytics, delivering flexibility and power in all product management processes with a top-notch data-driven framework. Product ideas, hypothesis verification, and bottleneck identification became data-driven, enabling us to convert engineering efforts and investments into precisely calculated ROIs.

Notably, Amplitude was applied to:

• Identify retention dependency on content consumption diversity and seasonality by analyzing user behavior data
• Raise hypotheses on how to improve retention drops with direct influence on those dependencies
• Make data-driven predictions based on yesterday’s weather and statistical analysis, and use them to suggest project KPIs
• Create an AI recommendations MVP with an effective metrics dashboard

Amplitude enabled us to attain comprehensive understanding, clear visibility, accurate predictability, and effective control of our product, to efficiently target and successfully execute our KPIs.

Major Technological Challenges to Address

Provectus approached the project with confidence, drawing on our extensive experience in AI adoption and ML development, and using our Customer 360 AI solution as a robust foundation for the AI recommendations project.

Crystal Engine leverages our own toolset and engineering foundation, enabling us to quickly launch AI-enabled, Customer 360 solutions while avoiding technological risks. Building a custom solution on top of our product proved to be more efficient than starting from scratch.

Despite some technological challenges, we were able to successfully overcome them.

The primary objective of Provectus was to:

Keep the project within budget while proving our basic hypotheses and minimizing the project scope and timeline.

We also wanted to cover a few personalization use cases, to avoid misleading hypothesis verification results. Our goal was to increase feature discoverability and set the direction for future iterations.

Another important objective was to:

Keep the architecture scalable, sustainable, and as native to Azure as possible, to minimize costs.

We wanted to avoid infrastructural overhead, and over-investing in operationalization. Midway through the project, we pivoted to focus more on optimizing cost-efficiency, which was under leadership scrutiny at that time. Although the architectural plan was already cost-efficient, we were able to strategically allocate computing resources with accurate pipeline scheduling, and by trimming down data sets to avoid excessive processing.

Along the way, we discovered that most of the pre-built models were not relevant due to data sparsity. We also faced the challenge of balancing recommendation precision versus diversity and catalog coverage. Initial experiments with different models showed promising results in precision and effectiveness, but post-processing results did not initially hit the mark. We faced a lot of experimentation and turnarounds, focusing on performance optimization to achieve perfect results and hit our business goals.

Finally, a challenge arose from different user personas, each of them having unique behavioral patterns. We had to consider users’ “discovery and diversity propensity,” as greater content diversity could be beneficial for one user, but completely destructive for another. Therefore, we had to deal with “sliding diversity” that could adapt to the behavior patterns of individual users.

Given the nature of Nugs.net as a product, Provectus could not directly apply common industry solutions to help them achieve their main business goals. We needed to create a sophisticated and specific custom AI solution that would meet the unique needs of the customer.

Moving from Idea to Product: Initial Workshops, Product Design, and AIML Development

We began the project by conducting a series of comprehensive workshops to delve into business objectives, goals, and hypotheses, define key performance indicators, and analyze user flow and data. This allowed us to gain a holistic understanding of the project before proceeding with development. At Provectus, we believe that precise product and problem discovery, as well as architectural and feature design, is crucial to any project’s success.

After completing the workshops, Provectus had a well-defined vision for the MVP of Nugs.net’s AI-powered recommendation and personalization system. With focused efforts and efficient resource allocation, we were able to deliver the MVP on a tight budget and in an impressive three-month timeframe. This rapid time-to-market enabled us to test our primary product hypotheses and promptly adapt our product roadmap, which is crucial in such a dynamic market.

Note: At Provectus, we do not view any AIML engineering efforts in isolation. Even well-crafted and engineered AI systems will not bring business success unless results are delivered to the end consumer effectively, and packaged with a seamless UI/UX. Our ML team works in close collaboration with product designers, product managers, and user researchers to avoid underperformance of advanced AI systems in terms of executing business goals.

Quality Control Framework

Ensuring quality was a top priority for the Provectus team during the development of the ML models. Our objectives were to achieve “technological quality,” ensure high model performance in real-world scenarios, and deliver “business quality” that correlates with Nugs.net’s business objectives.

One of the biggest challenges we faced was connecting AI/ML metrics, such as precision, to such product KPIs as recommendation impressions (a metric indicating how many times a certain recommendation is shown to a user).

To address this challenge, we proposed a comprehensive “quality control framework” consisting of three metric assessment processes:

• AI/ML metrics
• Product metrics
• Expert & User opinions

In this framework, every element is considered in relation to the others, allowing Provectus and Nugs.net to gain a better understanding of their correlation and direction for the project.

By implementing this framework, we were able to ensure that our ML models not only met technological requirements, but also delivered results that aligned with the business objectives of Nugs.net. The combination of these processes allowed us to achieve a higher level of quality control and make more informed decisions for the project.

Realizing the Benefits of AI Recommendations to Drive Efficiencies and Business Growth

Nugs.net’s AI-powered recommendation system has yielded significant results for the company. Positive feedback from users and stakeholders validates the efficacy of the recommendations. However, it was the objective product KPIs results that truly stood out.

One key metric that improved was artist following. After the system’s release, there was a significant influx of artist followings, which stabilized at an almost 2x rate.

Playback diversity also increased significantly, breaking through the four artists per month threshold, and is predicted to reach six in 2023. This trend pushed the product into a significantly more powerful retention tier.

Another interesting finding was that users started to play up to 15% more “old” shows versus newly released ones, indicating deeper and wider catalog observability.

As a result, Nugs.net’s daily user retention has nearly doubled.

According to the Provectus analysis, there is still more potential for the AI recommendation system to increase user engagement through personalization, which can be achieved through further improvements of UI/UX, and by adding new features.

The Provectus team also discovered that users who listened to recommendations had at least a 40% higher retention rate.

Their stickiness was completely different, too.

Most importantly, trial users who listened to recommendations had more than a 2x higher trial-to-paid conversion rate.

These findings demonstrate the effectiveness of the AI-powered recommendation system in improving UX and content personalization, leading to dramatically increased user engagement and retention.

Based on product performance results, Provectus calculated that the ROI for the AI recommendations MVP project was projected to be 183% in 12 months after release. This outstanding result is attributed to the complex work of the product, design, mobile, web, backend, and AIML teams.

To sum it up:

• The AI/ML-enabled personalized recommendations project has significantly improved key product metrics of Nugs.net’s platform and demonstrated a high ROI over the long term.
• The findings validate the importance of taking a holistic approach to AI adoption and ML development, combining product design, product development, AIML engineering, and Data & Analytics work to achieve outstanding results.

With the help of Provectus, Nugs.net is now better prepared to thrive in an increasingly competitive market by providing the highest level of UX personalization enabled by AI. The company is committed to enhancing their services to meet the evolving needs of both music fans and performers. The leaders of Nugs.net are confident that their partnership with Provectus will continue to help them to achieve their growth objectives in years to come.

Moving Forward

1. Learn more about the Provectus Customer 360 solution and Crystal Engine tool
2. Download the Provectus AI/ML Transformation Canvas
3. Apply for Customer-Centric AI Transformation Program to get started