How can skincare companies modernize their data infrastructure to support real-time business operations?

Skincare companies can modernize by transitioning from REST API-based on-premises systems to cloud-based real-time data streaming architectures using Apache Kafka and Confluent. This involves implementing Kafka producers to extract data from existing systems like SAP Commerce Cloud, creating multiple Kafka topics for different business functions such as customer enrollments and orders, and deploying Kafka consumers on cloud platforms like GCP Kubernetes. The approach enables real-time data access for sales forecasting, partner management, and operational decision-making while providing scalability and high availability.

What are the main challenges skincare businesses face with traditional data architectures?

Traditional data architectures in skincare businesses often suffer from slow data processing speeds that become unacceptable as business volume grows. Companies typically rely on complex IT infrastructures with multiple systems like SAP Commerce Cloud, Gigya, Apache Solr, and Salesforce Marketing Cloud connected through REST API requests. This creates bottlenecks that prevent real-time access to critical sales data, customer information, and order details. For companies with multi-level marketing arrangements and sales partners, these delays significantly impact the ability to make accurate sales forecasts and provide timely support to partners.

How does Apache Kafka improve data streaming for skincare manufacturers?

Apache Kafka improves data streaming by enabling real-time data flow from source systems like Hybris databases to multiple endpoints without the delays of REST API requests. Kafka producers written in Java extract data and store it in organized topics covering customer enrollments, orders, products, and KPIs. Kafka consumers then poll these topics and distribute data to various end systems. This architecture supports over 40 different topics and consumers, deployed on cloud platforms like GCP Kubernetes for scalability. The streaming approach eliminates bottlenecks and provides immediate access to business-critical information.

What role does the Dead Letter Queue play in preventing data loss for skincare companies?

The Dead Letter Queue serves as a safety net that captures any failed data transmissions to prevent information loss in skincare company operations. When Kafka consumers encounter technical failures during data processing, the failed messages are automatically stored in this catch-all topic. End system consumers monitor both their designated topics and the DLQ for complete data coverage. Technical failure messages can be automatically resynced to consumer applications, while business failure messages are sent back to e-commerce applications for reprocessing. This ensures comprehensive data integrity across all business operations.

How do skincare companies monitor performance in real-time data streaming systems?

Performance monitoring involves using Kafka analytics APIs and Prometheus to track key performance indicators like consumer lags, active connections, and traffic volume. These metrics are pulled into handler applications that continuously assess system health and performance. Companies set specific performance thresholds and configure automated alerts to proactively identify potential issues before they impact operations. Additionally, tools like Splunk assist with debugging logs and troubleshooting. This comprehensive monitoring approach ensures consistent system performance and enables rapid response to any emerging problems in the data streaming infrastructure.

What business benefits do skincare companies achieve with cloud-based streaming architectures?

Cloud-based streaming architectures deliver multiple business advantages including zero downtime through redundant virtual machines, real-time data access enabling rapid business decisions, and high availability that increases partner satisfaction. Companies can rapidly shift market focus by adding new Kafka topics and consumers without system restarts. The architecture supports cost-effective future expansion through dynamic scaling of bandwidth, processing power, and storage without additional hardware investment. For skincare companies with complex partner networks, this translates to better sales forecasting, improved partner support, and enhanced operational agility in fast-changing market conditions.

Why is real-time data access crucial for skincare companies with multi-level marketing structures?

Real-time data access is essential for skincare companies with multi-level marketing because it enables accurate sales forecasting and immediate partner support. These business models rely heavily on sales partners who need current information about customer enrollments, orders, product performance, and commission calculations. Delays in data availability can impact partner satisfaction and business decisions. With real-time streaming, companies can provide partners with up-to-date sales metrics, inventory levels, and customer insights. This immediate access to business-critical information allows both the company and its partners to make fast, well-informed decisions essential in today's competitive skincare market.

Real-Time Data Insights with Confluent and Kafka

About the Client

A well-known American manufacturer of skincare products with $1.5B annual sales.

The Challenge

Over the years, the company developed a complex IT infrastructure to communicate sales information with company management and its partners. The infrastructure includes SAP Commerce Cloud (formerly known as Hybris), Gigya, Apache Solr search engine, and Salesforce Marketing Cloud, among other technologies.

The company’s business relies upon its sales partners, organized in a multi-level marketing arrangement. They stored most of their sales data in a Hybris database, including customer and order data, product information, and coupons,

As business volume grew, the time taken to get accurate real-time data streaming information information became unacceptably slow. The company identified several bottlenecks that primarily stemmed from its reliance on REST API request-based components.

Our Information Analytics team was called in to support the company’s decision to move to a cloud-based streaming architecture based upon Apache Kafka.

In short, the company was looking to:

Modernize their complex on-premises infrastructure: The company wanted to transition as much data infrastructure as possible from on-premises to the cloud.
Minimize reliance upon a REST API-based architecture: The company's legacy REST API-based architecture was part of their problem. They needed help in transitioning to a modern real-time data streaming architecture based on Kafka.
Make accurate sales forecasts: To make accurate sales forecasts and to best assist its partners, the company needed real-time access to data.

Our Solution

Our IA team developed a solution that sends data from the Hybris database to different third parties for reports and further processing. To achieve this, we developed a series of Kafka producers written in Java that produce data from Hybris and store them in Kafka topics. Then, we developed a set of Java-based Kafka Consumers that poll the topics and send data to various end systems.

Our IA team designed over 40 Kafka topics on Confluent Kafka brokers to manage various aspects of the business - customer enrollments, customer updates, orders, products, event registration, and many customer KPIs.

Additionally, we developed a matching set of Kafka consumers, also written in Java, to read the associated topic streams. The 40+ Kafka consumers were deployed on a Kubernetes engine in GCP. Our team configured multiple GCP pods for each consumer, providing scalability and high availability.

Failed data is stored in a catch-all Kafka topic called Dead Letter Queue (DLQ) to avoid data loss. End system consumers read the DLQs in addition to their designated topics. If necessary, the consumers are programmed to DLQ technical failure messages for automatically resyncing the data to consumer applications and other endpoints. The DLQ messages that pertain to business failures are sent back to the e-commerce applications for reprocessing.

Our team used Kafka analytics monitoring APIs - and Prometheus to pull KPIs such as consumer lags, active connections, and traffic volume into a handler application to ensure consistent performance. We also set performance thresholds and configured alerts. Our team used Splunk to assist with debugging logs.

Key takeaways from the solution include:

Shift to a cloud-based streaming infrastructure: Our team assisted the company in moving away from a REST API-based on-premises system to a modern cloud-based real-time data streaming infrastructure.
No data loss:The new system can easily handle a massive amount of company data with no loss. Another unique feature of our solution captures and reprocesses any failed data in the catch-all Kafka DLQ topic.
Zero downtime: The new system features multiple redundant cloud-based virtual machines that ensure zero downtime.

Business Impact

Real-time access to data allows rapid response: The company and its partners now have real-time access to data, allowing them to make fast and well-informed business decisions - essential in today’s fast-changing world.
High availability increases partner satisfaction:The solution features multiple redundancies ensuring 100% uptime. With its performance threshold alerts, the new system’s monitoring allows the company to proactively address potential issues before they affect system performance.
Rapidly shift market focus without impacting operations: Kafka makes it easier to add topics and consumers without restarting the entire system.
Cost-effective future expansion: The cloud-based real-time data streaming solution can dynamically upscale its bandwidth, processing power, and storage space, which allows the business to grow without additional development or hardware investment.

Related Capabilities

Utilize Actionable Insights from Multiple Data Hubs to Gain More Customers and Boost Sales

Unlock the power of data insights buried deep within your diverse systems across the organization. We empower businesses to effectively collect, beautifully visualize, critically analyze, and intelligently interpret data to support organizational goals. Our team ensures good returns on the big data technology investment with effective use of the latest data and analytics tools.

Related Services

Technologies Used

Confluent Kafka
Prometheus
SAP Commerce Cloud (Hybris)
GCP BigQuery
GCP Firestore
SAP Gigya
Kubernetes
Oracle Java