Our IA team studied the problem and recommended the implementation of Overwatch, a tool developed by Databricks, for analyzing log data from Databricks workspaces.
Databricks Overwatch is a powerful monitoring and alerting solution designed to provide insights into the performance, cost, and usage of Databricks workspaces and clusters. Overwatch offers granular details such as pipeline performance, cost, ingress, and egress data. It can assist the company by optimizing data-driven decision-making. It allows the company to capture workspace activities through structured datasets.
The team collaborated closely with company stakeholders to ensure Overwatch's successful deployment and integration across all relevant platforms. Additionally, our engineers designed the solution to be extensible, making it suitable for use with multiple workspaces.
The new system enables user activity logging through Databricks Event Hub integration and extracts the logged data on clusters, notebooks, account logins, and jobs using Overwatch. Extracted data is structured in the form of delta tables to be used for dashboard creation and further analysis.
Here are a few highlights of the expanded system:
- Effective user activity monitoring: The company can now get an accurate count of active users and the number of unique logins.
- Comprehensive cloud usage metrics: The new system gives the company real-time information on its Databricks component usage. The insights drawn from usage metrics allow the company to allocate its resources more efficiently, saving time and money.
- Fully automated extensible solution: GSPANN engineers designed the implementation to be fully automated, extensible, and reusable. The same solution can easily be extended to multiple workspaces, saving the company significant future development costs.
