Anomaly detection of monitoring data in Scala ecosystem

Machine learning for programmers could be a tricky subject. We’d like to present a system that predicts the potential user over charging problem by detecting anomalies in the monitoring data. It wouldn’t be a data scientist talk, rather a one focused on implementation details using tools from Scala ecosystem. What’s difficult in the billing system for the shared cloud infrastructure? Noisy neighbour is an effect of monopolisation of bandwidth, disk I/O, CPU and other resources.

Virtual machines and applications that share the infrastructure suffer from uneven cloud network performance. In such environment, without proper analysis of real-time monitoring data, it’s not easy to distinguish valid complaints from ungrounded grumbles. Akka Streams with Kafka, Spark and Cloudant is a nice mixture of tools that can help you build a robust streaming solution without a lot of effort in both the development and maintenance phase. The presentation is dedicated to software developers, system architects and data scientists who would like to learn the quick and painless way of building reactive systems.