Automated Kafka Consumer Scaling

CrowdStrike Computer Science, 2019-20

Liaison(s): Eric Schow, Luke Hunter ’03
Advisor(s): Lucas Bang
Students(s): Ali Parker (PM), Owen Gillespie, Quinn Hirsohn, Julius Lauw, Zack Rossman

CrowdStrike is on the forefront of cyber security as a leading provider of cloud-native endpoint protection. The goal of the CrowdStrike Clinic project is to develop an application to automate consumer group scaling decisions in real-time for CrowdStrike’s Kafka clients. The system will ingest live application metrics, use an algorithm to make scaling decisions and send these scaling decisions to an infrastructure endpoint. The aim of these scaling decisions is to strike a balance between the cost of deploying consumers and the potential for delayed processing in order to minimize manual intervention during the scaling process.