Kafka Consumer Simulation and Scaling

CrowdStrike Computer Science, 2020–21

Liaison(s): Luke Hunter ’03, Julius Lauw ‘20
Advisor(s): Lucas Bang
Students(s): Anna Fry (PM-S), Ben Hinthorne (PM-F), Arielle Isaacs, Shivam Malpani

CrowdStrike is a leading provider of cloud-native endpoint protection. The changing volume of security event traffic motivates the scaling of consumer groups for CrowdStrike’s Kafka clients. The CrowdStrike Clinic Team is building upon an existing automatic scaling algorithm and testing environment to visualize Kafka scaling decisions made by the algorithm. They are comparing these decisions to those made by the algorithm in Kubernetes HPA. The most effective algorithm will strike a balance between the cost of deploying consumers and the potential for delayed processing to best support CrowdStrike and its customers.