Consultant - Kafka Admin

Date Opened: Mar 4, 2019

Location: Bangalore, Karnataka, India

Company: Allstate Insurance Company

Job Description

Kafka Administrator – NiFi (Data Flow – Track)

Band: B1/B2

Shift: 1PM to 9.30PM Shift

 

Position Summary

As a Kafka Administrator, you will be responsible for assisting with the design, architecture, implementation, and on-going support of Arity’s Kafka clusters on AWS. 

 

Key Responsibilities

  1. Manage large scale multi-nodes Kafka cluster environments residing on AWS.
  2. Handle all Kafka environment builds, including design, capacity planning, cluster setup, performance tuning and ongoing monitoring.
  3. Perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka Cluster.
  4. Creation of key performance metrics, measuring the utilization, performance and overall health of the cluster.
  5. Capacity planning and implementation of new/upgraded hardware and software releases as well as for storage infrastructure.
  6. Research and recommend innovative, and where possible, automated approaches for system administration tasks.
  7. Ability to closely calibrate with product managers and lead engineers.
  8. Provide guidance in the creation and modification of standards and procedures
  9. Proactively monitor and setup alerting mechanism for Kafka Cluster and supporting hardware to ensure system health and maximum availability

 

Technical Experiences Required

  1. 3+ years of solid Kafka Admin experience in managing critical 24/7 applications
  2. Design, build, assemble, and configure application or technical architecture components using business requirements.
  3. Hands-on experience with Kafka clusters hosted on Amazon cloud is a plus.
  4. Experience in Kafka build pipelines using Ansible, Cloud formation templates, shells etc.
  5. Experience in Jenkins, GitHub
  6. Experience in implementing security & authorization (permission based) on Kafka cluster.
  7. Experience in open source Kafka, zookeepers, Kafka connect, schema registry Avro schemas.
  8. High availability cluster setup, maintenance and ongoing support
  9. Create topics, setup redundancy cluster, deploy monitoring tools, alerts Has good knowledge of best practices
  10. Exposure to Kafka APIs 
  11. Hands on experience in standing up and administrating Kafka platform which includes creating a backup & mirroring of Kafka Cluster brokers, broker sizing, topic sizing, h/w sizing, performance monitoring, broker security, topic security, consumer/producer access management(ACL) 
  12. Knowledge of Kafka API (development experience is a plus)
  13. Provide technical expertise and guidance to production support staff.
  14. Involvement with grouping/clustering and high volume systems.
  15. Handle all Kafka environment builds, including design, capacity planning, cluster setup, performance tuning and ongoing monitoring.
  16. Manage large scale multi-nodes cluster environments residing on AWS
  17. perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka cluster
     
  18. Knowledge of best practices related to security, performance, and disaster recovery.

 

Job Qualifications