Kafka Streams

⬢ ระดับ 2เทคนิค

สูง

ผลกระทบต่อเงินเดือน

6 เดือน

เวลาในการเรียนรู้

ยาก

ระดับความยาก

—

อาชีพ

ภาพรวม

Kafka Streams is a library for building real-time data pipelines on Apache Kafka: process millions of events per second with windowing, joins, and state stores. Career path: Practitioner (basic streams, KStream/KTable, $110-140k) → Specialist (windowing, joins, state management, $140-180k) → Architect (schema evolution, Kafka cluster tuning, multi-datacenter replication, $180-240k+) over 5-7 months. Salary premium: $30k-$70k above base (Data Engineer → Streaming Engineer tier). Tools: Confluent Cloud, ksqlDB, Kafka Connect, Schema Registry, MirrorMaker 2. Competes with Flink (lower learning curve, higher operational complexity) and Spark Streaming (batch microbatching, not true streaming).

Kafka Streams คืออะไร

Stream processing library for Apache Kafka. Build real-time data pipelines and streaming applications. Process events as they happen for live analytics, fraud detection, monitoring. Learning Curve: Hard (distributed systems, event-driven architecture)

🔧 เครื่องมือและระบบนิเวศ

Apache KafkaKafka StreamsksqlDBConfluent CloudMSK (AWS Managed Streaming for Kafka)Schema RegistryKafka ConnectApache FlinkKafka MirrorMakerStrimzi (Kubernetes Kafka operator)

📋 ก่อนที่คุณจะเริ่มต้น

Distributed Systems Backend Development Apache Spark Message Queues

💰 เงินเดือนตามภูมิภาค

ภูมิภาค	เจ้าหน้าที่จูเนียร์	ระดับกลาง	เจ้าหน้าที่อาวุโส
USA	$115k	$155k	$220k
UK	£70k	£100k	£145k
EU	€75k	€105k	€155k
CANADA	C$120k	C$160k	C$230k

🎓 ใบรับรอง

Confluent Certified Developer for Apache Kafka Confluent Certified Associate for Apache Kafka AWS Certified Big Data - Specialty

⚖ เปรียบเทียบกับ

Apache Spark Event Driven Architecture Websockets Real Time

❓ คำถามที่พบบ่อย

Kafka vs RabbitMQ, which should I use?

Kafka: persistent log, replay, high throughput (trillions/day), topic-based, distributed by default. RabbitMQ: queuing, immediate delivery, lower latency for small workloads, easy to operate. Use Kafka for data pipelines (logging, event streaming, analytics). Use RabbitMQ for task queues (job processing, microservice messaging). Kafka is the standard for event-driven architectures at scale.

What is exactly-once semantics and why is it hard?

Exactly-once = every event processed exactly one time, no duplicates and no loss. Hard because: (1) distributed systems can fail mid-processing, (2) must store state atomically with offset, (3) must handle retries idempotently. Kafka Streams handles this via transactional writes: write result + offset in single transaction. Cost: additional latency + state store overhead. Enable with `processing.guarantee: exactly_once_v2` (not the deprecated v1).

How do I handle late-arriving data and out-of-order events?

Windowing with grace period: `window.until(Duration.ofMinutes(5))` delays window closure to catch late events. Out-of-order: use event timestamps (not processing time) with `TimestampExtractor`. For critical accuracy: wider windows (hourly vs per-minute) and state stores for deduplication. Kafka Streams 3.5+ supports timestamp recovery for replayed data.

Kafka Streams vs Apache Flink, when do I pick which?

Kafka Streams: library (embedded in your app), stateful, runs next to your data, simpler for Kafka-native pipelines, Java/Scala only. Flink: framework (dedicated cluster), lower operational overhead, supports SQL, polyglot (Java/Python/SQL), better complex CEP patterns. Kafka Streams wins if you control the app; Flink if you need a shared cluster serving multiple use cases.

How do I evolve schemas without breaking consumers?

Use Schema Registry with Avro/Protobuf and compatibility checking. Enable `BACKWARD` (new schema reads old data), `FORWARD` (old schema reads new data), or `FULL` (both). Add optional fields with defaults. Never rename/remove fields without a deprecation window. Test schema changes locally with `avsc` files before pushing to production.

When should I NOT use Kafka?

Don't use Kafka for: (1) request-response patterns (use HTTP/gRPC), (2) low-latency <10ms requirements (network overhead is 5-10ms), (3) small data (Kafka's overhead + cluster cost doesn't pay off under 10k events/sec), (4) transactional consistency across services (use distributed transactions or sagas instead), (5) single-machine deployments (use SQLite + polling).

How do I monitor consumer lag and detect problems?

Consumer lag = max offset - committed offset. Monitor via Confluent Control Center, Prometheus (JMX metrics), or Kafka Admin API. Alerting: lag > 1 hour = investigate. Check: (1) if consumers are running, (2) if processing is stuck (check app logs), (3) if topic is receiving data. Scale consumers = partition count; more partitions = more parallelism. Use dedicated lag monitoring tool (Burrow, Kafka Exporter) for multi-cluster visibility.

ไม่แน่ใจว่าทักษะนี้เหมาะสำหรับคุณหรือไม่

ทำการ Career Match — เราจะแนะนำเส้นทางที่เหมาะสม

ค้นหาทักษะที่เหมาะสมที่สุดสำหรับฉัน →

ค้นหาเส้นทางอาชีพในอุดมคติของคุณ

การจับคู่ตามทักษะข้ามอาชีพ 2,536 ตำแหน่ง ฟรี ประมาณ 2 นาที

ทำการ Career Match — ฟรี →

ทักษะทั้งหมด