Streaming Analytics and Real-Time Signal Processing with Apache Kafka
| Imagine you can process and analyze real-time event streams for intelligence to mitigate cyber threats or keep soldiers constantly alerted to risks and precautions they should take based on events. In this episode, Jeffrey Needham (Senior Solutions Engineer, Advanced Technology Group, Confluent) shares use cases on how Apache Kafka® can be used for real-time signal processing to mitigate risk before it arises. He also explains the classic Kafka transactional processing defaults and the distinction between transactional and analytic processing.
Jeffrey is part of the customer solutions and innovations division (CSID), which involves designing event streaming platforms and innovations to improve productivity for organizations by pushing the envelope of Kafka for real-time signal processing.
What is signal intelligence? Jeffrey explains that it’s not always affiliated with the military. Signal processing improves your operational or situational awareness by understanding the petabyte datasets of clickstream data, or the telemetry coming in from sensors, which could be the satellite or sensor arrays along a water pipeline. That is, bringing in event data from external sources to analyze, and then finding the pattern in the series of events to make informed decisions.
Conventional On-Line Analytical Processing (OLAP) or data warehouse platforms evolved out of the transaction processing model. However, when analytics or even AI processing is applied to any data set, these algorithms never look at a single column or row, but look for patterns within millions of rows of transactionally derived data. Transaction-centric solutions are designed to update and delete specific rows and columns in an “ACID” compliant manner, which makes them inefficient and usually unaffordable at scale because this capability is less critical when the analytic goal is to look for a pattern within millions or even billions of these rows.
Kafka was designed as a step forward from classic transaction processing technologies, which can also be configured in a way that’s optimized for signal processing high velocities of noisy or jittery data streams, in order to make sense, in real-time, of a dynamic, non-transactional environment.
With its immutable, write-append commit logs, Kafka functions as a flight data recorder, which remains resilient even when network communications, or COMMs, are poor or nonexistent. Jeffrey shares the disconnected edge project he has been working on—smart soldier, which runs Kafka on a Raspberry Pi and x64-based handhelds. These devices are ergonomically integrated on each squad member to provide real-time visibility into the soldiers’ activities or situations. COMMs permitting, the topic data is then mirrored upstream and aggregated at multiple tiers—mobile command post, battalion, HQ—to provide ever-increasing views of the entire battlefield, or whatever the sensor array is monitoring, including the all important supply chain. Jeffrey also shares a couple of other use cases on how Kafka can be used for signal intelligence, including cybersecurity and protecting national critical infrastructure.
EPISODE LINKS
► Apache Kafka Internals:
► Using Kafka for Analytic Processing:
► Kris Jenkins’ Twitter:
► Join the Confluent Community:
► Learn more on Confluent Developer:
TIMESTAMPS
00:00 - Intro
02:24 - Advanced Technology Group
07:39 - Signal Intelligence
17:00 - Transactional processing vs Analytic processing
23:59 - Adaptive signal processing
29:27 - Separating storage and compute
36:40 - Topic compaction
41:53 - Smart soldiers
50:07 - Use cases
01:04:53 - It’s a wrap!
CONNECT
Subscribe:
Site:
GitHub:
Facebook:
Twitter:
LinkedIn:
Instagram:
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit .
#signalprocessing #streamprocessing #apachekafka #kafka #confluent
5 views
6
1
3 weeks ago 01:32:32 1
Mose - Balance (Continuous Mix)
3 weeks ago 00:00:41 1
Папичу не дают стримить
4 weeks ago 00:00:00 1
LIVE TRIAL | MA. v Karen Read Trial Day 1 Opening Statements
1 month ago 00:06:27 1
What Are KPIs? 5 TOP TIPS For Defining YOURS. BI For Beginners
1 month ago 00:10:03 1
10 Mégas Projets Du Sénégal En 2025 | Africa Evolution
1 month ago 03:06:09 1
Российские акции: что покупать? Разбор в прямом эфире
2 months ago 00:03:01 2
Method Man - Training Day (feat. Cortez) (Official Music Video)
2 months ago 00:02:55 1
На страже права | ТНВ
2 months ago 00:56:17 1
СВОЯ ИГРА ТЕМА DOTA 2 с АНАЛИТИКОЙ ОШИБОК, Lowskill, Gudvin, Finargot, ShergaratVladimir
2 months ago 00:14:18 1
Who Blew Up The Nord Stream Pipeline? Origins & Its Importance |
2 months ago 01:05:36 1
🔴 СВОЯ ИГРА СОСТАВ 2 по Доте 2 с АНАЛИТИКОЙ ОШИБОК, INKOGNITO, STARIY BOG, TIMSMERT, GUDVIN
2 months ago 00:59:12 1
The Event That Will Change the World with Edward Riordan and Daz Smith (Episode 227)
2 months ago 00:45:16 1
Making Kafka Queryable with Apache Pinot • Tim Berglund • GOTO 2023
2 months ago 00:50:51 1
Power BI on Databricks Best Practices
2 months ago 01:18:08 1
Summary of the Ukraine War so far; Analysis on current geopolitical situation/realities - DPA x TPC
2 months ago 00:25:37 1
САМАЯ ГЛУБОКАЯ АНАЛИТИКА НА ОЗОНА 671 | OZON 671 GAMES | ЛЕГЕНДА ТЕРПЕНИЯ С ФИЛОСОФИЕЙ СОКРАТА
2 months ago 00:09:39 5
AOC district neighborhood labeled ’Third World’ as migrants clog streets and prostitutes overrun ...
2 months ago 00:07:39 2
ПИКЕМ НА МАЖОР ОТ АНАЛИТИКА, СТАЖ 7 ЛЕТ.
2 months ago 01:15:58 1
Did feminism FAIL men?
3 months ago 00:04:36 5
МАРМОК ОБНОВИЛ ПК! НОВЫЙ ПК МАРМОКА! КАКОЙ ПК У МАРМОКА? НА ЧЕМ МОНТИРУЕТ, ИГРАЕТ, СТРИМИТ МАРИН?
3 months ago 04:28:24 1
Starfield Was a Mistake | An Analytical Review
3 months ago 01:01:46 1
Webinar - Managing Profitable Growth with a Value Stream Income Statement
3 months ago 00:00:00 1
Watch Live: CLOSING ARGUMENTS in the Fani Willis v. Donald Trump Saga!
3 months ago 00:00:00 1
WATCH LIVE: Terrence Bradley FORCED to TESTIFY Against Fani Willis and Nathan Wade