Google Cloud · Software · YoctoIT tech page

Dataflow & Pub/Sub

Serverless streaming and messaging: events collected, transformed and delivered in real time — with no clusters to babysit.

FOCUS · DATA IN REAL TIMEFrom the nightly batch to continuous streaming: pipelines that don't sleep
Request the Free Assessment Learn more
YoctoIT material for clients and partners · Google Cloud, BigQuery, Gemini and the other products mentioned are trademarks of Google LLC.
01 · What it is

Dataflow & Pub/Sub, made clear.

Pub/Sub is the nervous system: global serverless messaging, millions of events per second, no brokers to manage. Dataflow processes them in flight (Apache Beam): transformations, time windows, enrichments — the same code for streaming and batch, autoscaling included. Real time as a service.

Serverless
no brokers, no clusters: you publish and process, the rest is Google's
Beam
one code for streaming and batch: the pipeline written once
Exactly-once
processing without duplicates or gaps: the numbers add up
Dataflow & Pub/Sub
OFFICIAL GOOGLE CLOUD BRANDING · DATAFLOW & PUB/SUB
CONSOLE REALE · DATAFLOW JOB GRAPH · FONTE: GOOGLE CLOUD DOCS
REAL CONSOLE · DATAFLOW JOB GRAPH · SOURCE: GOOGLE CLOUD DOCS
02 · How to use it well

The things that make the difference.

Il flusso

Sources: apps, IoT, databases (CDC)events are born everywhere
Pub/Sub
Dataflow
BigQuery & sinks
transport · processing · destination
Autoscaling & monitoringthe pipeline that breathes with the load
Datastream for CDCthe ERP in streaming
From event to decision, in seconds

Event-driven architecture

Topics and schemas designed well: the decoupling that makes systems evolvable.

CDC from the ERP

Datastream from Oracle/MySQL/Postgres to BigQuery: the warehouse updated to the minute, without batches.

Windows and delays

Watermarks and late data: serious streaming manages time, it doesn't ignore it.

Elastic costs

Autoscaling with limits and the streaming engine: real time at a reasonable rate.

03 · In depth

Streaming: Beam, windowing and exactly-once

Pub/Sub decouples producers from consumers (at-least-once by default, exactly-once and ordering keys when needed, 31 days of retention); Dataflow runs Apache Beam batch+streaming pipelines with horizontal and vertical autoscaling, windowing (fixed/sliding/session) and watermarks for late data, the Streaming Engine separating state and compute. The pair carries IoT ingestion, CDC (with Datastream) and the real-time views on BigQuery.

  • Exactly-once — the strong semantics when duplicates aren't allowed
  • Ordering key — per-key ordering where it counts (per entity, not global)
  • Windowing — windows and watermarks: the event's time, not the arrival's
  • Autoscaling — workers following the flow: the peak absorbed on its own
  • Streaming Engine — the state outside the workers: smoother scaling and updates
  • Template — ready pipelines (Pub/Sub→BigQuery, CDC): streaming without code
04 · Numbers and lifecycle

The numbers that matter.

ms
the Pub/Sub end-to-end latencies
31 gg
the configurable message retention
1
model (Beam) for batch and streaming
0
clusters to manage: both serverless
Reliable streaming is a design: semantics, windows and costs designed by us — real time that doesn't drop pieces.
05 · Use cases

Where it really pays off.

Operational dashboards

Sales, logistics and production seen by the minute.

IoT and telemetry

Millions of sensors collected and aggregated in flight.

Fraud & alerting

The anomaly seen when it happens, not tomorrow morning.

The nightly batch is a habit, not a destiny: managed streaming, designed by us.