Data Processing
The layer between ingesting data and analyzing it in a warehouse like BigQuery. Three services do the processing and preparation, and the exam makes you tell them apart:
- Dataflow - serverless, code-based stream + batch ETL on Apache Beam.
- Dataproc - managed Spark/Hadoop clusters for lift-and-shift big-data jobs.
- Dataprep - no-code, visual data wrangling (a Trifacta partner service).
Learn what each is FOR and its one-line trigger; the decision at the bottom is the payoff. (Pub/Sub usually sits in front of these as the ingest buffer.)
Dataflow
Managed stream and batch data processing built on Apache Beam. It transforms, enriches, aggregates, and writes data - the classic pipeline is Pub/Sub → Dataflow → BigQuery. Pick when you see "ETL", "transform/process a stream", or "windowed aggregation".
The classic pipeline is Pub/Sub → Dataflow → BigQuery: Pub/Sub ingests and buffers, Dataflow transforms, BigQuery stores/analyzes. Don't pick Dataflow to just "send messages between services" (that's Pub/Sub) or Pub/Sub to "transform data" (that's Dataflow).
That Pub/Sub → Dataflow → BigQuery line is one slice of a wider shape: Dataflow ingests from many sources as streams or batches, transforms once, and fans the result out to several analytics sinks.
Dataproc
Fast, fully managed service for running Apache Spark and Apache Hadoop clusters. Bring existing Spark, Hadoop, Pig, or Hive jobs and run them unchanged - no new tools or APIs, no redevelopment. Google runs the cluster; you focus on the data. Pick when you see existing Spark/Hadoop/Pig/Hive workloads to migrate.
Why it wins over self-managed Spark/Hadoop:
- Low cost - per-second billing, and preemptible workers cut cost further.
- Fast - create, scale, and shut down clusters in 90 seconds or less on average, versus 5-30 minutes to stand up Spark/Hadoop on-prem or on other IaaS.
- Integrated - built-in hooks to BigQuery, Cloud Storage, Bigtable, Cloud Logging, and Cloud Monitoring, so you get a data platform, not just a cluster.
- Managed - spin clusters up on demand, turn them off when idle, and stop paying.
- Cluster create / scale / shut down: ~90 seconds or less on average.
- Self-managed Spark/Hadoop stand-up (on-prem or other IaaS): 5-30 minutes.
- Billing is per second; preemptible workers reduce cost further.
Cluster anatomy
A cluster is a master plus worker nodes - and each node is a real Compute Engine VM you can see in the console.
- Standard mode: 1 master + N workers (default 2 workers).
- High-availability mode: 3 masters instead of 1.
- VM names:
<cluster>-m(master),<cluster>-w-0,<cluster>-w-1, ... (workers, zero-indexed). - Example: a 4-vCPU master + two 4-vCPU workers = 12 vCPUs total.
Even though Dataproc is a managed service, the master and worker nodes show up as ordinary VMs under Compute Engine. Resizing the worker count (e.g. 2 → 3) creates a new worker VM and updates the master so submitted jobs use it. Deleting the cluster deletes all its data and cannot be undone - the backing VMs are stopped and removed.
Typical workflow (console)
- Big data → Dataproc → Create cluster. Name it, pick region/zone and mode (single master, or HA with 3 masters), set master and worker machine types and worker count.
- Optionally under Advanced: make workers preemptible, set the network / firewall tags, force internal IP only, choose a staging bucket, image, and encryption.
- Create - watch the master and worker VMs appear in Compute Engine; the cluster then initializes (software install) before it's ready.
- Resize anytime: edit the worker count (2 → 3) and save.
- Submit a job: pick the cluster, choose job type (e.g. Spark), set the main class, arguments, and jar; submit and watch output.
- Delete the cluster when done so you only pay while it runs.
Dataprep
Serverless, no-code service for visually exploring, cleaning, and preparing structured and unstructured data. The UI suggests and predicts your next transformation with each input, and auto-detects schema, data types, joins, and anomalies, so you skip manual profiling. An integrated partner service operated by Trifacta (based on Trifacta Wrangler). Pick when you see "clean/wrangle data without writing code", "visual/interactive data prep", or "prepare data for analysis/ML".
Dataprep sits at the front of the analytics pipeline: it wrangles raw data from BigQuery, Cloud Storage, or a file upload, then hands off to Dataflow; the refined data returns to BigQuery or Cloud Storage for analysis and ML.
Both prepare data, but the trigger is who writes the logic. Dataprep is the point-and-click, UI-driven wrangler (built by Trifacta) - it can generate the underlying Dataflow job for you. Dataflow is the programmatic Apache Beam pipeline you author in code. See "no code / visual / interactive" → Dataprep; see "Apache Beam / custom transforms / streaming ETL at scale" → Dataflow.
Which processing service?
All three process or prepare data, so the exam makes you disambiguate. The fastest split:
The close call is Dataproc vs Dataflow - both are code-based engines. Start from Hadoop/Spark lock-in, then from how much of the cluster you want to manage.
If the workload names a Hadoop or Spark tool/package (Pig, Hive, HBase, existing Spark jobs), pick Dataproc regardless of ops preference. Only when there's no ecosystem dependency does the DevOps-vs-serverless question decide it: manual/DevOps → Dataproc, serverless → Dataflow.