Skip to main content

Cloud Storage

Exam guide§2.2
Cloud Storage

Cloud Storage is Google's object storage for unstructured blobs - images, backups, logs, static sites. Objects live in buckets, bucket names are globally unique, and there are no real folders: a "folder" is just a / in the object's name.

BucketsNaming requirementsCannot be nestedObjectsInherit storage class of bucket when createdNo minimum size; unlimited storageAccessgcloud storage command(RESTful) JSON API or XML APIObjectBucket
Objects are placed into buckets and inherit the bucket's storage class on upload. You reach them by the gcloud storage CLI or the JSON / XML APIs - there is no file system, just a bucket path.

Every bucket comes down to three choices: how cheap vs how reachable the data is (storage class), where the bytes physically live (location type), and what happens to objects over time (lifecycle and protection). This page walks them in that order.

Storage classes

Pick a storage class by how often you read the data. Colder = cheaper storage, pricier retrieval, longer minimum billing - durability and first-byte latency stay identical across all four. Walk it as a decision: is the data even unstructured (else use a database), then funnel by read frequency down to the coldest class that still fits.

StartStructured orunstructured data?structuredConsider a structureddatabase serviceunstructuredRead < 1per year?YesConsiderArchive StorageRead < 1 per90 days?YesConsiderColdline StorageRead < 1 per30 days?YesConsiderNearline StorageNoStandardStorage
Cloud Storage is for unstructured data - structured data belongs in a database. For the rest, pick the class by how rarely you read it: colder = cheaper storage, and Standard is the default when reads are frequent.

The matrix below is the same decision seen side-on, with the price and durability detail the flow leaves out:

Use caseMin. storagedurationRetrieval costAvailabilitySLAStandard
Hot data, frequent access, short periods
nonenone99.95% (multi/dual)99.90% (region)
Nearline
Infrequent - backups, archiving (~monthly)
30 days$0.01 / GB99.90% (multi/dual)99.00% (region)
Coldline
Infrequent - read/modify ≤ once a quarter
90 days$0.02 / GB99.90% (multi/dual)99.00% (region)
Archive
Archiving, backup, DR - < once a year
365 days$0.05 / GBno SLA
Durability11 nines - 99.999999999% (identical for every class)
One matrix, hot to cold. Left to right the class gets colder: cheaper at-rest storage but a longer minimum duration, higher retrieval cost, and lower availability. Durability is identical across all four.
GotchaEarly-delete fees apply to cold classes

Delete or overwrite an Archive object after 10 days and you're still billed for the full 365-day minimum. All classes have the SAME 11-nines durability and millisecond first-byte latency - Archive is NOT tape and needs no "restore/thaw" wait.

GotchaDurability is not availability

11-nines durability means you won't lose data - it says nothing about whether you can reach it right now. Availability (the SLA above) is that "can I open the vault today" guarantee, and it drops for colder classes and single regions. Like a bank: your money is safe in the vault (durable), but you can't withdraw it while the branch is closed (available).

Location types

A storage class decides how the data is stored; a location type decides where. Every class is offered in all three, and you fix the choice when you create the bucket.

Multi-regionLarge area, e.g. "United States"OregonIowaS. Carolinageo-redundantDual-regionA specific pair of regionsFinlandNetherlandsgeo-redundantRegionA single place, e.g. "London"Londonsingle location
A bucket lives in one of three location types, chosen at creation. Multi-region and dual-region keep copies in geographically separate places (geo-redundant); a single region does not.

An object can also carry its own storage class that overrides the bucket default - set it at upload or change it later in place, without a new bucket or a new URL.

GotchaLocation type is fixed for life

You can change a bucket's default storage class anytime, but you can never switch its location type - a regional bucket can't become multi/dual-region and vice versa. To "move" it you create a new bucket and copy the objects across.

Managing objects over time

Once an object is written, four features decide its fate - read them as a spectrum from letting it go to holding it forever: lifecycle rules age it out, versioning remembers old copies, Soft Delete catches accidental deletes, and Retention Lock forbids deletion outright.

Lifecycle rules

Lifecycle rules run automatically on a bucket to transition (downgrade class) or delete objects by age, version count, or creation date. A rule can only move objects to a colder class, never warm them back up. Common rules: downgrade objects older than a year to Coldline, delete objects created before a specific date, or keep only the N most recent versions.

GotchaLifecycle actions are not immediate

Object inspection runs in asynchronous batches, and a lifecycle config change takes up to 24 hours to take effect - so Cloud Storage may keep acting on the OLD rules for up to a day after you edit them.

Autoclass

Lifecycle rules are one-way and hand-authored; Autoclass does the class management for you, both directions. Enable it on a bucket and every object - whatever class the upload requested - begins in Standard, then transitions colder when it sits unread past a threshold and back to Standard the instant it is read. Reach for it when access patterns are unknown, unpredictable, or varied across objects.

Object is read → back to StandardStandardNot accessed for30 days → NearlineStays for 30 daysNearlineNot accessed for90 days → ColdlineStays for 60 daysColdlineNot accessed for365 days → ArchiveStays for 275 daysArchiveColdest tier -stays until readStays until accessed
Autoclass moves an object one step colder each time it sits unread past the threshold, and jumps it straight back to Standard the moment it is read - no lifecycle rules to author.
GotchaAutoclass waives the cold-class fees, but adds its own

Because Autoclass owns the transitions, buckets with it enabled have no early-deletion charges, no retrieval charges, and no storage-class transition charges - the exact fees that bite you with manual lifecycle rules. In exchange it charges a small per-object management fee. Don't run lifecycle rules and Autoclass on the same bucket.

Object versioning

Objects are immutable - an uploaded object never changes during its lifetime. To recover objects that are deleted or overwritten, enable Object Versioning on the bucket. Each time the live version is overwritten or deleted, Cloud Storage keeps an archived copy, uniquely identified by a generation number.

Cloud Storage Object VersioningGoogle CloudBucketObject A (g1)Object A (g2)ArchiveObject A (g1)New Object A
Objects are immutable, so overwriting "Object A" stores a new live version (g2) and keeps the previous one (g1) as an archived copy - identified by its generation number - that you can list, restore, or permanently delete.

With versioning on you can list archived versions, restore the live version to an older state, or permanently delete an archived version. You can turn versioning on or off at any time; turning it off leaves existing archived versions in place and just stops accumulating new ones.

GotchaEvery version is billed as a separate object

Archived versions are charged as if they were multiple files, so a heavily-overwritten object quietly multiplies your storage bill. Combine versioning with a lifecycle rule (e.g. "keep only the 3 newest versions") to cap the cost.

Soft delete

Soft Delete is the safety net for accidental or malicious deletion, and every bucket gets it by default. Deleted objects - including versions removed by an overwrite - are retained for a set duration: restorable during that window, permanently gone after.

GotchaPrefer Soft Delete over Object Versioning for deletion protection

To guard against accidental or malicious deletion, Google recommends Soft Delete. Reach for Object Versioning instead only when you genuinely need a browsable history of every object revision.

NumbersSoft Delete retention
SettingValue
Enabled by defaultYes, on every new bucket
Default retention duration7 days
Maximum retention duration90 days
Disable itSet retention duration to 0

Object retention lock

Object Retention Lock sets a retention configuration on individual objects (in buckets where the feature is enabled). The configuration governs how long an object must be retained, with the option to permanently prevent that retention time from being reduced or removed.

GotchaRetention Lock is for compliance, and can be irreversible

Its purpose is meeting regulatory data-retention rules such as FINRA, SEC, and CFTC. Locking a retention period is one-way - you cannot shorten or remove it afterward - so set it deliberately.

Access control

Cloud Storage offers four ways to grant access, from coarse to fine, and they combine on the same bucket.

ProjectBucketObjectCoarse-grainedFine-grainedIAM
Roles inherited project → bucket → object
ACLs
Per-object read / write grants
Signed URL
Time-limited, no Google login
Signed policy doc
Restricts what may be uploaded
Can be used together
Four ways to grant access, coarsest to finest, and they stack on the same bucket. IAM roles (inherited project -> bucket -> object) cover most needs; ACLs, signed URLs, and signed policy documents add progressively finer, more temporary control.

IAM vs ACLs is the choice you make most:

  • Uniform bucket-level access - IAM only, applied at the bucket. Simple, recommended, auditable. All objects inherit bucket permissions.
  • Fine-grained (ACLs) - per-object ACLs on top of IAM. Legacy; only use when individual objects need different access.
GotchaUniform is one-way after 90 days

You can switch a bucket to uniform anytime, but once uniform has been on for 90 days you can NOT revert to fine-grained. Default new buckets to uniform unless you truly need per-object ACLs.

DECISIONGrant public or temporary access?
Anyone on the internet reads all objectsGrant allUsers the Storage Object Viewer role
Time-limited access to ONE object, no IAMSigned URL (expires, works for GET or PUT)
Browser upload form to a bucketSigned policy document
Pick this when: never make a bucket public just to share one file
ScopeACLsPermissionMax: 100 ACL entriesOwnerWriterReaderExamples:collaborator@gmail.comallUsersallAuthenticatedUsers
Each ACL entry pairs a scope (who) with a permission (what). Permissions nest: Owner includes Writer, Writer includes Reader. Up to 100 entries per bucket or object.
Gotcha`allUsers` vs `allAuthenticatedUsers`

allUsers = literally anyone on the internet, no login required. allAuthenticatedUsers = anyone signed in with any Google account (not just yours, not your org). Neither means "people in my organization" - use IAM for that.

A signed URL grants temporary access to a specific object without the caller having a Google identity - the URL itself carries the (expiring) credential. Think of it as a valet key.

GotchaA signed URL is out of your control once issued

It works for GET, PUT, or DELETE - not POST. After you hand it out you can't revoke it, so set a short expiry. A signed policy document tightens it further by restricting what a holder is allowed to upload.

Cross-project access

To let a workload in one project reach a bucket in another project, don't make the bucket public - grant a service account the right Storage role in the bucket's project, then hand its key to the workload. The classic recipe: create the SA in the bucket's project, grant it Storage Object Viewer (read) or Storage Object Admin (read + write) on the bucket, download its JSON key, and on the accessing VM run gcloud auth activate-service-account --key-file=…. The VM now makes every request as that SA.

Project A - runs the VMCompute Engine VMcredentials.jsonactivated as the SAgcloud auth activate-service-account --key-file=credentials.jsonProject B - owns the bucketBucketcross-project-storage@…service account, defined in Project BStorage Object Viewer → read (ls, cp down)Storage Object Admin → read + write (cp up)② acts as SA① key copied to VM
A VM in Project A reaches a bucket in Project B by borrowing a service account defined in Project B: download the SA’s JSON key, activate it on the VM, and the VM then acts as that SA - Storage Object Viewer grants read, Storage Object Admin grants read and write.
GotchaThe SA's role decides read vs write, not the VM's project

Access is bound to the service account's IAM role on the target bucket, so cross-project access works even though the VM lives elsewhere. With only Storage Object Viewer, a cp up into the bucket fails - you must add Storage Object Admin for writes. The downloaded JSON key is a long-lived credential: anyone holding it acts as the SA, so protect it like a password.

Consistency

Cloud Storage is strongly globally consistent: the moment an upload succeeds, the object is readable - with correct metadata - everywhere Google serves, whether it's brand new or an overwrite. You never get a 404 Not Found or stale data on a read-after-write.

NumbersWhat is strongly consistent
OperationGuarantee
Read-after-writeNew/overwritten object is immediately readable
Read-after-metadata-updateUpdated metadata is immediately visible
Read-after-deleteDeleted object immediately returns 404 Not Found
Bucket listingA just-created bucket appears in the list immediately
Object listingA just-uploaded object appears in the list immediately

Other features to know

FactsMore Cloud Storage features
FeatureWhat it does
Customer-supplied encryption keys (CSEK)Supply your own key instead of Google-managed keys (see CMEK page)
Directory synchronizationSync a VM directory with a bucket
Object change notificationsFire events on object changes - configured using Pub/Sub

Bulk import

Getting data into Cloud Storage - pick the tool by source, size, and network.

DECISIONHow do I move the data?
A few files from your machine / a scriptgcloud storage cp (or gsutil cp)
Recurring transfer from S3, Azure, other GCS, or a URL listStorage Transfer Service
Huge on-prem dataset, but you HAVE decent bandwidthStorage Transfer Service (on-prem agent)
Petabytes on-prem, slow/no networkTransfer Appliance (ship a physical box)
Physical media (arrays, HDDs, tapes, USB) handed to a providerOffline Media Import (third-party service)
Load files from GCS into BigQuerybq load
Pick this when: match volume + connectivity to the tool
GotchaUse -m for many files

A large number of small objects uploads slowly one at a time. gsutil -m (or parallel composite uploads for big single files) parallelizes it. This is a common "speed up the upload" answer.

For larger datasets, three services bulk-load into buckets - pick by network vs volume:

  • Storage Transfer Service - managed and online. Moves data from AWS S3, Azure Blob, other GCS buckets, HTTP/HTTPS URL lists, or on-prem (via an agent) into Cloud Storage, with schedules and incremental syncs. Needs bandwidth.
  • Transfer Appliance - a rugged physical box Google ships to you. You fill it, ship it back, and Google loads it into Cloud Storage. Offline, for petabyte-scale (hundreds of TB up to 1 PB).
  • Offline Media Import - a third-party provider uploads your physical media (storage arrays, HDDs, tapes, USB drives) into Cloud Storage. Use it when the data already lives on removable media.
NumbersOnline vs offline rule of thumb

If moving the data over your network would take weeks or more, use Transfer Appliance. Otherwise use Storage Transfer Service. Roughly: 1 TB over 100 Mbps ≈ a full day.

Recap

CommandsLifecycle + class
gcloud storage buckets create gs://my-bucket --location=US --default-storage-class=STANDARD
gcloud storage cp file.txt gs://my-bucket/
gcloud storage rsync ./dir gs://my-bucket/dir # sync only changes
gsutil -m cp -r ./dir gs://my-bucket/ # -m = parallel, for many files
gcloud storage buckets update gs://my-bucket --lifecycle-file=rules.json
CommandsTime-limited signed URL
gcloud storage sign-url gs://my-bucket/object --private-key-file=key.json --duration=10m
CommandsACLs, versioning, and directory sync (gsutil)
# per-object ACL: read one, then make an object publicly readable
gsutil acl get gs://my-bucket/setup.html
gsutil acl set private gs://my-bucket/setup.html
gsutil acl ch -u AllUsers:R gs://my-bucket/setup.html
 
# object versioning: enable, then list every archived generation
gsutil versioning set on gs://my-bucket
gsutil ls -a gs://my-bucket/setup.html # each line is name#generation
 
# directory sync: mirror a local tree into the bucket (recursive)
gsutil rsync -r ./firstlevel gs://my-bucket/firstlevel
CommandsCross-project bucket access
# on the accessing VM: become the service account, then use the other project's bucket
gcloud auth activate-service-account --key-file=credentials.json
gsutil ls gs://other-project-bucket/ # needs Storage Object Viewer
gsutil cp ./file gs://other-project-bucket/ # needs Storage Object Admin