File Formats for BigQuery Loading Data

Also read Bigquery Partitioning Overview - Clustering versus Partitioning

The format and the type of storage (Cloud Storage, BigQuery, BigTable, CloudSQL...) dictate query performance.

Formats suitable for Cloud Storage

If your dataset sits in Cloud Storage, use either Parquet or Avro.

Column Formats

Parquet - Compressed, Columnar format

Avro - columnar, binary format

Row Formats

JSON row format, duplication of row headers. Also allows binary

CSV - Row format

Summary

The format of the data is important to optimize data loading and retrieving. If your dataset sits in Cloud Storage, use either Parquet or Avro, both columnar format, for optimal performance.

 

 

 

 

anuj

I am passionate about getting cloud solutions right the first time. I help customers transition to the public cloud with the appropriate security controls and guardrails. GCP, AWS and Azure Security Specialist. <a href="https://calendly.com/anujvarma/google-cloud-architect-initial-consultation">Initial Consultation</a>

3 responses to “File Formats for BigQuery Loading Data”

Leave a reply

Your email address will not be published. Required fields are marked *