Cleaning data prior to BigQuery Loading

Although bigquery supports most data types, there are two things that trip up a successful data load

  1. null data type, null values
  2. Automatic type conversion by bigquery

To deal with these two issues, read this post

To architect it a little differently, instead of loading directly into bigquery, use dataprep as a first step. DataPrep to BigQuery is generally a cleaner architecture, allowing for data pre-treatment.

anuj

I am passionate about getting cloud solutions right the first time. I help customers transition to the public cloud with the appropriate security controls and guardrails. GCP, AWS and Azure Security Specialist. <a href="https://calendly.com/anujvarma/google-cloud-architect-initial-consultation">Initial Consultation</a>

Leave a reply

Your email address will not be published. Required fields are marked *