Cleaning data prior to BigQuery Loading
Although bigquery supports most data types, there are two things that trip up a successful data load null data type, null values Automatic type conversion by bigquery To…
Insights
Blog
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There’s a couple of options here. Leave the sources on premises Create Linked External…
BigQuery – Dealing with Unsafe DataTypes
Encountering unknown data types during a bigquery import (bq load) results in a common error. To get around these errors, try these tips: Return Null for unknown Data…
More on VPC Service Controls…
Also read Service Accounts in GCP and VPC Service Controls – 2 minute GCP Security Solution Protects against theft of OAuth Credentials or Service Account credentials. Network egress…
Video Streaming – Cloud storage streaming service versus Transfer service
Also read External Backends for cloud CDN and hosting a website on GCP Cloud Storage using Cloud CDN Cloud Storage versus Cloud Storage Streaming Service To Transfer large…
BigQuery Billing Exports
Daily usage and cost estimates – can be exported – on a daily basis , to BigQuery. Billing in GCP Billing costs can be exported to BigQuery for…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview – Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL)…
Querying External data sources in BigQuery
Also read BigQuery roles and BigQuery File Formats for loading data Querying external datasets is fairly straightforward in Google BigQuery Create a table definition file for the file…
Container Resizing in GKE
To add additional nodes to a GKE cluster, one does not have to go down to the managed instance group level. The addition of a new node can…
Project Audit Logs for Auditors on GCP
GCP Audit Logs for Future archival / External Access Say you have a requirement to make all your audit logs accessible to a group of auditors. Cloud Logging…