Author Archives: anuj - Page 10
Cleaning data prior to BigQuery Loading
Although bigquery supports most data types, there are two things that trip up a successful data load null data type, null values Automatic type conversion by bigquery To deal with…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There's a couple of options here. Leave the sources on premises Create Linked External Data Sources…
BigQuery – Dealing with Unsafe DataTypes
Encountering unknown data types during a bigquery import (bq load) results in a common error. To get around these errors, try these tips: Return Null for unknown Data Types The…
More on VPC Service Controls…
Also read Service Accounts in GCP and VPC Service Controls - 2 minute GCP Security Solution Protects against theft of OAuth Credentials or Service Account credentials. Network egress control -…
Video Streaming – Cloud storage streaming service versus Transfer service
Also read External Backends for cloud CDN and hosting a website on GCP Cloud Storage using Cloud CDN Cloud Storage versus Cloud Storage Streaming Service To Transfer large files to…
BigQuery Billing Exports
Daily usage and cost estimates - can be exported - on a daily basis , to BigQuery. Billing in GCP Billing costs can be exported to BigQuery for better analysis.…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview - Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL) , and…
Querying External data sources in BigQuery
Also read BigQuery roles and BigQuery File Formats for loading data Querying external datasets is fairly straightforward in Google BigQuery Create a table definition file for the file in the…
Container Resizing in GKE
To add additional nodes to a GKE cluster, one does not have to go down to the managed instance group level. The addition of a new node can be done…
Project Audit Logs for Auditors on GCP
GCP Audit Logs for Future archival / External Access Say you have a requirement to make all your audit logs accessible to a group of auditors. Cloud Logging comes with…