What is BigTable good for – GCP
When you have non relational data – e.g. Metadata from GCP services (Compute Engine, Cloud storage …), this can easily be streamed to either DataStore or BigTable. For…
Archive
Author: anuj
I am passionate about getting cloud solutions right the first time. I help customers transition to the public cloud with the appropriate security controls and guardrails. GCP, AWS and Azure Security Specialist.
Initial Consultation
Re-Running DataFlow Jobs – Cloud Composer
Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before…
Cleaning data prior to BigQuery Loading
Although bigquery supports most data types, there are two things that trip up a successful data load null data type, null values Automatic type conversion by bigquery To…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There’s a couple of options here. Leave the sources on premises Create Linked External…
BigQuery – Dealing with Unsafe DataTypes
Encountering unknown data types during a bigquery import (bq load) results in a common error. To get around these errors, try these tips: Return Null for unknown Data…
More on VPC Service Controls…
Also read Service Accounts in GCP and VPC Service Controls – 2 minute GCP Security Solution Protects against theft of OAuth Credentials or Service Account credentials. Network egress…
Video Streaming – Cloud storage streaming service versus Transfer service
Also read External Backends for cloud CDN and hosting a website on GCP Cloud Storage using Cloud CDN Cloud Storage versus Cloud Storage Streaming Service To Transfer large…
BigQuery Billing Exports
Daily usage and cost estimates – can be exported – on a daily basis , to BigQuery. Billing in GCP Billing costs can be exported to BigQuery for…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview – Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL)…
Querying External data sources in BigQuery
Also read BigQuery roles and BigQuery File Formats for loading data Querying external datasets is fairly straightforward in Google BigQuery Create a table definition file for the file…