Author Archives: anuj - Page 9
Folders and Billing in GCP
Folders in GCP are not billable entities - Projects and Org level billing is all that can be enforced.
What is BigTable good for – GCP
When you have non relational data - Metadata from GCP services (Compute Engine, Cloud storage ...), this can easily be streamed to either DataStore or BigTable. For large volume of…
Re-Running DataFlow Jobs – Cloud Composer
Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before storing the…
Cleaning data prior to BigQuery Loading
Although bigquery supports most data types, there are two things that trip up a successful data load null data type, null values Automatic type conversion by bigquery To deal with…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There's a couple of options here. Leave the sources on premises Create Linked External Data Sources…
BigQuery – Dealing with Unsafe DataTypes
Encountering unknown data types during a bigquery import (bq load) results in a common error. To get around these errors, try these tips: Return Null for unknown Data Types The…
More on VPC Service Controls…
Also read Service Accounts in GCP and VPC Service Controls - 2 minute GCP Security Solution Protects against theft of OAuth Credentials or Service Account credentials. Network egress control -…
Video Streaming – Cloud storage streaming service versus Transfer service
Also read External Backends for cloud CDN and hosting a website on GCP Cloud Storage using Cloud CDN Cloud Storage versus Cloud Storage Streaming Service To Transfer large files to…
BigQuery Billing Exports
Daily usage and cost estimates - can be exported - on a daily basis , to BigQuery. Billing in GCP Billing costs can be exported to BigQuery for better analysis.…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview - Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL) , and…