Author Archives: anuj - Page 9
GCS Lifecycle Rules – Lifecycle Conditions
All versions that are not LIVE (and 30 days old) to be deleted. All other versions to be moved to COLDLINE (after 365 day) {"lifecycle": { "rule": } }]}} Run…
Folders and Billing in GCP
Folders in GCP are not billable entities - Projects and Org level billing is all that can be enforced.
What is BigTable good for – GCP
When you have non relational data - Metadata from GCP services (Compute Engine, Cloud storage ...), this can easily be streamed to either DataStore or BigTable. For large volume of…
Re-Running DataFlow Jobs – Cloud Composer
Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before storing the…
Cleaning data prior to BigQuery Loading
Although bigquery supports most data types, there are two things that trip up a successful data load null data type, null values Automatic type conversion by bigquery To deal with…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There's a couple of options here. Leave the sources on premises Create Linked External Data Sources…
BigQuery – Dealing with Unsafe DataTypes
Encountering unknown data types during a bigquery import (bq load) results in a common error. To get around these errors, try these tips: Return Null for unknown Data Types The…
More on VPC Service Controls…
Also read Service Accounts in GCP and VPC Service Controls - 2 minute GCP Security Solution Protects against theft of OAuth Credentials or Service Account credentials. Network egress control -…
Video Streaming – Cloud storage streaming service versus Transfer service
Also read External Backends for cloud CDN and hosting a website on GCP Cloud Storage using Cloud CDN Cloud Storage versus Cloud Storage Streaming Service To Transfer large files to…
BigQuery Billing Exports
Daily usage and cost estimates - can be exported - on a daily basis , to BigQuery. Billing in GCP Billing costs can be exported to BigQuery for better analysis.…