BigQuery Optimizations
Secure Google Cloud Billing Analytics with BigQuery and Looker Studio Cloud billing data deserves the same care as other sensitive enterprise data. It can reveal infrastructure usage, negotiated…
Archive
Category: GCP BigQuery
Cost Management, Budgets, Alerts, Exports to Bigquery
GCP Governance, Billing and VMware Migration Guide A Practical Guide to GCP Governance, Billing, VMware Migration, and Disaster Recovery A successful Google Cloud deployment requires more than moving…
Building a Data Lake on GCP
Building a data lake on Google Cloud Platform (GCP) involves leveraging a variety of services that are designed to handle storage, processing, and management of large-scale data. Here…
Sharing out a bigquery dataset
The owner of the dataset? To share out a dataset, you need to have the (roles/bigquery.dataOwner) or the roles/bigquery.admin role. The user of the dataset? For the user, assuming…
GCP IAM troubleshooting and IAM best practices
IAM Access Analyzer Only works for allows, not denys GCP Recommender – also includes IAM recs Export to bigquery some recommendations are around IAM. Unused IAM roles unused…
Cost Optimization from billing export, Recommendations export – Sample BigQuery Queries
There is no excerpt because this is a protected post.
Arrays in BigQuery
Arrays in BigQuery Create an Array create table test_dataset_av.test_dataset_av as WITH a as ( select (‘first’,’second’, ‘third’) as sequence_num ) select a.sequence_num from a Create a dataset CREATE SCHEMA my-sandbox.test_dataset_av OPTIONS ( default_kms_key_name = ‘KMS_KEY_NAME’, default_partition_expiration_days = PARTITION_EXPIRATION, default_table_expiration_days = TABLE_EXPIRATION, description = ‘test_billing_queries’,…
Re-Running DataFlow Jobs – Cloud Composer
Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There’s a couple of options here. Leave the sources on premises Create Linked External…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview – Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL)…