Archives for GCP BigQuery
Building a Data Lake on GCP
Building a data lake on Google Cloud Platform (GCP) involves leveraging a variety of services that are designed to handle storage, processing, and management of large-scale data. Here are some…
Sharing out a bigquery dataset
The owner of the dataset? To share out a dataset, you need to have the () or the role. The user of the dataset? For the user, assuming read-only access:…
GCP IAM troubleshooting and IAM best practices
IAM Access Analyzer Only works for allows, not denys GCP Recommender - also includes IAM recs Export to bigquery some recommendations are around IAM. Unused IAM roles unused IAM roles…
Cost Optimization from billing export, Recommendations export – Sample BigQuery Queries
This content is password protected. To view it please enter your password below: Password:
Arrays in BigQuery
Arrays in BigQuery Create an Array create table as WITH a as ( select ('first','second', 'third') as sequence_num ) select from a Create a dataset CREATE SCHEMA OPTIONS ( default_kms_key_name = 'KMS_KEY_NAME', default_partition_expiration_days = PARTITION_EXPIRATION, default_table_expiration_days = TABLE_EXPIRATION, description = 'test_billing_queries', labels = , location = 'us-central1', max_time_travel_hours = HOURS, storage_billing_model = STORAGE_BILLING_MODEL);…
Re-Running DataFlow Jobs – Cloud Composer
Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before storing the…
MySQL to BigQuery – Moving On Prem SQL Databases into GCP
A common use case is around utilizing on premises SQL Databases for Cloud Analytics. There's a couple of options here. Leave the sources on premises Create Linked External Data Sources…
Dataflow to BigQuery
Also read Bigquery Partitioning Overview - Clustering versus Partitioning and File Formats for Loading Data into BigQuery Overview Dataflow provides a very powerful basis for transforming data (ETL) , and…
BigQuery PreDefined Roles
Also read - File Formats for loading data into BigQuery This post discusses a list of helpful BigQuery roles, in order of decreasing privileges. BigQuery Admin User: Users in this role…
BigQuery – Ingestion Time Partitioning
Also read Partitioning Overview in BigQuery , Bigquery Partitioning Overview - Clustering versus Partitioning and File Formats for Loading Data into BigQuery Ingestion time partitioning Step 1 - Define the…