Use Case : Data Flow pipeline reads a CSV file every morning at 8 AM (or any other source file, like a JSON file) to process it before storing the cleansed data in BigQuery. What if the file changes during the work day? What is the best way to allow Data Flow to pick up the changes?
Solution: Use Cloud Composer. Create a custom sensor to detect if the file has changed. If so, re-run the dataflow pipeline.
