Apart from usual data transformations, I found myself in need to create data automatically. Mostly it revolves around GDPR compliance and creating AEAD keys for new incoming data. dbt can't handle this as it would require an incremental insert and sometimes an update/delete.
Any suggestions what's the best way to solve it? At first I thought git repo (a must) with operational queries and use either Cloud Workflows or Scheduled Queries. Perhaps there are better ways?
Hey @ms4446, do you have any expertise in this area?
In BigQuery, managing data transformations for GDPR compliance and handling AEAD keys demands careful architectural design to ensure both data security and adherence to regulations. Here are some options and additional strategies that might fit your needs:
Scheduled Queries in BigQuery:
Cloud Workflows:
Best Practices and Considerations
Integrating Google Cloud's BigQuery, Cloud Workflows, dbt, Dataflow, Cloud Functions, and Cloud Run with a focus on operational best practices offers a comprehensive and robust framework for GDPR compliance and data security. This strategic approach ensures not only regulatory adherence but also fosters an environment of continuous improvement and adaptation to the evolving landscape of data protection and privacy.
Is this an extract from Gemini?