 | dbt BigQuery (EDK preview)
|
The dbt-bigquery
utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse. This utility is in a preview stage of readiness. It is based upon our Extension Developer Kit (EDK) which is the new way to build and add plugins to Meltano Hub.
This preview of dbt will replace the native transformer once it comes out of preview. In the meantime, we would appreciate any feedback in the EDK repo and in the dbt Extension repo.EDK Preview
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the dbt-bigquery utility to your project
using
:meltano add -
Configure the dbt-bigquery settings using
:meltano config
meltano add utility dbt-bigquerymeltano config dbt-bigquery set --interactiveNext steps
- If you're running dbt for the first time in a new environment:
# create a starter dbt_project.yml file, a profiles.yml file, and models directory
meltano invoke dbt-bigquery:initialize
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project. dbt also has adapter-specific documentation for BigQuery.
The
dbt-bigquery settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
skip_pre_invoketypeproject_dirprofiles_dirauth_methodprojectdatasetrefresh_tokenclient_idclient_secrettoken_urikeyfile
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Skip Pre-invoke (skip_pre_invoke)
-
Environment variable:
DBT_BIGQUERY_SKIP_PRE_INVOKE -
Default Value:
false
Whether to skip pre-invoke hooks which automatically run dbt clean and deps
dbt Profile Type (type)
-
Environment variable:
DBT_BIGQUERY_TYPE -
Default Value:
bigquery
Project Directory (project_dir)
-
Environment variable:
DBT_BIGQUERY_PROJECT_DIR -
Default Value:
$MELTANO_PROJECT_ROOT/transform
Profiles Directory (profiles_dir)
-
Environment variable:
DBT_BIGQUERY_PROFILES_DIR -
Default Value:
$MELTANO_PROJECT_ROOT/transform/profiles/bigquery
Authentication Method (auth_method)
-
Environment variable:
DBT_BIGQUERY_AUTH_METHOD
The auth method to use. One of: "oauth", "oauth-secrets", or "service-account"
Project (project)
-
Environment variable:
DBT_BIGQUERY_PROJECT
The BigQuery project ID.
Dataset (dataset)
-
Environment variable:
DBT_BIGQUERY_DATASET
The dataset to use.
Refresh Token (refresh_token)
-
Environment variable:
DBT_BIGQUERY_REFRESH_TOKEN
The refresh token, if authenticating via oauth-secrets method.
Client ID (client_id)
-
Environment variable:
DBT_BIGQUERY_CLIENT_ID
The client id to use, if authenticating via oauth-secrets method.
Client Secret (client_secret)
-
Environment variable:
DBT_BIGQUERY_CLIENT_SECRET
The client secret to use, if authenticating via oauth-secrets method.
Token URI (token_uri)
-
Environment variable:
DBT_BIGQUERY_TOKEN_URI
The token redirect URI
Keyfile (keyfile)
-
Environment variable:
DBT_BIGQUERY_KEYFILE
The path to the `keyfile.json`` to use, if authenticating via service-account method.
Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-generalInstall
meltano add utility dbt-bigqueryMaintenance Status
-brightgreen){kind=icon}
Maintainer
Meltano Stats
Keywords
