ploosh.

Documentation

BigQuery

This connector is used to query Google BigQuery using SQL.

Connection configuration

NameMandatoryDefaultDescription
credentialstypenoserviceaccountAuthentication type: serviceaccount or currentuser
credentialsnoGoogle keyfile encoded in base64 (for serviceaccount mode)
projectidnoGoogle Cloud project ID (for current_user mode)
When credentialstype is serviceaccount, the credentials parameter must contain a base64-encoded Google service account keyfile.

When credentialstype is currentuser, the connector uses pandasgbq with the default credentials from the environment (e.g. gcloud CLI or the GOOGLEAPPLICATION_CREDENTIALS environment variable).

⚠️ It is highly recommended to use a custom parameter to pass the credentials value.

Example

connections:
  bigquery_example:
    type: bigquery
    credentialstype: serviceaccount
    credentials: $var.gbq_credentials

Test case configuration

NameMandatoryDefaultDescription
queryyesSQL query to execute

Example

Example BigQuery:
  source:
    type: bigquery
    connection: bigquery_example
    query: |
      SELECT *
      FROM rh.employees
      WHERE hire_date < "2000-01-01"
  expected:
    type: csv
    path: data/employeesbefore2000.csv

Requirements

  • pip install pandas-gbq (included in ploosh full installation)
  • For service_account mode: a valid Google Cloud service account keyfile
```