ploosh.

Documentation

Parquet

This connector is used to read local Parquet files.

Connection configuration

No connection is required by this connector.

Test case configuration

NameMandatoryDefaultDescription
pathyesPath to the Parquet file
columnsnoList of columns to load (subset)
enginenoautoParquet engine: auto, pyarrow, or fastparquet
filtersnoList of row group filters to apply
filters.columnyesColumn name to filter on
filters.operatoryesComparison operator: ==, =, >, >=, <, <=, !=
filters.valueyesValue to filter by

Example

Example Parquet:
  source:
    type: parquet
    path: ./data/employees.parquet
  expected:
    type: csv
    path: ./data/expected_employees.csv

Example with column selection

Example Parquet with columns:
  source:
    type: parquet
    path: ./data/employees.parquet
    columns:
      - id
      - name
      - department
  expected:
    type: empty

Example with filters

Example Parquet with filters:
  source:
    type: parquet
    path: ./data/employees.parquet
    filters:
      - column: department_id
        operator: "=="
        value: 5
  expected:
    type: csv
    path: ./data/expected_dept5.csv