Airbyte (dagster-airbyte)¶

Ops¶

dagster_airbyte.airbyte_sync_op = <dagster._core.definitions.op_definition.OpDefinition object>[source]¶

Config Schema:

connection_id (String):

The Airbyte Connection ID that this op will sync. You can retrieve this value from the “Connections” tab of a given connector in the Airbyte UI.

poll_interval (Float, optional):

The time (in seconds) that will be waited between successive polls.

Default Value: 10

poll_timeout (Union[Float, None], optional):

The maximum time that will waited before this operation is timed out. By default, this will never time out.

Default Value: None

yield_materializations (Bool, optional):

If True, materializations corresponding to the results of the Airbyte sync will be yielded when the op executes.

Default Value: True

asset_key_prefix (List[String], optional):

If provided and yield_materializations is True, these components will be used to prefix the generated asset keys.

Default Value: [‘airbyte’]

Executes a Airbyte job sync for a given connection_id, and polls until that sync completes, raising an error if it is unsuccessful. It outputs a AirbyteOutput which contains the job details for a given connection_id.

It requires the use of the airbyte_resource, which allows it to communicate with the Airbyte API.

Examples:

from dagster import job
from dagster_airbyte import airbyte_resource, airbyte_sync_op

my_airbyte_resource = airbyte_resource.configured(
    {
        "host": {"env": "AIRBYTE_HOST"},
        "port": {"env": "AIRBYTE_PORT"},
    }
)

sync_foobar = airbyte_sync_op.configured({"connection_id": "foobar"}, name="sync_foobar")

@job(resource_defs={"airbyte": my_airbyte_resource})
def my_simple_airbyte_job():
    sync_foobar()

@job(resource_defs={"airbyte": my_airbyte_resource})
def my_composed_airbyte_job():
    final_foobar_state = sync_foobar(start_after=some_op())
    other_op(final_foobar_state)

Resources¶

dagster_airbyte.airbyte_resource ResourceDefinition[source]¶

Config Schema:

host (dagster.StringSource):

The Airbyte Server Address.

port (dagster.StringSource):

Port for the Airbyte Server.

username (dagster.StringSource, optional):

Username if using basic auth.

password (dagster.StringSource, optional):

Password if using basic auth.

use_https (Bool, optional):

Use https to connect in Airbyte Server.

Default Value: False

request_max_retries (Int, optional):

The maximum number of times requests to the Airbyte API should be retried before failing.

Default Value: 3

request_retry_delay (Float, optional):

Time (in seconds) to wait between each request retry.

Default Value: 0.25

request_timeout (Int, optional):

Time (in seconds) after which the requests to Airbyte are declared timed out.

Default Value: 15

request_additional_params (permissive dict, optional):

Any additional kwargs to pass to the requests library when making requests to Airbyte.

Default Value:

{}

forward_logs (Bool, optional):

Whether to forward Airbyte logs to the compute log, can be expensive for long-running syncs.

Default Value: True

This resource allows users to programatically interface with the Airbyte REST API to launch syncs and monitor their progress. This currently implements only a subset of the functionality exposed by the API.

For a complete set of documentation on the Airbyte REST API, including expected response JSON schema, see the Airbyte API Docs.

To configure this resource, we recommend using the configured method.

Examples:

from dagster import job
from dagster_airbyte import airbyte_resource

my_airbyte_resource = airbyte_resource.configured(
    {
        "host": {"env": "AIRBYTE_HOST"},
        "port": {"env": "AIRBYTE_PORT"},
        # If using basic auth
        "username": {"env": "AIRBYTE_USERNAME"},
        "password": {"env": "AIRBYTE_PASSWORD"},
    }
)

@job(resource_defs={"airbyte":my_airbyte_resource})
def my_airbyte_job():
    ...

class dagster_airbyte.AirbyteResource(host, port, use_https, request_max_retries=3, request_retry_delay=0.25, request_timeout=15, request_additional_params=None, log=<Logger dagster.builtin (DEBUG)>, forward_logs=True, username=None, password=None)[source]¶

This class exposes methods on top of the Airbyte REST API.

Assets¶

dagster_airbyte.load_assets_from_airbyte_instance(airbyte, workspace_id=None, key_prefix=None, create_assets_for_normalization_tables=True, connection_to_group_fn=<function _clean_name>, io_manager_key=None, connection_to_io_manager_key_fn=None, connection_filter=None, connection_to_asset_key_fn=None)[source]¶

Loads Airbyte connection assets from a configured AirbyteResource instance. This fetches information about defined connections at initialization time, and will error on workspace load if the Airbyte instance is not reachable.

Parameters:

• airbyte (ResourceDefinition) – An AirbyteResource configured with the appropriate connection details.

• workspace_id (Optional[str]) – The ID of the Airbyte workspace to load connections from. Only required if multiple workspaces exist in your instance.

• key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.

• create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.

• connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.

• io_manager_key (Optional[str]) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.

• connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an IO manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.

• connection_filter (Optional[Callable[[AirbyteConnectionMetadata], bool]]) – Optional function which takes in connection metadata and returns False if the connection should be excluded from the output assets.

• connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.

Examples:

Loading all Airbyte connections as assets:

from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance

airbyte_instance = airbyte_resource.configured(
    {
        "host": "localhost",
        "port": "8000",
    }
)
airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)

Filtering the set of loaded connections:

from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance

airbyte_instance = airbyte_resource.configured(
    {
        "host": "localhost",
        "port": "8000",
    }
)
airbyte_assets = load_assets_from_airbyte_instance(
    airbyte_instance,
    connection_filter=lambda meta: "snowflake" in meta.name,
)

dagster_airbyte.load_assets_from_airbyte_project(project_dir, workspace_id=None, key_prefix=None, create_assets_for_normalization_tables=True, connection_to_group_fn=<function _clean_name>, io_manager_key=None, connection_to_io_manager_key_fn=None, connection_filter=None, connection_directories=None, connection_to_asset_key_fn=None)[source]¶

Loads an Airbyte project into a set of Dagster assets.

Point to the root folder of an Airbyte project synced using the Octavia CLI. For more information, see https://github.com/airbytehq/airbyte/tree/master/octavia-cli#octavia-import-all.

Parameters:

• project_dir (str) – The path to the root of your Airbyte project, containing sources, destinations, and connections folders.

• workspace_id (Optional[str]) – The ID of the Airbyte workspace to load connections from. Only required if multiple workspace state YAMLfiles exist in the project.

• key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.

• create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.

• connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.

• io_manager_key (Optional[str]) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.

• connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an IO manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.

• connection_filter (Optional[Callable[[AirbyteConnectionMetadata], bool]]) – Optional function which takes in connection metadata and returns False if the connection should be excluded from the output assets.

• connection_directories (Optional[List[str]]) – Optional list of connection directories to load assets from. If omitted, all connections in the Airbyte project are loaded. May be faster than connection_filter if the project has many connections or if the connection yaml files are large.

• connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.

Examples:

Loading all Airbyte connections as assets:

from dagster_airbyte import load_assets_from_airbyte_project

airbyte_assets = load_assets_from_airbyte_project(
    project_dir="path/to/airbyte/project",
)

Filtering the set of loaded connections:

from dagster_airbyte import load_assets_from_airbyte_project

airbyte_assets = load_assets_from_airbyte_project(
    project_dir="path/to/airbyte/project",
    connection_filter=lambda meta: "snowflake" in meta.name,
)

dagster_airbyte.build_airbyte_assets(connection_id, destination_tables, asset_key_prefix=None, normalization_tables=None, upstream_assets=None, schema_by_table_name=None)[source]¶

Builds a set of assets representing the tables created by an Airbyte sync operation.

Parameters:

• connection_id (str) – The Airbyte Connection ID that this op will sync. You can retrieve this value from the “Connections” tab of a given connector in the Airbyte UI.

• destination_tables (List[str]) – The names of the tables that you want to be represented in the Dagster asset graph for this sync. This will generally map to the name of the stream in Airbyte, unless a stream prefix has been specified in Airbyte.

• normalization_tables (Optional[Mapping[str, List[str]]]) – If you are using Airbyte’s normalization feature, you may specify a mapping of destination table to a list of derived tables that will be created by the normalization process.

• asset_key_prefix (Optional[List[str]]) – A prefix for the asset keys inside this asset. If left blank, assets will have a key of AssetKey([table_name]).

• upstream_assets (Optional[Set[AssetKey]]) – A list of assets to add as sources.

Managed Config¶

class dagster_airbyte.AirbyteManagedElementReconciler(*args, **kwargs)[source]¶

Reconciles Python-specified Airbyte connections with an Airbyte instance.

Passing the module containing an AirbyteManagedElementReconciler to the dagster-airbyte CLI will allow you to check the state of your Python-code-specified Airbyte connections against an Airbyte instance, and reconcile them if necessary.

This functionality is experimental and subject to change.

dagster_airbyte.load_assets_from_connections(airbyte, connections, key_prefix=None, create_assets_for_normalization_tables=True, connection_to_group_fn=<function _clean_name>, io_manager_key=None, connection_to_io_manager_key_fn=None, connection_to_asset_key_fn=None)[source]¶

Loads Airbyte connection assets from a configured AirbyteResource instance, checking against a list of AirbyteConnection objects. This method will raise an error on repo load if the passed AirbyteConnection objects are not in sync with the Airbyte instance.

Parameters:

• airbyte (ResourceDefinition) – An AirbyteResource configured with the appropriate connection details.

• connections (Iterable[AirbyteConnection]) – A list of AirbyteConnection objects to build assets for.

• key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.

• create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.

• connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.

• io_manager_key (Optional[str]) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.

• connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an IO manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.

• connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.

class dagster_airbyte.AirbyteSource(name, source_type, source_configuration)[source]¶

Represents a user-defined Airbyte source.

__init__(name, source_type, source_configuration)[source]¶

Parameters:

• name (str) – The display name of the source.

• source_type (str) – The type of the source, from Airbyte’s list of sources https://airbytehq.github.io/category/sources/.

• source_configuration (Mapping[str, Any]) – The configuration for the source, as defined by Airbyte’s API.

class dagster_airbyte.AirbyteDestination(name, destination_type, destination_configuration)[source]¶

Represents a user-defined Airbyte destination.

__init__(name, destination_type, destination_configuration)[source]¶

Parameters:

• name (str) – The display name of the destination.

• destination_type (str) – The type of the destination, from Airbyte’s list of destinations https://airbytehq.github.io/category/destinations/.

• destination_configuration (Mapping[str, Any]) – The configuration for the destination, as defined by Airbyte’s API.

class dagster_airbyte.AirbyteConnection(name, source, destination, stream_config, normalize_data=None, destination_namespace=AirbyteDestinationNamespace.SAME_AS_SOURCE)[source]¶

A user-defined Airbyte connection, pairing an Airbyte source and destination and configuring which streams to sync.

class dagster_airbyte.AirbyteSyncMode(json_repr)[source]¶

Represents the sync mode for a given Airbyte stream, which governs how Airbyte reads from a source and writes to a destination.

For more information, see https://docs.airbyte.com/understanding-airbyte/connections/.

classmethod full_refresh_append()[source]¶

Syncs the entire data stream from the source, appending to the destination.

https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append/

classmethod full_refresh_overwrite()[source]¶

Syncs the entire data stream from the source, replaces data in the destination by overwriting it.

https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite

classmethod incremental_append(cursor_field=None)[source]¶

Syncs only new records from the source, appending to the destination. May optionally specify the cursor field used to determine which records are new.

https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/

classmethod incremental_append_dedup(cursor_field=None, primary_key=None)[source]¶

Syncs new records from the source, appending to an append-only history table in the destination. Also generates a deduplicated view mirroring the source table. May optionally specify the cursor field used to determine which records are new, and the primary key used to determine which records are duplicates.

https://docs.airbyte.com/understanding-airbyte/connections/incremental-append-dedup/

Managed Config Generated Sources¶

class dagster_airbyte.managed.generated.sources.StravaSource(name, client_id, client_secret, refresh_token, athlete_id, start_date, auth_type=None)[source]¶

__init__(name, client_id, client_secret, refresh_token, athlete_id, start_date, auth_type=None)[source]¶

Airbyte Source for Strava

Documentation can be found at https://docs.airbyte.com/integrations/sources/strava

class dagster_airbyte.managed.generated.sources.AppsflyerSource(name, app_id, api_token, start_date, timezone=None)[source]¶

__init__(name, app_id, api_token, start_date, timezone=None)[source]¶

Airbyte Source for Appsflyer

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.GoogleWorkspaceAdminReportsSource(name, credentials_json, email, lookback=None)[source]¶

__init__(name, credentials_json, email, lookback=None)[source]¶

Airbyte Source for Google Workspace Admin Reports

Documentation can be found at https://docs.airbyte.com/integrations/sources/google-workspace-admin-reports

class dagster_airbyte.managed.generated.sources.CartSource(name, credentials, start_date)[source]¶

__init__(name, credentials, start_date)[source]¶

Airbyte Source for Cart

Documentation can be found at https://docs.airbyte.com/integrations/sources/cart

class dagster_airbyte.managed.generated.sources.LinkedinAdsSource(name, credentials, start_date, account_ids=None)[source]¶

__init__(name, credentials, start_date, account_ids=None)[source]¶

Airbyte Source for Linkedin Ads

Documentation can be found at https://docs.airbyte.com/integrations/sources/linkedin-ads

class dagster_airbyte.managed.generated.sources.MongodbSource(name, host, port, database, user, password, auth_source, replica_set=None, ssl=None)[source]¶

__init__(name, host, port, database, user, password, auth_source, replica_set=None, ssl=None)[source]¶

Airbyte Source for Mongodb

Documentation can be found at https://docs.airbyte.com/integrations/sources/mongodb

class dagster_airbyte.managed.generated.sources.TimelySource(name, account_id, start_date, bearer_token)[source]¶

__init__(name, account_id, start_date, bearer_token)[source]¶

Airbyte Source for Timely

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.StockTickerApiTutorialSource(name, stock_ticker, api_key)[source]¶

__init__(name, stock_ticker, api_key)[source]¶

Airbyte Source for Stock Ticker Api Tutorial

Documentation can be found at https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date

class dagster_airbyte.managed.generated.sources.WrikeSource(name, access_token, wrike_instance, start_date=None)[source]¶

__init__(name, access_token, wrike_instance, start_date=None)[source]¶

Airbyte Source for Wrike

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.CommercetoolsSource(name, region, host, start_date, project_key, client_id, client_secret)[source]¶

__init__(name, region, host, start_date, project_key, client_id, client_secret)[source]¶

Airbyte Source for Commercetools

Documentation can be found at https://docs.airbyte.com/integrations/sources/commercetools

class dagster_airbyte.managed.generated.sources.GutendexSource(name, author_year_start=None, author_year_end=None, copyright=None, languages=None, search=None, sort=None, topic=None)[source]¶

__init__(name, author_year_start=None, author_year_end=None, copyright=None, languages=None, search=None, sort=None, topic=None)[source]¶

Airbyte Source for Gutendex

Documentation can be found at https://docs.airbyte.com/integrations/sources/gutendex

class dagster_airbyte.managed.generated.sources.IterableSource(name, api_key, start_date)[source]¶

__init__(name, api_key, start_date)[source]¶

Airbyte Source for Iterable

Documentation can be found at https://docs.airbyte.com/integrations/sources/iterable

class dagster_airbyte.managed.generated.sources.QuickbooksSingerSource(name, client_id, client_secret, refresh_token, realm_id, user_agent, start_date, sandbox)[source]¶

__init__(name, client_id, client_secret, refresh_token, realm_id, user_agent, start_date, sandbox)[source]¶

Airbyte Source for Quickbooks Singer

Documentation can be found at https://docs.airbyte.com/integrations/sources/quickbooks

class dagster_airbyte.managed.generated.sources.BigcommerceSource(name, start_date, store_hash, access_token)[source]¶

__init__(name, start_date, store_hash, access_token)[source]¶

Airbyte Source for Bigcommerce

Documentation can be found at https://docs.airbyte.com/integrations/sources/bigcommerce

class dagster_airbyte.managed.generated.sources.ShopifySource(name, shop, credentials, start_date)[source]¶

__init__(name, shop, credentials, start_date)[source]¶

Airbyte Source for Shopify

Documentation can be found at https://docs.airbyte.com/integrations/sources/shopify

class dagster_airbyte.managed.generated.sources.AppstoreSingerSource(name, key_id, private_key, issuer_id, vendor, start_date)[source]¶

__init__(name, key_id, private_key, issuer_id, vendor, start_date)[source]¶

Airbyte Source for Appstore Singer

Documentation can be found at https://docs.airbyte.com/integrations/sources/appstore

class dagster_airbyte.managed.generated.sources.GreenhouseSource(name, api_key)[source]¶

__init__(name, api_key)[source]¶

Airbyte Source for Greenhouse

Documentation can be found at https://docs.airbyte.com/integrations/sources/greenhouse

class dagster_airbyte.managed.generated.sources.ZoomSingerSource(name, jwt)[source]¶

__init__(name, jwt)[source]¶

Airbyte Source for Zoom Singer

Documentation can be found at https://docs.airbyte.com/integrations/sources/zoom

class dagster_airbyte.managed.generated.sources.TiktokMarketingSource(name, credentials, start_date=None, end_date=None, report_granularity=None)[source]¶

__init__(name, credentials, start_date=None, end_date=None, report_granularity=None)[source]¶

Airbyte Source for Tiktok Marketing

Documentation can be found at https://docs.airbyte.com/integrations/sources/tiktok-marketing

class dagster_airbyte.managed.generated.sources.ZendeskChatSource(name, start_date, credentials, subdomain=None)[source]¶

__init__(name, start_date, credentials, subdomain=None)[source]¶

Airbyte Source for Zendesk Chat

Documentation can be found at https://docs.airbyte.com/integrations/sources/zendesk-chat

class dagster_airbyte.managed.generated.sources.AwsCloudtrailSource(name, aws_key_id, aws_secret_key, aws_region_name, start_date)[source]¶

__init__(name, aws_key_id, aws_secret_key, aws_region_name, start_date)[source]¶

Airbyte Source for Aws Cloudtrail

Documentation can be found at https://docs.airbyte.com/integrations/sources/aws-cloudtrail

class dagster_airbyte.managed.generated.sources.OktaSource(name, credentials, domain=None, start_date=None)[source]¶

__init__(name, credentials, domain=None, start_date=None)[source]¶

Airbyte Source for Okta

Documentation can be found at https://docs.airbyte.com/integrations/sources/okta

class dagster_airbyte.managed.generated.sources.InsightlySource(name, token=None, start_date=None)[source]¶

__init__(name, token=None, start_date=None)[source]¶

Airbyte Source for Insightly

Documentation can be found at https://docs.airbyte.com/integrations/sources/insightly

class dagster_airbyte.managed.generated.sources.LinkedinPagesSource(name, org_id, credentials)[source]¶

__init__(name, org_id, credentials)[source]¶

Airbyte Source for Linkedin Pages

Documentation can be found at https://docs.airbyte.com/integrations/sources/linkedin-pages/

class dagster_airbyte.managed.generated.sources.PersistiqSource(name, api_key)[source]¶

__init__(name, api_key)[source]¶

Airbyte Source for Persistiq

Documentation can be found at https://docs.airbyte.com/integrations/sources/persistiq

class dagster_airbyte.managed.generated.sources.FreshcallerSource(name, domain, api_key, start_date, requests_per_minute=None, sync_lag_minutes=None)[source]¶

__init__(name, domain, api_key, start_date, requests_per_minute=None, sync_lag_minutes=None)[source]¶

Airbyte Source for Freshcaller

Documentation can be found at https://docs.airbyte.com/integrations/sources/freshcaller

class dagster_airbyte.managed.generated.sources.AppfollowSource(name, ext_id, cid, api_secret, country)[source]¶

__init__(name, ext_id, cid, api_secret, country)[source]¶

Airbyte Source for Appfollow

Documentation can be found at https://docs.airbyte.com/integrations/sources/appfollow

class dagster_airbyte.managed.generated.sources.FacebookPagesSource(name, access_token, page_id)[source]¶

__init__(name, access_token, page_id)[source]¶

Airbyte Source for Facebook Pages

Documentation can be found at https://docs.airbyte.com/integrations/sources/facebook-pages

class dagster_airbyte.managed.generated.sources.JiraSource(name, api_token, domain, email, projects=None, start_date=None, additional_fields=None, expand_issue_changelog=None, render_fields=None, enable_experimental_streams=None)[source]¶

__init__(name, api_token, domain, email, projects=None, start_date=None, additional_fields=None, expand_issue_changelog=None, render_fields=None, enable_experimental_streams=None)[source]¶

Airbyte Source for Jira

Documentation can be found at https://docs.airbyte.com/integrations/sources/jira

class dagster_airbyte.managed.generated.sources.GoogleSheetsSource(name, spreadsheet_id, credentials, row_batch_size=None)[source]¶

__init__(name, spreadsheet_id, credentials, row_batch_size=None)[source]¶

Airbyte Source for Google Sheets

Documentation can be found at https://docs.airbyte.com/integrations/sources/google-sheets

class dagster_airbyte.managed.generated.sources.DockerhubSource(name, docker_username)[source]¶

__init__(name, docker_username)[source]¶

Airbyte Source for Dockerhub

Documentation can be found at https://docs.airbyte.com/integrations/sources/dockerhub

class dagster_airbyte.managed.generated.sources.UsCensusSource(name, query_path, api_key, query_params=None)[source]¶

__init__(name, query_path, api_key, query_params=None)[source]¶

Airbyte Source for Us Census

Documentation can be found at https://docs.airbyte.com/integrations/sources/us-census

class dagster_airbyte.managed.generated.sources.KustomerSingerSource(name, api_token, start_date)[source]¶

__init__(name, api_token, start_date)[source]¶

Airbyte Source for Kustomer Singer

Documentation can be found at https://docs.airbyte.com/integrations/sources/kustomer

class dagster_airbyte.managed.generated.sources.AzureTableSource(name, storage_account_name, storage_access_key, storage_endpoint_suffix=None)[source]¶

__init__(name, storage_account_name, storage_access_key, storage_endpoint_suffix=None)[source]¶

Airbyte Source for Azure Table

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.ScaffoldJavaJdbcSource(name, host, port, database, username, replication_method, password=None, jdbc_url_params=None)[source]¶

__init__(name, host, port, database, username, replication_method, password=None, jdbc_url_params=None)[source]¶

Airbyte Source for Scaffold Java Jdbc

Documentation can be found at https://docs.airbyte.com/integrations/sources/scaffold_java_jdbc

class dagster_airbyte.managed.generated.sources.TidbSource(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

__init__(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

Airbyte Source for Tidb

Documentation can be found at https://docs.airbyte.com/integrations/sources/tidb

class dagster_airbyte.managed.generated.sources.QualarooSource(name, token, key, start_date, survey_ids=None)[source]¶

__init__(name, token, key, start_date, survey_ids=None)[source]¶

Airbyte Source for Qualaroo

Documentation can be found at https://docs.airbyte.com/integrations/sources/qualaroo

class dagster_airbyte.managed.generated.sources.YahooFinancePriceSource(name, tickers, interval=None, range=None)[source]¶

__init__(name, tickers, interval=None, range=None)[source]¶

Airbyte Source for Yahoo Finance Price

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.GoogleAnalyticsV4Source(name, credentials, start_date, view_id, custom_reports=None, window_in_days=None)[source]¶

__init__(name, credentials, start_date, view_id, custom_reports=None, window_in_days=None)[source]¶

Airbyte Source for Google Analytics V4

Documentation can be found at https://docs.airbyte.com/integrations/sources/google-analytics-universal-analytics

class dagster_airbyte.managed.generated.sources.JdbcSource(name, username, jdbc_url, password=None, jdbc_url_params=None)[source]¶

__init__(name, username, jdbc_url, password=None, jdbc_url_params=None)[source]¶

Airbyte Source for Jdbc

Documentation can be found at https://docs.airbyte.com/integrations/sources/postgres

class dagster_airbyte.managed.generated.sources.FakerSource(name, count, seed=None, records_per_sync=None, records_per_slice=None)[source]¶

__init__(name, count, seed=None, records_per_sync=None, records_per_slice=None)[source]¶

Airbyte Source for Faker

Documentation can be found at https://docs.airbyte.com/integrations/sources/faker

class dagster_airbyte.managed.generated.sources.TplcentralSource(name, url_base, client_id, client_secret, user_login_id=None, user_login=None, tpl_key=None, customer_id=None, facility_id=None, start_date=None)[source]¶

__init__(name, url_base, client_id, client_secret, user_login_id=None, user_login=None, tpl_key=None, customer_id=None, facility_id=None, start_date=None)[source]¶

Airbyte Source for Tplcentral

Documentation can be found at https://docs.airbyte.com/integrations/sources/tplcentral

class dagster_airbyte.managed.generated.sources.ClickhouseSource(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

__init__(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

Airbyte Source for Clickhouse

Documentation can be found at https://docs.airbyte.com/integrations/destinations/clickhouse

class dagster_airbyte.managed.generated.sources.FreshserviceSource(name, domain_name, api_key, start_date)[source]¶

__init__(name, domain_name, api_key, start_date)[source]¶

Airbyte Source for Freshservice

Documentation can be found at https://docs.airbyte.com/integrations/sources/freshservice

class dagster_airbyte.managed.generated.sources.ZenloopSource(name, api_token, date_from=None, survey_id=None, survey_group_id=None)[source]¶

__init__(name, api_token, date_from=None, survey_id=None, survey_group_id=None)[source]¶

Airbyte Source for Zenloop

Documentation can be found at https://docs.airbyte.com/integrations/sources/zenloop

class dagster_airbyte.managed.generated.sources.OracleSource(name, host, port, connection_data, username, encryption, password=None, schemas=None, jdbc_url_params=None)[source]¶

__init__(name, host, port, connection_data, username, encryption, password=None, schemas=None, jdbc_url_params=None)[source]¶

Airbyte Source for Oracle

Documentation can be found at https://docs.airbyte.com/integrations/sources/oracle

class dagster_airbyte.managed.generated.sources.KlaviyoSource(name, api_key, start_date)[source]¶

__init__(name, api_key, start_date)[source]¶

Airbyte Source for Klaviyo

Documentation can be found at https://docs.airbyte.com/integrations/sources/klaviyo

class dagster_airbyte.managed.generated.sources.GoogleDirectorySource(name, credentials)[source]¶

__init__(name, credentials)[source]¶

Airbyte Source for Google Directory

Documentation can be found at https://docs.airbyte.com/integrations/sources/google-directory

class dagster_airbyte.managed.generated.sources.InstagramSource(name, start_date, access_token)[source]¶

__init__(name, start_date, access_token)[source]¶

Airbyte Source for Instagram

Documentation can be found at https://docs.airbyte.com/integrations/sources/instagram

class dagster_airbyte.managed.generated.sources.ShortioSource(name, domain_id, secret_key, start_date)[source]¶

__init__(name, domain_id, secret_key, start_date)[source]¶

Airbyte Source for Shortio

Documentation can be found at https://developers.short.io/reference

class dagster_airbyte.managed.generated.sources.SquareSource(name, is_sandbox, credentials, start_date=None, include_deleted_objects=None)[source]¶

__init__(name, is_sandbox, credentials, start_date=None, include_deleted_objects=None)[source]¶

Airbyte Source for Square

Documentation can be found at https://docs.airbyte.com/integrations/sources/square

class dagster_airbyte.managed.generated.sources.DelightedSource(name, since, api_key)[source]¶

__init__(name, since, api_key)[source]¶

Airbyte Source for Delighted

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.AmazonSqsSource(name, queue_url, region, delete_messages, max_batch_size=None, max_wait_time=None, attributes_to_return=None, visibility_timeout=None, access_key=None, secret_key=None)[source]¶

__init__(name, queue_url, region, delete_messages, max_batch_size=None, max_wait_time=None, attributes_to_return=None, visibility_timeout=None, access_key=None, secret_key=None)[source]¶

Airbyte Source for Amazon Sqs

Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-sqs

class dagster_airbyte.managed.generated.sources.YoutubeAnalyticsSource(name, credentials)[source]¶

__init__(name, credentials)[source]¶

Airbyte Source for Youtube Analytics

Documentation can be found at https://docs.airbyte.com/integrations/sources/youtube-analytics

class dagster_airbyte.managed.generated.sources.ScaffoldSourcePythonSource(name, fix_me=None)[source]¶

__init__(name, fix_me=None)[source]¶

Airbyte Source for Scaffold Source Python

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.LookerSource(name, domain, client_id, client_secret, run_look_ids=None)[source]¶

__init__(name, domain, client_id, client_secret, run_look_ids=None)[source]¶

Airbyte Source for Looker

Documentation can be found at https://docs.airbyte.com/integrations/sources/looker

class dagster_airbyte.managed.generated.sources.GitlabSource(name, api_url, private_token, start_date, groups=None, projects=None)[source]¶

__init__(name, api_url, private_token, start_date, groups=None, projects=None)[source]¶

Airbyte Source for Gitlab

Documentation can be found at https://docs.airbyte.com/integrations/sources/gitlab

class dagster_airbyte.managed.generated.sources.ExchangeRatesSource(name, start_date, access_key, base=None, ignore_weekends=None)[source]¶

__init__(name, start_date, access_key, base=None, ignore_weekends=None)[source]¶

Airbyte Source for Exchange Rates

Documentation can be found at https://docs.airbyte.com/integrations/sources/exchangeratesapi

class dagster_airbyte.managed.generated.sources.AmazonAdsSource(name, client_id, client_secret, refresh_token, auth_type=None, region=None, report_wait_timeout=None, report_generation_max_retries=None, start_date=None, profiles=None, state_filter=None)[source]¶

__init__(name, client_id, client_secret, refresh_token, auth_type=None, region=None, report_wait_timeout=None, report_generation_max_retries=None, start_date=None, profiles=None, state_filter=None)[source]¶

Airbyte Source for Amazon Ads

Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-ads

class dagster_airbyte.managed.generated.sources.MixpanelSource(name, credentials, project_id=None, attribution_window=None, project_timezone=None, select_properties_by_default=None, start_date=None, end_date=None, region=None, date_window_size=None)[source]¶

__init__(name, credentials, project_id=None, attribution_window=None, project_timezone=None, select_properties_by_default=None, start_date=None, end_date=None, region=None, date_window_size=None)[source]¶

Airbyte Source for Mixpanel

Documentation can be found at https://docs.airbyte.com/integrations/sources/mixpanel

class dagster_airbyte.managed.generated.sources.OrbitSource(name, api_token, workspace, start_date=None)[source]¶

__init__(name, api_token, workspace, start_date=None)[source]¶

Airbyte Source for Orbit

Documentation can be found at https://docs.airbyte.com/integrations/sources/orbit

class dagster_airbyte.managed.generated.sources.AmazonSellerPartnerSource(name, lwa_app_id, lwa_client_secret, refresh_token, aws_access_key, aws_secret_key, role_arn, replication_start_date, aws_environment, region, app_id=None, auth_type=None, replication_end_date=None, period_in_days=None, report_options=None, max_wait_seconds=None)[source]¶

__init__(name, lwa_app_id, lwa_client_secret, refresh_token, aws_access_key, aws_secret_key, role_arn, replication_start_date, aws_environment, region, app_id=None, auth_type=None, replication_end_date=None, period_in_days=None, report_options=None, max_wait_seconds=None)[source]¶

Airbyte Source for Amazon Seller Partner

Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-seller-partner

class dagster_airbyte.managed.generated.sources.CourierSource(name, api_key)[source]¶

__init__(name, api_key)[source]¶

Airbyte Source for Courier

Documentation can be found at https://docs.airbyte.io/integrations/sources/courier

class dagster_airbyte.managed.generated.sources.CloseComSource(name, api_key, start_date=None)[source]¶

__init__(name, api_key, start_date=None)[source]¶

Airbyte Source for Close Com

Documentation can be found at https://docs.airbyte.com/integrations/sources/close-com

class dagster_airbyte.managed.generated.sources.BingAdsSource(name, client_id, refresh_token, developer_token, reports_start_date, auth_method=None, tenant_id=None, client_secret=None)[source]¶

__init__(name, client_id, refresh_token, developer_token, reports_start_date, auth_method=None, tenant_id=None, client_secret=None)[source]¶

Airbyte Source for Bing Ads

Documentation can be found at https://docs.airbyte.com/integrations/sources/bing-ads

class dagster_airbyte.managed.generated.sources.PrimetricSource(name, client_id, client_secret)[source]¶

__init__(name, client_id, client_secret)[source]¶

Airbyte Source for Primetric

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.PivotalTrackerSource(name, api_token)[source]¶

__init__(name, api_token)[source]¶

Airbyte Source for Pivotal Tracker

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.ElasticsearchSource(name, endpoint, authenticationMethod)[source]¶

__init__(name, endpoint, authenticationMethod)[source]¶

Airbyte Source for Elasticsearch

Documentation can be found at https://docs.airbyte.com/integrations/source/elasticsearch

class dagster_airbyte.managed.generated.sources.BigquerySource(name, project_id, credentials_json, dataset_id=None)[source]¶

__init__(name, project_id, credentials_json, dataset_id=None)[source]¶

Airbyte Source for Bigquery

Documentation can be found at https://docs.airbyte.com/integrations/sources/bigquery

class dagster_airbyte.managed.generated.sources.WoocommerceSource(name, shop, start_date, api_key, api_secret, conversion_window_days=None)[source]¶

__init__(name, shop, start_date, api_key, api_secret, conversion_window_days=None)[source]¶

Airbyte Source for Woocommerce

Documentation can be found at https://docs.airbyte.com/integrations/sources/woocommerce

class dagster_airbyte.managed.generated.sources.SearchMetricsSource(name, api_key, client_secret, country_code, start_date)[source]¶

__init__(name, api_key, client_secret, country_code, start_date)[source]¶

Airbyte Source for Search Metrics

Documentation can be found at https://docs.airbyte.com/integrations/sources/seacrh-metrics

class dagster_airbyte.managed.generated.sources.TypeformSource(name, start_date, token, form_ids=None)[source]¶

__init__(name, start_date, token, form_ids=None)[source]¶

Airbyte Source for Typeform

Documentation can be found at https://docs.airbyte.com/integrations/sources/typeform

class dagster_airbyte.managed.generated.sources.WebflowSource(name, site_id, api_key)[source]¶

__init__(name, site_id, api_key)[source]¶

Airbyte Source for Webflow

Documentation can be found at https://docs.airbyte.com/integrations/sources/webflow

class dagster_airbyte.managed.generated.sources.FireboltSource(name, username, password, database, account=None, host=None, engine=None)[source]¶

__init__(name, username, password, database, account=None, host=None, engine=None)[source]¶

Airbyte Source for Firebolt

Documentation can be found at https://docs.airbyte.com/integrations/sources/firebolt

class dagster_airbyte.managed.generated.sources.FaunaSource(name, domain, port, scheme, secret, collection)[source]¶

__init__(name, domain, port, scheme, secret, collection)[source]¶

Airbyte Source for Fauna

Documentation can be found at https://github.com/fauna/airbyte/blob/source-fauna/docs/integrations/sources/fauna.md

class dagster_airbyte.managed.generated.sources.IntercomSource(name, start_date, access_token)[source]¶

__init__(name, start_date, access_token)[source]¶

Airbyte Source for Intercom

Documentation can be found at https://docs.airbyte.com/integrations/sources/intercom

class dagster_airbyte.managed.generated.sources.FreshsalesSource(name, domain_name, api_key)[source]¶

__init__(name, domain_name, api_key)[source]¶

Airbyte Source for Freshsales

Documentation can be found at https://docs.airbyte.com/integrations/sources/freshsales

class dagster_airbyte.managed.generated.sources.AdjustSource(name, api_token, dimensions, ingest_start, metrics, additional_metrics=None, until_today=None)[source]¶

__init__(name, api_token, dimensions, ingest_start, metrics, additional_metrics=None, until_today=None)[source]¶

Airbyte Source for Adjust

Documentation can be found at https://docs.airbyte.com/integrations/sources/adjust

class dagster_airbyte.managed.generated.sources.BambooHrSource(name, subdomain, api_key, custom_reports_fields=None, custom_reports_include_default_fields=None)[source]¶

__init__(name, subdomain, api_key, custom_reports_fields=None, custom_reports_include_default_fields=None)[source]¶

Airbyte Source for Bamboo Hr

Documentation can be found at https://docs.airbyte.com/integrations/sources/bamboo-hr

class dagster_airbyte.managed.generated.sources.GoogleAdsSource(name, credentials, customer_id, start_date, end_date=None, custom_queries=None, login_customer_id=None, conversion_window_days=None)[source]¶

__init__(name, credentials, customer_id, start_date, end_date=None, custom_queries=None, login_customer_id=None, conversion_window_days=None)[source]¶

Airbyte Source for Google Ads

Documentation can be found at https://docs.airbyte.com/integrations/sources/google-ads

class dagster_airbyte.managed.generated.sources.HellobatonSource(name, api_key, company)[source]¶

__init__(name, api_key, company)[source]¶

Airbyte Source for Hellobaton

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.SendgridSource(name, apikey, start_time)[source]¶

__init__(name, apikey, start_time)[source]¶

Airbyte Source for Sendgrid

Documentation can be found at https://docs.airbyte.com/integrations/sources/sendgrid

class dagster_airbyte.managed.generated.sources.MondaySource(name, credentials)[source]¶

__init__(name, credentials)[source]¶

Airbyte Source for Monday

Documentation can be found at https://docs.airbyte.com/integrations/sources/monday

class dagster_airbyte.managed.generated.sources.DixaSource(name, api_token, start_date, batch_size=None)[source]¶

__init__(name, api_token, start_date, batch_size=None)[source]¶

Airbyte Source for Dixa

Documentation can be found at https://docs.airbyte.com/integrations/sources/dixa

class dagster_airbyte.managed.generated.sources.SalesforceSource(name, client_id, client_secret, refresh_token, is_sandbox=None, auth_type=None, start_date=None, streams_criteria=None)[source]¶

__init__(name, client_id, client_secret, refresh_token, is_sandbox=None, auth_type=None, start_date=None, streams_criteria=None)[source]¶

Airbyte Source for Salesforce

Documentation can be found at https://docs.airbyte.com/integrations/sources/salesforce

class dagster_airbyte.managed.generated.sources.PipedriveSource(name, authorization, replication_start_date)[source]¶

__init__(name, authorization, replication_start_date)[source]¶

Airbyte Source for Pipedrive

Documentation can be found at https://docs.airbyte.com/integrations/sources/pipedrive

class dagster_airbyte.managed.generated.sources.FileSource(name, dataset_name, format, url, provider, reader_options=None)[source]¶

__init__(name, dataset_name, format, url, provider, reader_options=None)[source]¶

Airbyte Source for File

Documentation can be found at https://docs.airbyte.com/integrations/sources/file

class dagster_airbyte.managed.generated.sources.GlassfrogSource(name, api_key)[source]¶

__init__(name, api_key)[source]¶

Airbyte Source for Glassfrog

Documentation can be found at https://docs.airbyte.com/integrations/sources/glassfrog

class dagster_airbyte.managed.generated.sources.ChartmogulSource(name, api_key, start_date, interval)[source]¶

__init__(name, api_key, start_date, interval)[source]¶

Airbyte Source for Chartmogul

Documentation can be found at https://docs.airbyte.com/integrations/sources/chartmogul

class dagster_airbyte.managed.generated.sources.OrbSource(name, api_key, start_date=None, lookback_window_days=None, string_event_properties_keys=None, numeric_event_properties_keys=None)[source]¶

__init__(name, api_key, start_date=None, lookback_window_days=None, string_event_properties_keys=None, numeric_event_properties_keys=None)[source]¶

Airbyte Source for Orb

Documentation can be found at https://docs.withorb.com/

class dagster_airbyte.managed.generated.sources.CockroachdbSource(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

__init__(name, host, port, database, username, password=None, jdbc_url_params=None, ssl=None)[source]¶

Airbyte Source for Cockroachdb

Documentation can be found at https://docs.airbyte.com/integrations/sources/cockroachdb

class dagster_airbyte.managed.generated.sources.ConfluenceSource(name, api_token, domain_name, email)[source]¶

__init__(name, api_token, domain_name, email)[source]¶

Airbyte Source for Confluence

Documentation can be found at https://docsurl.com

class dagster_airbyte.managed.generated.sources.PlaidSource(name, access_token, api_key, client_id, plaid_env, start_date=None)[source]¶

__init__(name, access_token, api_key, client_id, plaid_env, start_date=None)[source]¶

Airbyte Source for Plaid

Documentation can be found at https://plaid.com/docs/api/