 | S3 Avro
Avro serialization format written to AWS S3. |
The target-s3-avro loader sends data into S3 Avro after it was pulled from a source using an extractor
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the target-s3-avro loader to your project
using
:meltano add -
Configure the target-s3-avro settings using
:meltano config
meltano add loader target-s3-avromeltano config target-s3-avro set --interactiveNext steps
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
The
target-s3-avro settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
verifyaws_session_tokenapi_versionendpoint_urlaws_secret_access_keyaws_access_key_idflatten_delimiterregion_nametmp_diruse_ssltarget_schema_bucket_keyconfigtarget_bucket_key
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Verify (verify)
-
Environment variable:
TARGET_S3_AVRO_VERIFY
Whether or not to verify SSL certificates.
Aws Session Token (aws_session_token)
-
Environment variable:
TARGET_S3_AVRO_AWS_SESSION_TOKEN
The session token to use when creating the client. Same semantics as aws_access_key_id.
Api Version (api_version)
-
Environment variable:
TARGET_S3_AVRO_API_VERSION
The API version to use. By default, botocore will use the latest API version when creating a client. You only need to specify this parameter if you want to use a previous API version of the client.
Endpoint Url (endpoint_url)
-
Environment variable:
TARGET_S3_AVRO_ENDPOINT_URL
The complete URL to use for the constructed client. Normally, botocore will automatically construct the appropriate URL to use when communicating with a service. You can specify a complete URL (including the "http/https" scheme) to override this behavior. If this value is provided, then use_ssl is ignored.
Aws Secret Access Key (aws_secret_access_key)
-
Environment variable:
TARGET_S3_AVRO_AWS_SECRET_ACCESS_KEY
The secret key to use when creating the client. Same semantics as aws_access_key_id above.
Aws Access Key Id (aws_access_key_id)
-
Environment variable:
TARGET_S3_AVRO_AWS_ACCESS_KEY_ID
The access key to use when creating the client. This is entirely optional, and if not provided, the credentials configured for the session will automatically be used. You only need to provide this argument if you want to override the credentials used for this specific client.
Flatten Delimiter (flatten_delimiter)
-
Environment variable:
TARGET_S3_AVRO_FLATTEN_DELIMITER
The delimiter to use when flatttening records.
Region Name (region_name)
-
Environment variable:
TARGET_S3_AVRO_REGION_NAME
The name of the region associated with the client. A client is associated with a single region.
Tmp Dir (tmp_dir)
-
Environment variable:
TARGET_S3_AVRO_TMP_DIR
The directory path for where to write temp files. Default to current working directory.
Use SSL (use_ssl)
-
Environment variable:
TARGET_S3_AVRO_USE_SSL
Whether or not to use SSL. By default, SSL is used. Note that not all services support non-ssl connections.
Target Schema Bucket Key (target_schema_bucket_key)
-
Environment variable:
TARGET_S3_AVRO_TARGET_SCHEMA_BUCKET_KEY
Schema destination bucket and key folder in the format of {bucket}/{key}
Config (config)
-
Environment variable:
TARGET_S3_AVRO_CONFIG
Advanced client configuration options. If region_name is specified in the client config, its value will take precedence over environment variables and configuration values, but not over a region_name value passed explicitly to the method. See botocore config documentation for more details. See boto Docs for more details.
Target Bucket Key (target_bucket_key)
-
Environment variable:
TARGET_S3_AVRO_TARGET_BUCKET_KEY
Destination bucket and key folder in the format of {bucket}/{key}.
Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-generalInstall
meltano add loader target-s3-avro
Maintenance Status
Maintainer
Meltano Stats
-1-c0c0c4){kind=icon}
-1-c0c0c4){kind=icon}
Keywords
