Skip to main content

Setup Amazon S3 source

Updated

Loads CSV, JSON, Parquet, Excel, XML, AVRO and DynamoDB export files from an S3 bucket — single files, whole folders, or zipped archives.

Setup at a glance

  1. 01Amazon S3Acquire credentials
  2. 02KleeneComplete Setup
  3. Connected
  • Connector type Cloud storage
  • Auth method AWS access key & secret key

Connect Amazon S3 to Kleene

  1. Get your Amazon S3 credentials

    1. Get your AWS Access Key and AWS Secret Key — see Amazon documentation here.

  2. Add the source in Kleene

    1. In the Kleene app, go to Add a source.
    2. Scroll down or search for Amazon S3.
    3. Click Connect to open the credentials panel.

  3. Enter your credentials and connect

    • RegionThe region your bucket is in, e.g. eu-west-1
    • AWS Access Key
    • AWS Secret Key

    The new source now appears on the Sources screen.

Before you extract — Amazon S3 limits and report notes

This guide covers source setup only. When you move on to creating extracts, these are the connector-specific things to know.

Connector capabilities

Incremental Yes — for incremental loads, only files that have not been loaded previously will be processed

What you can extract

There are no fixed reports — the load type and file type you choose at extract level determine the output. See the full Amazon S3 connector documentation for the complete list of options.

Extract notes

  • S3 Bucket Name — the bucket to read from.
  • Load type — load a single file (enter the path to the file), load all files inside a folder (enter the path to the folder), load a single file inside a ZIP (enter the path to the zip, and separately the path to the file within the zip), or load all files inside a ZIP (enter the path to the zip).
  • File Type — CSV, JSON, Parquet, TXT, XLS, XLSX, XML, Dynamo DB export or AVRO.
  • Values delimiter — if TXT is chosen, choose the method of delimitation: Tab, Pipe (|) or Comma (,).
  • Header — if checked, the output columns are generated from the key names of the first row of data in the input files. Otherwise, columns are named column1, column2 … columnN.
  • Sheet Name — applies only to XLS and XLSX. Enter a value to load data from the specified sheet; leave blank to load from the first sheet.
  • Load only files newer than X number of days ago — applies when "Load all files inside a folder" is selected. Any files older than X days ago are ignored.
  • Filter Files — select all files that follow a certain pattern. Input a wildcard in the Regex pattern field — e.g. *.csv returns all files following the pattern <something>.csv.
  • Once the files have been loaded, the list of ingested files is stored in the _KLEENE_FILENAME column in the destination warehouse table.

Limits and gotchas

Identical data structure required for folder loads

When loading all files from inside a folder, the data structure of each file must be identical.

Zipped files are handled automatically

Files ending .zip are automatically unzipped and loaded.

Limitations

  • When "Load all files inside a folder" has been selected, only files with extensions that end with the respective file type are processed. For example, when File Type: JSON is selected, only files ending with .JSON are processed.
  • For DynamoDB export files, a file extension of JSON.gz is expected for the input files.
  • For JSON files, a list of JSON objects is expected on the top level, where each object is a single row in the output. If the top-level object is a single JSON object or hashmap, only a single row will be output.
  • For XML files, no additional un-nesting or processing is done — only one row is output. The entire XML is output as JSON format as a single row into the table column named content.
  • Format and file limitations are rooted in the Snowflake environment — see the Snowflake reference.

Vendor reference: AWS credentials documentation

Related to

Powered by Zendesk