> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/reference-1/sql-commands/jobs/create-job/transformation/job-options/amazon-s3.md).

# Amazon S3

## Job options

```sql
[ AGGREGATION_PARALLELISM = <integer> ]
[ COMMENT = '<comment>' ]
[ COMPRESSION = { NONE 
                | GZIP 
                | SNAPPY 
                | ZSTD } ]
[ COMPUTE_CLUSTER = <cluster_identifier> ]
[ DATE_PATTERN = '<date_pattern>' ]
[ END_AT = { NOW | timestamp } ]
[ FILE_FORMAT = { CSV 
                | JSON 
                | PARQUET
                | AVRO 
                | TSV } ]
[ OUTPUT_OFFSET = <integer> { MINUTE[S] | HOUR[S] | DAY[S] } ]
[ RUN_INTERVAL = <integer> { MINUTE[S] | HOUR[S] | DAY[S] } ]
[ RUN_PARALLELISM = <integer> ]
[ START_FROM = { NOW | BEGINNING | timestamp } ]
```

#### Jump to

Amazon S3 job options:

* [`AGGREGATION_PARALLELISM`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#aggregation_parallelism-editable)
* [`COMPRESSION`](#compression)
* [`DATE_PATTERN`](#date_pattern)
* [`FILE_FORMAT`](#file_format)
* [`OUTPUT_OFFSET`](#output_offset)

General job options:

* [`COMMENT`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#comment-editable)
* [`COMPUTE_CLUSTER`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#compute_cluster-editable)
* [`END_AT`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#end_at-editable)
* [`RUN_INTERVAL`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#run_interval)
* [`RUN_PARALLELISM`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#run_parallelism-editable)
* [`START_FROM`](/content/reference-1/sql-commands/jobs/create-job/transformation/job-options.md#start_from)

#### `AGGREGATION_PARALLELISM` — editable

Type: `integer`

Default: `1`

(Optional) Only supported when the query contains aggregations. Formally known as "output sharding."

#### `COMPRESSION`

Values: `{ NONE | GZIP | SNAPPY | ZSTD }`

Default: `NONE`

(Optional) The compression for the output files.

#### `DATE_PATTERN`

Type: `text`

Default:  `'yyyy/MM/dd/HH/mm'`

(Optional) Upsolver uses the date pattern to partition the output on the S3 bucket. Upsolver supports partitioning up to the minute, for example: 'yyyy/MM/dd/HH/mm'. For more options, see [Java SimpleDateFormat](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html)

#### **`FILE_FORMAT`**

Values: `{ CSV | TSV | AVRO | PARQUET | JSON }`

The file format for the output file. The following options can be configured for CSV and TSV formats:

#### **`CSV`**

```sql
FILE_FORMAT = (
    TYPE = CSV
    [ DELIMITER = '<delimiter>' ]
  )
```

**`DELIMITER`**&#x20;

Type: `text`&#x20;

Default: `,`&#x20;

(Optional) Configures the delimiter to separate the values in the output file. For binary targets, use `DELIMITER = '\u0001'`

#### **`TSV`**

```sql
FILE_FORMAT = (
    TYPE = TSV
    [ HEADERLESS = { TRUE | FALSE } ]
  )
```

**`HEADERLESS`**&#x20;

Type: `Boolean`&#x20;

Default: `false`&#x20;

(Optional) When `true`, the column names are used as the header row in the output file.

#### `OUTPUT_OFFSET`

Value: `<integer> { MINUTE[S] | HOUR[S] | DAY[S] }`

Default: 0

(Optional) By default, the file 2023/01/01/00/01 contains data for 2023-01-01 00:00 - 2023-01-01 00:00.59.999. Setting OUTPUT\_OFFSET to `1 MINUTE` add to that so a value of the first minute will move the file name to 02, if you want to move it back you can use negative values.

## Location Options

#### `LOCATION`

Type: `text`

The target location to write files to, as a full S3 URI. The location URI pattern can include macros referring to data columns, this allows custom partitioning of the data in the target location.&#x20;

Supported macros:\
\
**Time:** `{time:<date-pattern>}` \
This macro will be replaced with the job execution time at runtime. The date pattern provided must be in [Java's date formatting syntax](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). Only a single-time macro can be used in the location.

**Column:** `{col:<column-name>}`\
This macro will be replaced with the value of the column provided. The column provided must appear in the select statement of the job.&#x20;

**Shard:** `{shard:format}`\
This macro will be replaced by the output shard number writing the current file. It is important to use this as part of your pattern if you are using `RUN_PARALLELISM,` otherwise, each shard will overwrite the file. \
The supported format is a **subset** of [Java's string fromat syntax](https://docs.oracle.com/javase/8/docs/api/java/util/Formatter.html). The supported options are either: \
1\. %0xd - Will result in a shard number padded with x-1 leading 0's.  For example, %05d will result in 00001 for shard number 1. \
2\. %d - Will simply use the shard number with no padding. \
Usually, it's recommended to include padding to ensure alphabetical sorting of the output files.

{% hint style="info" %}
If the location provided ends with a `/` and contains no date pattern, a default date pattern is added to the end of the path
{% endhint %}

Example location URI:\
`s3://my-bucket/some/prefix/{time:yyyy-MM-dd-HH-mm}/{col:country}/output.json`


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://upsolver.gitbook.io/content/reference-1/sql-commands/jobs/create-job/transformation/job-options/amazon-s3.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.