# Create a data source

This API enables you to create a new data source. All API calls require an [API token](https://upsolver.gitbook.io/upsolver-1/guide-for-developers/upsolver-rest-api).

```http
POST https://api.upsolver.com/inputs
```

## Amazon S3 (Quick)

Connect to your AWS S3 Bucket.&#x20;

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

{% hint style="info" %}
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. \
**See:** [S3 connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/amazon-s3-connection)
{% endhint %}

#### Fields

| Field                   | Name                 | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| bucket                  | Bucket               | String            | The Amazon S3 bucket to read from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |          |
| globFilePattern         | Glob File Pattern    | String            | The pattern for files to ingest.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| datePattern             | Date Pattern         | String            | The date pattern in the file name/folder structure (e.g. `yyyy/MM/dd/HH/mm`). The date format specification must be set according to [Java DateTimeFormatter format](https://upsolver.gitbook.io/upsolver-1/getting-started/glossary/language-guide/functions/calculated-functions/external-apis-functions).                                                                                                                                                                                                                                      |          |
| contentType             | Content Format       | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| compression             | Compression          | Compression       | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| displayData.name        | Name                 | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description          | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| softRetention           | Soft Retention       | Boolean           | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| prefix                  | Folder               | String            | If the data resides in a sub folder within the defined cloud storage, specify this folder.                                                                                                                                                                                                                                                                                                                                                                                                                                                        | +        |
| startExecutionFrom      | Start Ingestion From | String (ISO-8601) | <p>The time from which to ingest the data from.</p><p>Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.</p>                                                                                                                                                                                                                                                                                                                                                     | +        |
| workspaces              | Workspaces           | String\[]         | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example 1

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "QuickS3StorageInputRequest",
	"bucket" : "bucket",
	"globFilePattern" : "globFilePattern",
	"datePattern" : "yyyy/MM/dd/HH/mm",
	"contentType" : {
		"clazz" : "JsonContentType"
	},
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"displayData" : {
		"name" : "First Amazon S3 Data Source",
		"description" : "Description of first Amazon 
S3 data source"
	},
	"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
```

#### Example 2

```bash
curl 
-X POST 
-H "content-type: application/json" 
-H "Authorization: token" 
-d '{"clazz":"QuickS3StorageInputRequest", 
     "bucket": "upsolver-tutorials-orders", 
     "globFilePattern": "*", 
     "datePattern": "yyyy/MM/dd/HH", 
     "prefix": "data/", 
     "contentType":
                  {
                     "clazz": "JsonContentType"
                  }, 
     "compression": 
                  {
                     "clazz": "AutoDetectCompression"
                  }, 
     "displayData": 
                  {
                     "name": "API test data source", 
                     "description": "Description of first Amazon S3 data source"}, 
                     "softRetention":false
                  }' 
      "https://your-api.upsolver.com/inputs/"
```

## Amazon S3 (Advanced)

Connect to your AWS S3 Bucket.&#x20;

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

{% hint style="info" %}
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. \
**See:** [S3 connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/amazon-s3-connection)
{% endhint %}

#### Fields

| Field                   | Name                 | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Optional |
| ----------------------- | -------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name                 | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |          |
| displayData.description | Description          | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |          |
| sourceStorage           | S3 Connection        | String            | The cloud storage to ingest files from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |          |
| datePattern             | Date Pattern         | String            | The date pattern in the file name/folder structure. For example: `yyyy/MM/dd/HH/mm`. The date format specification must be set according to [Java DateTimeFormatter format](https://upsolver.gitbook.io/upsolver-1/getting-started/glossary/language-guide/functions/calculated-functions/external-apis-functions).                                                                                                                                                                                                              |          |
| fileMatchPattern        | File Name Pattern    | FileNameMatcher   | The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from the bucket specified.                                                                                                                                                                                                                                                                                                                       |          |
| contentType             | Content Format       | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p>See <a href="content-formats">Content Formats</a>.</p> |          |
| computeEnvironment      | Compute Cluster      | String            | The compute cluster to run the calculation on. See [Adding a Compute Cluster](https://upsolver.gitbook.io/upsolver-1/administration/managing-clusters#adding-a-compute-cluster).                                                                                                                                                                                                                                                                                                                                                 |          |
| destinationStorage      | Target Storage       | String            | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                 |          |
| compression             | Compression          | Compression       | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                   |          |
| interval                | Interval             | Int (Minutes)     | The sliding interval to wait for data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| prefix                  | Folder               | String            | If the data resides in a sub folder within the defined cloud storage, specify this folder.                                                                                                                                                                                                                                                                                                                                                                                                                                       | +        |
| startExecutionFrom      | Start Ingestion From | String (ISO-8601) | <p>The time from which to ingest the data from.</p><p>Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.</p>                                                                                                                                                                                                                                                                                                                                    | +        |
| retention               | Retention            | Int (Minutes)     | The retention period for the data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | +        |
| dataDumpDate            | Data Dump Date       | String (ISO-8601) | The date that the data starts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | +        |
| maxDelay                | Max Delay            | Int (Minutes)     | The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.                                                                                                                                                                                                                                                                                                                                                                                                       | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "CloudStorageInputRequest",
	"displayData" : {
		"name" : "First Amazon S3 Data Source",
		"description" : "Description of first Amazon 
S3 data source"
	},
	"sourceStorage" : "aa302f0a-e6ee-44aa-aa38-e28f1ff455f7",
	"datePattern" : "yyyy/MM/dd/HH/mm",
	"fileMatchPattern" : {
		"clazz" : "AllMatcher"
	},
	"contentType" : {
		"type" : "JsonContentType"
	,
	"computeEnvironment" : "53b314af-ffab-419a-9c2c-56032c6ef4c0",
	"destinationStorage" : "e019c0fe-bb80-4cf1-bc7b-aee579d8672e",
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Amazon Kinesis Stream (Quick)

Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.

{% hint style="info" %}
A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream. \
**See:** [Kinesis connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/amazon-kinesis)
{% endhint %}

#### Fields

| Field                   | Name           | Type        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| region                  | Region         | Region      | Your AWS region.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| streamName              | Stream         | String      | The name of the relevant Kinesis stream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |          |
| contentType             | Content Format | ContentType | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| compression             | Compression    | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| displayData.name        | Name           | String      | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description    | String      | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| softRetention           | Soft Retention | Boolean     | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| workspaces              | Workspaces     | String\[]   | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "QuickKinesisInputRequest",
	"region" : "region",
	"streamName" : "streamName",
	"contentType" : {
		"type" : "JsonContentType"
	},
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"displayData" : {
		"name" : "First Amazon Kinesis Stream Data 
Source",
		"description" : "Description of first Amazon 
Kinesis Stream data source"
	},
	"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Amazon Kinesis Stream (Advanced)

Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.

{% hint style="info" %}
A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream. \
**See:** [Kinesis connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/amazon-kinesis)
{% endhint %}

#### Fields

| Field                   | Name                 | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name                 | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description          | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| contentType             | Content Format       | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| kinesisConnection       | Kinesis Connection   | String            | The AWS credentials to connect to Kinesis.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |          |
| streamName              | Stream               | String            | The name of the relevant Kinesis stream.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |          |
| readFromStart           | Read From Start      | String            | <p>The time from which to ingest the data from.</p><p>Messages from before this time will be ignored. If you leave this field empty all messages are ingested.</p>                                                                                                                                                                                                                                                                                                                                                                                |          |
| computeEnvironment      | Compute Cluster      | String            | The compute cluster to run the calculation on. **See:** [Compute cluster](https://upsolver.gitbook.io/upsolver-1/administration/managing-clusters#adding-a-compute-cluster)                                                                                                                                                                                                                                                                                                                                                                       |          |
| connectionPointer       | Target Storage       | String            | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| isOnline                | Real Time Statistics | Boolean           | Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed.                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| shards                  | Shards               | Int               | How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| parallelism             | Parallelism          | Int               | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.                                                                                                                                                                                                                                                                                                                                    |          |
| compression             | Compression          | Compression       | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| retention               | Retention            | Int (Minutes)     | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                       | +        |
| endExecutionAt          | End Read At          | String (ISO-8601) | If configured, stop reading after this date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "KinesisInputRequest",
	"displayData" : {
		"name" : "First Amazon Kinesis Stream Data 
Source",
		"description" : "Description of first Amazon 
Kinesis Stream data source"
	},
	"contentType" : {
		"type" : "JsonContentType"
	},
	"kinesisConnection" : "dcfc39c1-c458-4ec5-87f1-0c7f437ea17e",
	"streamName" : "streamName",
	"readFromStart" : true,
	"computeEnvironment" : "fc2a356d-c3ae-4756-a4a3-1b15158df5e7",
	"connectionPointer" : "4e3787cd-daac-42ea-8dd4-642c516a4a31",
	"isOnline" : true,
	"shards" : 1,
	"parallelism" : 1,
	"compression" : {
		"clazz" : "AutoDetectCompression"
	}
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Amazon S3 over SQS

Connect to your AWS S3 Bucket using SQS Notifications.

{% hint style="info" %}
You will need to configure SQS Notifications from your S3 Bucket and open permissions to read and delete messages from the SQS Queue to the same access key and secret key you entered to give Upsolver permissions to read from the S3 Bucket. \
**See:** [S3 over SQS connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/amazon-s3-over-sqs)
{% endhint %}

#### Fields

| Field                   | Name                 | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name                 | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description          | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| sourceStorage           | Source Storage       | String            | The cloud storage to ingest files from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| contentType             | Content Format       | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment      | Compute Cluster      | String            | The compute cluster to run the calculation on. **See:**[ Compute cluster](https://upsolver.gitbook.io/upsolver-1/administration/managing-clusters#adding-a-compute-cluster)                                                                                                                                                                                                                                                                                                                                                                       |          |
| destinationStorage      | Target Storage       | String            | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| compression             | Compression          | Compression       | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| softRetention           | Soft Retention       | Boolean           | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| executionParallelism    | Parallelism          | Int               | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.                                                                                                                                                                                                                                                                                                                                    |          |
| prefix                  | Prefix               | String            | The prefix of the files or directories. To filter a specific directory, add a trailing `/`.                                                                                                                                                                                                                                                                                                                                                                                                                                                       | +        |
| suffix                  | Suffix               | String            | The suffix of the files to read.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | +        |
| startExecutionFrom      | Start Ingestion From | String (ISO-8601) | <p>The time from which to ingest the data from.</p><p>Messages from before this time will be ignored. If you leave this field empty all messages are ingested.</p>                                                                                                                                                                                                                                                                                                                                                                                | +        |
| endExecutionAt          | End Read At          | String (ISO-8601) | If configured, stop reading after this date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |
| retention               | Retention            | Int (Minutes)     | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                       | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "S3OverSQSInputRequest",
	"displayData" : {
		"name" : "First S3 Over SQS Data Source",
		"description" : "Description of first S3 
Over SQS data source"
	},
	"sourceStorage" : "6ee598b6-4928-4b07-b532-83a79464e5bb",
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "46c2945e-5a46-4d93-9e21-5a85653c28c5",
	"destinationStorage" : "2a08601c-52d2-425e-8d2c-324dbcea5858",
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"softRetention" : true,
	"executionParallelism" : 1
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Apache Kafka (Quick)

Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.&#x20;

{% hint style="info" %}
A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster. \
**See:** [Kafka connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/kafka-connection)
{% endhint %}

#### Fields

| Field                   | Name           | Type         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| kafkaHosts              | Kafka Hosts    | String       | The Kafka hosts separated with commas (e.g. `foo:9092,bar:9092`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| topicName               | Kafka Topic    | String       | The Kafka topic to ingest the data from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |          |
| contentType             | Content Format | ContentType  | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| compression             | Compression    | Compression  | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| displayData.name        | Name           | String       | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description    | String       | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| softRetention           | Soft Retention | Boolean      | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| maybekafkaVersion       | Kafka Version  | KafkaVersion | The version of the Kafka Servers. If unsure, use `0.10.x.x`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |
| workspaces              | Workspaces     | String\[]    | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "QuickKafkaInputRequest",
	"kafkaHosts" : "kafkaHosts",
	"topicName" : "topicName",
	"contentType" : {
		"type" : "JsonContentType"
	},
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"displayData" : {
		"name" : "First Kafka Data Source",
		"description" : "Description of first 
Kafka data source"
	},
	"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Apache Kafka (Advanced)

Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.&#x20;

{% hint style="info" %}
A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster. \
**See:** [Kafka connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/kafka-connection)
{% endhint %}

#### Fields

| Field                   | Name                      | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | ------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name                      | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description               | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| kafkaVersion            | Kafka Version             | KafkaVersion      | The version of the Kafka Servers. If unsure, use `0.10.x.x`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| kafkaHosts              | Kafka Hosts               | String            | The Kafka hosts separated with commas. For example: foo:9092,bar:9092                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| topicName               | Kafka Topic               | String            | The Kafka topic to ingest the data from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |          |
| readFromStart           | Read From Start           | Boolean           | Whether to read the data from the start of the topic or to begin from the end.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| contentType             | Content Format            | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment      | Compute Cluster           | String            | The compute cluster to run the calculation on. **See:** [Compute cluster](https://upsolver.gitbook.io/upsolver-1/administration/managing-clusters#adding-a-compute-cluster)                                                                                                                                                                                                                                                                                                                                                                       |          |
| connectionPointer       | Target Storage            | String            | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| softRetention           | Soft Retention            | Boolean           | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| shards                  | Shards                    | Int               | How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| executionParallelism    | Execution Parallelism     | Int               | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.                                                                                                                                                                                                                                                                                                                                    |          |
| isOnline                | Real Time Statistics      | Boolean           | Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed.                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| useSsl                  | Use SSL                   | Boolean           | Set this to true if your connection requires SSL. Contact us to ensure that your SSL certificate is supported.                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| storeRawData            | Store Raw Data            | Boolean           | Store an additional copy of the data in its original format.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| compression             | Compression               | Compression       | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| consumerProperties      | Kafka Consumer Properties | String            | Extra properties for Kafka Consumer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | +        |
| retention               | Retention                 | Int (Minutes)     | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                       | +        |
| endExecutionAt          | End Read At               | String (ISO-8601) | If configured, stop reading after this date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |
| workspaces              | Workspaces                | String\[]         | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" 
-H "Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "KafkaInputRequest",
	"displayData" : {
		"name" : "First Kafka Data Source",
		"description" : "Description of first 
Kafka data source"
	},
	"kafkaVersion" : "0.10.x.x",
	"kafkaHosts" : "kafkaHosts",
	"topicName" : "topicName",
	"readFromStart" : true,
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "5d8b6e27-6004-48b2-8dc1-1db9f25880cb",
	"connectionPointer" : "36a1d237-1ff0-4574-972b-b071482f3d08",
	"softRetention" : true,
	"shards" : 1,
	"executionParallelism" : 1,
	"isOnline" : true,
	"useSsl" : true,
	"storeRawData" : true,
	"compression" : {
		"clazz" : "AutoDetectCompression"
	}
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Azure Blob storage

Connect to your Azure Blob storage container.&#x20;

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

{% hint style="info" %}
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage.\
**See:** [Azure Blob storage connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/azure-blob-storage-connection)
{% endhint %}

#### Fields

| Field                    | Name                          | Type                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ------------------------ | ----------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name         | Name                          | String                   | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description  | Description                   | String                   | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| sourceStorage            | Azure Blob Storage Connection | String                   | The cloud storage to ingest files from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| datePattern              | Date Pattern                  | String                   | The date pattern in the file name/folder structure (e.g. `yyyy/MM/dd/HH/mm`). The date format specification must be set according to [Java DateTimeFormatter format](https://upsolver.gitbook.io/upsolver-1/getting-started/glossary/language-guide/functions/calculated-functions/external-apis-functions).                                                                                                                                                                                                                                      |          |
| fileMatchPattern         | File Name Pattern             | FileNameMatcher          | The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from the storage container specified.                                                                                                                                                                                                                                                                                                                             |          |
| contentType              | Content Format                | ContentType              | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment       | Compute Cluster               | String                   | <p>The compute cluster to run the calculation on. <br><strong>See:</strong><a href="../../../administration/managing-clusters#adding-a-compute-cluster"> Compute cluster</a></p>                                                                                                                                                                                                                                                                                                                                                                  |          |
| destinationStorage       | Target Storage                | String                   | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| softRetention            | Soft Retention                | Boolean                  | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| compression              | Compression                   | Compression              | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| interval                 | Interval                      | Int (Minutes)            | The sliding interval to wait for data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |          |
| prefix                   | Folder                        | String                   | If the data resides in a sub folder within the defined cloud storage, specify this folder.                                                                                                                                                                                                                                                                                                                                                                                                                                                        | +        |
| initialLoadConfiguration | Initial Load Configuration    | InitialLoadConfiguration | If you have initial data, enter in a prefix and regex pattern to list the relevant data and select the required files.                                                                                                                                                                                                                                                                                                                                                                                                                            | +        |
| startExecutionFrom       | Start Ingestion From          | String (ISO-8601)        | <p>The time from which to ingest the data from.</p><p>Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.</p>                                                                                                                                                                                                                                                                                                                                                     | +        |
| retention                | Retention                     | Int (Minutes)            | A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                  | +        |
| dataDumpDate             | Data Dump Date                | String (ISO-8601)        | The date that the data starts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | +        |
| maxDelay                 | Max Delay                     | Int (Minutes)            | The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.                                                                                                                                                                                                                                                                                                                                                                                                                        | +        |
| workspaces               | Workspaces                    | String\[]                | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "AzureBlobStorageInputRequest",
	"displayData" : {
		"name" : "First Azure Blob Storage Data 
Source",
		"description" : "Description of first Azure 
Blob Storage data source"
	},
	"sourceStorage" : "2da1c848-9c44-4b4e-a226-4ffebf0d9c49",
	"datePattern" : "yyyy/MM/dd/HH/mm",
	"fileMatchPattern" : {
		"clazz" : "AllMatcher"
	},
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "1bce245a-0c76-4fe7-acb5-6f2bf4b64d8f",
	"destinationStorage" : "004974e0-e384-466e-8ac4-7429c30614e3",
	"softRetention" : true,
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Google Cloud Storage

Connect to your Google Storage Bucket.&#x20;

In order for Upsolver to read events directly from your cloud strage, files should be partitioned by date and time (which defines the folder structure in the cloud storage)

{% hint style="info" %}
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage.\
**See:** [Google Storage connection](https://upsolver.gitbook.io/upsolver-1/administration/connections/google-storage-connection)
{% endhint %}

#### Fields

| Field                    | Name                       | Type                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ------------------------ | -------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name         | Name                       | String                   | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description  | Description                | String                   | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| sourceStorage            | Google Storage Connection  | String                   | The cloud storage to ingest files from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |          |
| datePattern              | Date Pattern               | String                   | The date pattern in the file name/folder structure (e.g. `yyyy/MM/dd/HH/mm`). The date format specification must be set according to [Java DateTimeFormatter format](https://upsolver.gitbook.io/upsolver-1/getting-started/glossary/language-guide/functions/calculated-functions/external-apis-functions).                                                                                                                                                                                                                                      |          |
| fileMatchPattern         | File Name Pattern          | FileNameMatcher          | The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from storage source specified.                                                                                                                                                                                                                                                                                                                                    |          |
| contentType              | Content Format             | ContentType              | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment       | Compute Cluster            | String                   | <p>The compute cluster to run the calculation on. <br><strong>See:</strong> <a href="../../administration/managing-clusters/cluster-types/adding-a-compute-cluster">Compute cluster</a></p>                                                                                                                                                                                                                                                                                                                                                       |          |
| destinationStorage       | Target Storage             | String                   | The data and metadata files for this data source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| softRetention            | Soft Retention             | Boolean                  | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| compression              | Compression                | Compression              | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| interval                 | Interval                   | Int (Minutes)            | The sliding interval to wait for data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |          |
| prefix                   | Folder                     | String                   | If the data resides in a sub folder within the defined cloud storage, specify this folder.                                                                                                                                                                                                                                                                                                                                                                                                                                                        | +        |
| initialLoadConfiguration | Initial Load Configuration | InitialLoadConfiguration | If you have initial data, enter in a prefix and regex pattern to list the relevant data and select the required files.                                                                                                                                                                                                                                                                                                                                                                                                                            |          |
| startExecutionFrom       | Start Ingestion From       | String (ISO-8601)        | <p>The time from which to ingest the data from.</p><p>Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.</p>                                                                                                                                                                                                                                                                                                                                                     | +        |
| retention                | Retention                  | Int (Minutes)            | A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                  | +        |
| dataDumpDate             | Data Dump Date             | String (ISO-8601)        | The date that the data starts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | +        |
| maxDelay                 | Max Delay                  | Int (Minutes)            | The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.                                                                                                                                                                                                                                                                                                                                                                                                                        | +        |
| workspaces               | Workspaces                 | String\[]                | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "GoogleCloudStorageInputRequest",
	"displayData" : {
		"name" : "First Google Cloud Storage 
Data Source",
		"description" : "Description of first Google 
Cloud Storage data source"
	},
	"sourceStorage" : "0a617e70-c2b2-4870-a908-f4c864c4961f",
	"datePattern" : "yyyy/MM/dd/HH/mm",
	"fileMatchPattern" : {
		"clazz" : "AllMatcher"
	},
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "d4f6ee48-0782-4789-b661-4e19679a541a",
	"destinationStorage" : "8ddd7397-e1ed-4b21-a9cb-91aa923ea165",
	"softRetention" : true,
	"compression" : {
		"clazz" : "AutoDetectCompression"
	},
	"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"
```

## Kinesis-backed HTTP

Connect your stream using HTTP requests from any source.

Once you create the connection, you will be provided with an HTTP endpoint. Upsolver receives the data as POST, with the data in the body, and the data is stored in a Kinesis Stream until processed by Upsolver.

Headers sent with the request are also ingested as part of the stream, so metadata can be added to the request header.

#### Fields

| Field                   | Name                 | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name                 | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description          | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| contentType             | Content Format       | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment      | Compute Cluster      | String            | <p>The compute cluster to run the calculation on. <br><strong>See:</strong><a href="../../../administration/managing-clusters#adding-a-compute-cluster"> Compute cluster</a></p>                                                                                                                                                                                                                                                                                                                                                                  |          |
| ingestionEnvironment    | Ingestion Cluster    | String            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |          |
| storageConnection       | Target Storage       | String            | The data and metadata files for this Data Source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| kinesisConnection       | Kinesis Connection   | String            | The data and metadata files for this Data Source will be stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          |
| softRetention           | Soft Retention       | Boolean           | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| retention               | Retention            | Int (Minutes)     | A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                  | +        |
| startExecutionFrom      | Start Ingestion From | String (ISO-8601) | <p>The time from which to ingest the data from.</p><p>Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.</p>                                                                                                                                                                                                                                                                                                                                                     | +        |
| endExecutionAt          | End Read At          | String (ISO-8601) | If configured, stop reading after this date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |
| workspaces              | Workspaces           | String\[]         | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "KinesisHttpInputRequest",
	"displayData" : {
		"name" : "First Kinesis backed HTTP Data 
Source",
		"description" : "Description of first Kinesis 
backed HTTP data source"
	},
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "bdb99c82-4874-4e41-b735-090cc53af71e",
	"ingestionEnvironment" : "c71ba158-8022-494e-84e1-e0623533ef1c",
	"storageConnection" : "f4376c08-1b13-495b-904f-768effa818da",
	"kinesisConnection" : "44a93a4b-5083-4f26-90cb-c3ea7284e3d3",
	"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
```

## HTTP

Connect your stream using HTTP requests from any source.

Once you create the connection, you will be provided with an HTTP endpoint. Upsolver receives the data as POST, with the data in the body.

Headers sent with the request are also ingested as part of the stream, so metadata can be added to the request header.

#### Fields

| Field                   | Name            | Type              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
| ----------------------- | --------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| displayData.name        | Name            | String            | The data source name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |          |
| displayData.description | Description     | String            | The data source description.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| contentType             | Content Format  | ContentType       | <p>The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.</p><p>For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.</p><p>Messages can be compressed, Upsolver automatically detects the compression type.</p><p>Supported compression types are: Zip, GZip, Snappy and None.</p><p><strong>See:</strong> <a href="content-formats">Content formats</a></p> |          |
| computeEnvironment      | Compute Cluster | String            | <p>The compute cluster to run the calculation on. <br><strong>See:</strong> <a href="../../../administration/managing-clusters#adding-a-compute-cluster">Compute cluster</a></p>                                                                                                                                                                                                                                                                                                                                                                  |          |
| connectionPointer       | Target Storage  | String            | The data and metadata files for this Data Source are stored in this storage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |          |
| softRetention           | Soft Retention  | Boolean           | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.                                                                                                                                                                                                                                                                                                                                                              |          |
| shards                  | Shards          | Int               | How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.                                                                                                                                                                                                                                                                                                                                                                                                    |          |
| retention               | Retention       | Int (Minutes)     | A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.                                                                                                                                                                                                                                                                                                                                                                                                                                  | +        |
| endExecutionAt          | End Read At     | String (ISO-8601) | If configured, stop reading after this date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |
| workspaces              | Workspaces      | String\[]         | The workspaces attached to this data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | +        |

#### Example

```bash
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "KinesisHttpInputRequest",
	"displayData" : {
		"name" : "First Kinesis backed HTTP Data 
Source",
		"description" : "Description of first 
Kinesis backed HTTP data source"
	},
	"contentType" : {
		"type" : "JsonContentType"
	},
	"computeEnvironment" : "bdb99c82-4874-4e41-b735-090cc53af71e",
	"ingestionEnvironment" : "c71ba158-8022-494e-84e1-e0623533ef1c",
	"storageConnection" : "f4376c08-1b13-495b-904f-768effa818da",
	"kinesisConnection" : "44a93a4b-5083-4f26-90cb-c3ea7284e3d3",
	"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
```
