# Create an Amazon S3 data source

{% embed url="<https://www.youtube.com/watch?v=eCeyEETvV4g&feature=emb_title>" %}

{% hint style="info" %}
If you haven’t signed up for Upsolver yet, follow this guide to [deploy Upsolver](https://upsolver.gitbook.io/upsolver-1/getting-started/start-using-upsolver/upsolver-deployment-guide).&#x20;
{% endhint %}

Upsolver supports ingesting all data formats from various data source types including but not limited to: &#x20;

* [Amazon S3](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/amazon-aws-data-sources/amazon-s3-data-source)
* [Amazon Kinesis Stream](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/amazon-aws-data-sources/amazon-kinesis-stream-data-source)
* [Kafka](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/kafka-data-output)
* [Amazon S3 over SQS](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/amazon-aws-data-sources/amazon-s3-over-sqs-data-source)
* [Microsoft Azure Blob storage](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/microsoft-azure-data-sources/microsoft-azure-blob-storage-data-sou)
* [Google Cloud Storage](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/google-cloud-storage-data-source)
* [File upload ](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/additional-data-sources)
* [JDBC](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/jdbc)
* [Microsoft Azure Events Hub](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/microsoft-azure-data-sources/microsoft-azure-events-hub-data-source)
* [HDFS](https://upsolver.gitbook.io/upsolver-1/connecting-data-sources/hdfs-data-source)

Upsolver can read events directly from your Amazon S3 bucket where the data should be partitioned by event date.&#x20;

This guide is using COVID-19 data from Amazon AWS, but you can also follow along with your own dataset.

## Create an S3 bucket with sample data

1\. Download a COVID-19 test data set from [here](https://drive.google.com/file/d/1YsM5XtlYhiqFV3BND2pN3A9UcxNXe42u/view?usp=sharing) (July 1, 2020). You can subscribe to [Amazon AWS COVID-19 testing data](https://aws.amazon.com/marketplace/pp/Coronavirus-Disease-COVID-19-Testing-Data-The-COVI/prodview-a2ev4blctqkwc) for the most up-to-date data sets, or you can use your own data set.

2\. Create a S3 bucket in **your own AWS account** with data partitioned by date. Since the data set being used here is from July 1, 2020, put the CSV file under `2020/07/01`.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGUK1fJQxJwm91bi2ra%2F-MGUNBpsjPHjuxej-19r%2FScreen%20Shot%202020-09-05%20at%2010.45.36%20AM.png?alt=media\&token=12f89561-e77d-4b91-aded-06119dbd923c)

## Create a new Amazon AWS S3 data source

1\. From your **Upsolver UI**, click on **New Data Source**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGUK1fJQxJwm91bi2ra%2F-MGUNnYH6DWMp0Es-ztS%2FScreen%20Shot%202020-09-05%20at%2010.47.25%20AM.png?alt=media\&token=f95d9db0-96a1-4995-a716-c59aa2d89356)

2\. Select **Amazon S3** as your data source type.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGUK1fJQxJwm91bi2ra%2F-MGUO6ANqlg7JXjWvSE5%2FScreen%20Shot%202020-09-05%20at%2010.49.46%20AM.png?alt=media\&token=8fce1fea-8401-4482-b5d0-d8a6ce1a0025)

3\. Since Upsolver is integrated with your S3 storage, your buckets are automatically detected. Select the **bucket** in which your data is located.

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.11.31.png)

4\.  The **Date Pattern** should match how S3 bucket data is partitioned from step 2.

{% hint style="info" %}
**Example:** For `2020/07/01`, the **Date Pattern** should be `yyyy/MM/dd/`.
{% endhint %}

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.11.43.png)

5\. Choose CSV as the **Content Format**.&#x20;

{% hint style="info" %}
Keep in mind that Upsolver supports all file formats and that different data types provide various options to parse the data. CSV is just being used as an example here.\
**See:** [Data formats](https://upsolver.gitbook.io/upsolver-1/getting-started/glossary/data-formats)
{% endhint %}

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.11.59.png)

For CSV files, you should check **Infer Types** if you wish to auto-detect data types; otherwise, Upsolver will take in each field as a **string**.&#x20;

You can also optionally define the headers of the columns you want to load under **Header** and define your own delimiter under **Delimiter**.

6\. Click **Continue**.&#x20;

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.19.23.png)

7\. Click **Launch Integration** to integrate your S3 bucket with Upsolver; a CloudFormation page will open up in a new tab.

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.19.30.png)

8\. Scroll down the CloudFormation page and check the **I acknowledg**e statement. Then click on **Create stack**.

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.19.37.png)

9\. Within a minute or so, the stack should be fully created, indicated by its status which should now be `CREATE_COMPLETE`.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkMwMrCwPhEM5lHLZk%2Fimage.png?alt=media\&token=7d50839f-1dee-4de5-800b-19ec0a4e425e)

10\. Go back to the Upsolver UI and click on **Done**.&#x20;

![](https://www.upsolver.com/wp-content/uploads/2020/07/Screen-Shot-2020-07-07-at-14.19.53.png)

11\. You will be shown a sample of the parsed data. If everything looks OK, click **Create**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkN1QDTgG7sw2WFPLf%2Fimage.png?alt=media\&token=50e6cfe9-32fa-43fe-affa-0f47de8faaed)

{% hint style="success" %}
Congratulations! You have successfully created an Amazon S3 data source. You can click on the **Parse Errors** tab to ensure everything was parsed properly.
{% endhint %}

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkN6muUH_EGItpWK-g%2Fimage.png?alt=media\&token=fac58a30-f859-4fdb-848c-df57d2f5fbcb)
