> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/quickstarts-1/data-ingestion-wizard/using-the-wizard/source-set-up/apache-kafka.md).

# Apache Kafka

## Step 1 - Connect to Kafka

### **Create a new connection**

Click **Create a new connection**, if it is not already selected. In **Kafka host\[s]**, enter a single host in the format of **hostname:port**, or a list of Kafka hosts in the format of **hostname1:port, hostname2:port**.

Ensure the host address is accessible to Upsolver.

When using Upsolver's sandbox cloud, Kafka needs public access. If you have already integrated Upsolver into your environment and you want to avoid exposing the Kafka cluster and save AWS transfer costs, you can [peer the VPC where the Kafka cluster runs to the VPC running Upsolver](/content/how-to-guides-1/setup/deploy-upsolver-on-aws/vpc-peering-guide.md).

Additional connection options may require configuration in order to provide the correct credentials to read from your Kafka cluster. 

For a standard connection, in **Consumer Properties**, use the following format:

```
bootstrap.servers = HOST:PORT
security.protocol = SASL_SSL
sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule
     required username = "API_KEY"   password = "SECRET";
ssl.endpoint.identification.algorithm = https
sasl.mechanism = PLAIN
```

where: 

* The the **boostrap.servers** value is same value as **Kafka host\[s]** you entered above.
* The **username** and **password** values are the **API\_KEY**, along with the corresponding secret configured on Kafka to allow access to the Kafka cluster.

To learn more about consumer properties visit [Consumer Configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html).

In order for Upsolver to connect to your Kafka cluster using SSL, follow these steps to [Configure SSL for your Kafka connection](/content/how-to-guides-1/connectors/configure-access/apache-kafka.md).

In the **Name your connection**, type in the name you want to give to this connection.

<figure><img src="/files/Jkey3guXrOdO3O8JPvqY" alt=""><figcaption><p>Create a new connection to Apache Kafka and select a topic to ingest.</p></figcaption></figure>

### Use an existing connection

By default, if you have already created a connection, Upsolver selects **Use an existing connection**, and your Apache Kafka connection is populated in the list.&#x20;

For organizations with multiple connections, select the source connection you want to use.&#x20;

<figure><img src="/files/701573AI6TpuPZ4crTtl" alt=""><figcaption><p>Select your existing Apache Kafka connection to use for your ingestion job. Then select your topic.</p></figcaption></figure>

## Step 2 - Select a topic to ingest

When the connection is established, use the **Select a topic for ingestion** list to choose the topic you want to ingest. Optionally, you can then **Select the source events content type** to change the default from **Automatic**, to a specific file type such as **CSV**, **JSON**, or **Avro**.

## Step 3 - Check that events are read successfully

As soon as you select a topic, Upsolver will attempt to load a sample of the events.&#x20;

If Upsolver did not load any sample events, try the following:

1. Verify that Kafka has events.
2. Select a [content type](#step-2-select-a-topic-to-ingest) that matches the content type of your topic.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://upsolver.gitbook.io/content/quickstarts-1/data-ingestion-wizard/using-the-wizard/source-set-up/apache-kafka.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.