> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/articles-1/data/system-columns.md).

# System Columns

When you ingest your data into Upsolver, your table is enriched with various system columns that are generated by Upsolver based on your data.

The specific columns and what they describe are detailed below:

* [General](#id-36lbpnq33cpf)
* [Storage](#id-1vywvyyfl0kx)
* [Streaming](#cisxx5240inu)
  * [Amazon Kinesis](#id-3qgezuybg4o6)
  * [Apache Kafka](#wyv866tn1qhp)
* [Databases (CDC)](#databases-cdc)
  * [Microsoft SQL Server](#microsoft-sql-server)
  * [MongoDB](#mongodb)
  * [MySQL](#mysql)
  * [PostgreSQL](#postgresql)

## General <a href="#id-36lbpnq33cpf" id="id-36lbpnq33cpf"></a>

These columns are applicable to all staging tables:

<table><thead><tr><th width="221.33333333333331">Column</th><th width="164">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$commit_time</td><td>timestamp</td><td><p>The time at which the record was committed to the current table. </p><p>When reading historical data, this is the current time at which we're reading the data. </p><p>When reading the latest data written to the source, this is the same as the source time.</p></td></tr><tr><td>$compression</td><td>string</td><td>The compression type, if applicable.</td></tr><tr><td>$event_date</td><td>date</td><td><p>The date at which the event was fired or recognized in the data source. </p><p>This is typically used as a date partition column for your staging table.</p></td></tr><tr><td>$event_timestamp</td><td>instant</td><td>The time at which the event was fired or recognized in the data source.</td></tr></tbody></table>

## Storage <a href="#id-1vywvyyfl0kx" id="id-1vywvyyfl0kx"></a>

These columns are applicable to staging tables with data ingested from storage connections, for example Amazon S3:

<table><thead><tr><th width="240">Column</th><th width="120.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$compressed_file_name</td><td>string</td><td>The name of the file that was read, if it was compressed.</td></tr><tr><td>$column_count</td><td>long</td><td>The number of columns read for CSV files.</td></tr><tr><td>$file_last_modified</td><td>long</td><td>The time the file was last modified.</td></tr><tr><td>$file_name</td><td>string</td><td>The name of the file that was read.</td></tr><tr><td>$file_size</td><td>long</td><td>The size of the file that was read in bytes.</td></tr><tr><td>$row_number</td><td>long</td><td>The index within the file corresponding to each record.</td></tr></tbody></table>

## Streaming <a href="#cisxx5240inu" id="cisxx5240inu"></a>

These columns are applicable to staging tables with data ingested from streaming sources:

<table><thead><tr><th width="237">Column</th><th width="176.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$partition</td><td>long</td><td>The partition this data was read from.</td></tr></tbody></table>

### Amazon Kinesis <a href="#id-3qgezuybg4o6" id="id-3qgezuybg4o6"></a>

These columns are applicable to staging tables with data ingested from Kinesis:

<table><thead><tr><th width="203.33333333333331">Column</th><th width="118">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$kinesis_timestamp</td><td>long</td><td>The time at which the event was fired or recognized in Kinesis.</td></tr><tr><td>$offset</td><td>long</td><td>The offset of the stream that the data is read from.</td></tr><tr><td>$shard</td><td>string</td><td>The ID of the shard the data was read from.</td></tr><tr><td>$stream</td><td>string</td><td>The name of the stream the data was read from.</td></tr></tbody></table>

### Apache Kafka <a href="#wyv866tn1qhp" id="wyv866tn1qhp"></a>

These columns are applicable to staging tables with data ingested from Kafka:

<table><thead><tr><th width="212">Column</th><th width="139.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$kafka_timestamp</td><td>long</td><td>The time at which the event was fired or recognized in Kafka.</td></tr><tr><td>$row_number</td><td>long</td><td>The index corresponding to each record.</td></tr><tr><td>$topic</td><td>string</td><td>The name of the topic the data was read from.</td></tr></tbody></table>

## Databases (CDC)

These columns apply to staging tables with data ingested from database (CDC) sources:

<table><thead><tr><th width="225">Column</th><th width="140.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$binlog_timestamp</td><td>timestamp</td><td>Timestamp for when the event was processed by the connector.</td></tr><tr><td>$database_name</td><td>string</td><td>The name of the source database.</td></tr><tr><td>$debezium_version</td><td>string</td><td>The version number and name of the Debezium engine.</td></tr><tr><td>$from_snapshot</td><td>Boolean</td><td>Whether the row was created from a snapshot or a change event. Values are:<br><strong>True</strong>: snapshot<br><strong>False</strong>: from latest change event</td></tr><tr><td>$is_delete</td><td>Boolean</td><td>Boolean representation for <code>$operation</code> = 'delete'. Values are:<br><strong>True</strong>: The row has been deleted<br><strong>False</strong>: The row has not been deleted.</td></tr><tr><td>$old_row</td><td>string</td><td>Optional. Represents the row prior the captured change.</td></tr><tr><td>$operation</td><td>string</td><td><p>Row operation. Values are: <strong>create</strong>, <strong>update</strong>, <strong>delete</strong>, or <strong>snapshot</strong>. </p><p><strong>TRUNCATE</strong> and <strong>MESSAGE</strong> operations are not currently supported and will result in an error.</p></td></tr><tr><td>$primary_key</td><td>string</td><td>If a source table includes one or more primary keys, this field holds a concatenation, separated by <code>~</code>, of all primary key values. </td></tr><tr><td>$table_name</td><td>string</td><td>The name of the source table.</td></tr></tbody></table>

### Microsoft SQL Server

These columns apply to staging tables with data ingested from Microsoft SQL Server:

| Column               | Data Type | Description |
| -------------------- | --------- | ----------- |
| $change\_lsn         |           |             |
| $commit\_lsn         |           |             |
| $commit\_time        |           |             |
| $event\_serial\_time |           |             |
| $full\_table\_name   |           |             |
| $item\_index         |           |             |
| $row\_number         |           |             |
| $schema\_name        |           |             |
| $shard\_number       |           |             |
| $source\_id          |           |             |
| $source\_name        |           |             |

### MongoDB

These columns apply to staging tables with data ingested from MongoDB:

| Column                  | Data Type | Description |
| ----------------------- | --------- | ----------- |
| $collection\_name       | string    |             |
| $commit\_time           | string    |             |
| $full\_collection\_name | string    |             |
| $replica\_set           | string    |             |
| $row\_number            | long      |             |
| $shard\_number          | long      |             |
| $source\_id             | string    |             |
| $source\_time           | string    |             |
| $transaction\_number    | long      |             |

### MySQL

These columns apply to staging tables with data ingested from MySQL:

<table><thead><tr><th width="234">Column</th><th width="128.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$binlog_file_name</td><td>string</td><td>Binlog name where the event was recorded.</td></tr><tr><td>$binlog_file_position</td><td>string</td><td>Position within the binlog.</td></tr><tr><td>$binlog_row</td><td>string</td><td>Row within the change event.</td></tr><tr><td>$full_table_name</td><td>string</td><td>Concatenation of <code>$database_name</code> and <code>$table_name</code> separated by a period <code>.</code>.</td></tr><tr><td>$gtid</td><td>string</td><td>If query logging is enabled, represents Global Transaction ID. <a href="https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-mysql-gtids">Learn more</a>.</td></tr><tr><td>$query</td><td>string</td><td>Optional. Original SQL query that produced the change event. <a href="https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-query-log-events">Instructions</a> to enable query logging.</td></tr></tbody></table>

### PostgreSQL

These columns apply to staging tables with data ingested from PostgreSQL:

<table><thead><tr><th width="207">Column</th><th width="145.33333333333331">Data type</th><th>Description</th></tr></thead><tbody><tr><td>$full_table_name</td><td>string</td><td>Concatenation of <code>$database_name</code>, <code>$schema_name</code>, and <code>$table_name</code> separated by a period ".".</td></tr><tr><td>$is_hearbeat</td><td>Boolean</td><td>True if the event originates from a <a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-wal-disk-space">heartbeat table</a>. False if it originates from a tracked table.</td></tr><tr><td>$lsn</td><td>string</td><td>Log Sequence Number (LSN) represents location in the transaction log where the specific change event was recorded.</td></tr><tr><td>$schema_name</td><td>string</td><td>The name of the source schema.</td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://upsolver.gitbook.io/content/articles-1/data/system-columns.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.