> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/reference-1/monitoring/system-catalog/insights.md).

# Insights

### Overview

The `system.insights` schema provides a collection of system tables that offer deep insights into various aspects of your Upsolver environment. These tables are designed to assist data engineers, data scientists, and administrators in tasks ranging from query optimization, job monitoring, data profiling, to compliance checks. The tables under this schema are generated and updated in real-time by Upsolver's stream processing engine.

### Key Tables

Here are two of the critical tables within the `system.insights` schema:

#### `job_output_column_stats`

* **Purpose**: Provides statistics about columns that are part of the output of Upsolver jobs.
* **Common Use Cases**: Job monitoring, query optimization on job outputs.

Learn more about [`job_output_column_stats`](/content/reference-1/monitoring/system-catalog/insights/job_output_column_stats.md).

#### `dataset_column_stats`

* **Purpose**: Offers detailed statistics about columns within datasets.
* **Common Use Cases**: Data profiling, query optimization, real-time analytics.

Learn more about [`dataset_column_stats`](/content/reference-1/monitoring/system-catalog/insights/dataset_column_stats.md).

### Features and Use Cases

1. **Query Optimization**: By understanding data density, value distributions, and other statistics, users can optimize queries for performance.
2. **Data Profiling**: Before diving into analytics, users can profile their data to understand characteristics like value ranges, uniqueness, and more.
3. **Job Monitoring**: Gain insights into the behavior and output of Upsolver jobs, helping with debugging and quality assurance.
4. **Real-Time Analytics**: Utilize real-time statistics for generating real-time analytics dashboards.
5. **Audit and Compliance**: Tables such as `job_output_column_stats` offer `first_seen` and `last_seen` timestamps, useful for auditing and compliance checks.
6. **Alerting on Schema Changes**: Detect changes in data density, distinct values, or new columns to trigger alerts for schema modifications.

### Accessing Tables in `system.insights`

To query a table within the `system.insights` schema, you can use the following SQL syntax, replacing `<table_name>` with the name of the table you want to query:

```sql
SELECT * FROM system.insights.<table_name>;
```

### Notes

* These tables are read-only and are updated in real-time.
* Deleted data does **not** affect the statistics; the statistics only reflect data written during the stream processing.
* The tables can be queried like any other Upsolver tables, but they cannot be modified or deleted.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://upsolver.gitbook.io/content/reference-1/monitoring/system-catalog/insights.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.