> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/reference-1/monitoring/job-status/stream-and-file-sources/monitoring-v1/job-execution-status.md).

# Job Execution Status

### Job executions currently running

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="234">Type</th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Informational</td><td></td></tr><tr><td>About this metric</td><td>The number of currently running job executions.</td><td></td></tr><tr><td>Timeframe</td><td>Now</td><td></td></tr></tbody></table>

#### More information

A job is scheduled to run every defined time interval (i.e every minute, hour, day etc). A job execution is defined as an execution of a specific time interval. As long as the job is up-to-date, a single job execution will be running according to the expected running schedule.&#x20;

In case there's a backlog, multiple job executions can run concurrently. Each execution will handle a different time interval. This can happen, for example, due to historical running of the job or a spike in the amount of data which causes the job execution duration to exceed the time between intervals.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', running_executions) AS     running_executions, 'OK' AS running_executions_severity 
FROM system.monitoring.jobs 
WHERE job_id = '<job_id>';
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Job executions currently in queue

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="248"></th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Warning</td><td></td></tr><tr><td>About this metric</td><td>The number of queued job executions pending.</td><td></td></tr><tr><td>Limits</td><td>Warn when > 0</td><td></td></tr><tr><td>Timeframe</td><td>Now</td><td></td></tr></tbody></table>

#### More information

This metric is the count of runnable jobs not running. The value for an up-to-date job should be 0. If you are performing a replay, this number could be very high, potentially in the thousands. However, as the replay runs, the number should steadily decrease at the rate at which work is being done. If it is not a replay, it can mean that the cluster is not big enough to handle the workload. A value below 10 is considered acceptable, but you may have a problem if it is increasing, and the cluster is not managing to keep up with the workload.

Be aware that an increasing queue will cause latency issues with your data. However, if the number is constant over time and the latency is acceptable, then this may not be an issue for you. If the queue increases, it should be investigated.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', queued_executions) AS queued_executions, IF_ELSE(queued_executions > 0, 'WARNING', 'OK') AS queued_executions_severity 
FROM system.monitoring.jobs 
WHERE job_id = '<job_id>';
```

{% endcode %}
{% endtab %}

{% tab title="Troubleshooting" %}
Consider increasing the cluster's server limit to allow more executions to run in parallel. This will allow the job to remain up-to-date and/or close any pending backlog.
{% endtab %}
{% endtabs %}

### Job executions completed - today

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="234">Type</th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Informational</td><td></td></tr><tr><td>About this metric</td><td>The number of job executions completed today.</td><td></td></tr><tr><td>Timeframe</td><td>Today (midnight UTC to now)</td><td></td></tr></tbody></table>

#### More information

The total number of job executions completed today. A job is scheduled to run every defined time interval (i.e. every minute, hour, day etc). A job execution is defined as an execution of a single time interval.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', completed_executions_today) AS completed_executions_today, 'OK' AS completed_executions_today_severity 
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Job executions completed - lifetime

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="217"></th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Informational</td><td></td></tr><tr><td>About this metric</td><td>The number of job executions completed over the lifetime of the job.</td><td></td></tr><tr><td>Timeframe</td><td>Job lifetime</td><td></td></tr></tbody></table>

#### More information

The total number of job executions completed over the lifetime of the job. A job is scheduled to run every defined time interval (i.e. every minute, hour, day etc). A job execution is defined as an execution of a single time interval.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', total_completed_executions) AS total_completed_executions, 'OK' AS total_completed_executions_severity
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Job executions currently waiting for dependencies

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="217"></th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Informational</td><td></td></tr><tr><td>About this metric</td><td>The number of job executions that are waiting for a dependency to complete.</td><td></td></tr></tbody></table>

#### More information

The number of job executions currently pending upon other jobs to complete their work before this execution can run.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', executions_waiting_for_dependencies) AS executions_waiting_for_dependencies, 'OK' AS executions_waiting_for_dependencies_severity
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Job executions currently retrying after failure

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="217"></th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Warning</td><td></td></tr><tr><td>About this metric</td><td>The number of job executions that encountered an error and are currently retrying.</td><td></td></tr><tr><td>Limits</td><td>Error when > 0</td><td></td></tr></tbody></table>

#### More information

The number of job executions that encountered an error and are currently retrying. Ideally, this should be 0. If a job encounters a transient error, the value will disappear after the retry is successful, otherwise, investigation is required to fix the issue. The retry will continue as long as the issue occurs and will stop only once it is resolved.
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT STRING_FORMAT('{0,number,#,###}', executions_retrying_after_failure) AS executions_retrying_after_failure, IF_ELSE(executions_retrying_after_failure > 0, 'ERROR', 'OK') AS executions_retrying_after_failure_severity 
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';
```

{% endcode %}
{% endtab %}

{% tab title="Troubleshooting" %}
See the`logs.tasks.task_executions` system table for details and error messages.

Run the following query to return tasks that have encountered an error:

{% code overflow="wrap" %}

```sql
SELECT job_id, job_name, stage_name, job_type, 
        task_name, task_start_time, task_end_time, task_error_message
FROM logs.tasks.task_executions 
WHERE task_error_message IS NOT NULL
ORDER BY job_id;
```

{% endcode %}
{% endtab %}
{% endtabs %}

### Execution Failure Reason

{% tabs %}
{% tab title="Metric" %}

<table data-header-hidden><thead><tr><th width="217"></th><th></th><th data-hidden></th></tr></thead><tbody><tr><td>Metric type</td><td>Warning</td><td></td></tr><tr><td>About this metric</td><td>The error message detailing why the job failed.</td><td></td></tr><tr><td>Timeframe</td><td>Now</td><td></td></tr></tbody></table>

#### More information

The error message detailing why the job failed, which will be unique to your job. You can adapt the query under the **See All Events (SQL Syntax)** tab to view the full error for the message relevant to your job.&#x20;
{% endtab %}

{% tab title="See All Events (SQL Syntax)" %}
Run the following SQL command in a query window in Upsolver, replacing **\<job\_id>** with the Id for your job. The Id for your job can be found in the [Details](/content/reference-1/monitoring/job-status/stream-and-file-sources/settings.md#details) section under the Settings tab.

For additional columns, alter this statement and use `SELECT *`.&#x20;

{% code overflow="wrap" %}

```sql
SELECT execution_failure_reason
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';
```

{% endcode %}
{% endtab %}
{% endtabs %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://upsolver.gitbook.io/content/reference-1/monitoring/job-status/stream-and-file-sources/monitoring-v1/job-execution-status.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.