> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/reference-1/functions-and-operators/functions/filter/is_duplicate.md).
# IS\_DUPLICATE
The `IS_DUPLICATE` function returns **TRUE** if it's not the first time a job sees the input `value` in the data within the interval specified in the `windowSize`.
The `windowSize` interval must be equal to or greater than the `RUN_INTERVAL` of the job, and be evenly divisible by the `RUN_INTERVAL` value. For example, if the `RUN_INTERVAL` is **5 minutes**, the `windowSize` can be set to **5 minutes**, **10 minutes**, **15 minutes**, and so on.
If you want to check for duplicates only within the data processed by the current job execution, set the interval value to `RUN_INTERVAL`.
The `value` being checked for duplication can be any data type.
## Syntax
```sql
IS_DUPLICATE(windowSize, value)
```
## Arguments
Name
Type
Description
windowSize
interval
The deduplication window size in minutes, hours, or the RUN_INTERVAL of the job.
value
any
Value to check for duplicates. This can be any data type.
## Returns
Boolean
***
## Example
### Create a job and check for duplicates
The following example creates a job named **transform\_orders\_and\_check\_duplicates**, which copies new events from the **default\_glue\_catalog.upsolver\_samples.orders\_raw\_data** staging table into the **default\_glue\_catalog.upsolver\_samples.orders\_is\_duplicate** table. The `RUN_INTERVAL` job option instructs Upsolver to run the job every minute:
```sql
CREATE SYNC JOB transform_orders_and_check_duplicates
START_FROM = BEGINNING
ADD_MISSING_COLUMNS = true
RUN_INTERVAL = 1 MINUTE
AS INSERT INTO default_glue_catalog.upsolver_samples.orders_is_duplicate
MAP_COLUMNS_BY_NAME
SELECT
orderid AS order_id,
MD5(customer.email) AS customer_id,
customer_name,
nettotal AS total,
IS_DUPLICATE(INTERVAL '1' HOUR, orderid) AS is_unique,
$event_time AS partition_date
FROM default_glue_catalog.upsolver_samples.orders_raw_data
-- create a computed column
LET customer_name = customer.firstname || ' ' || customer.lastname
WHERE ordertype = 'SHIPPING'
AND time_filter();
```
The `SELECT` statement defines the list of columns that will be loaded into the target table. The list includes the `IS_DUPLICATE` function to check for rows with a duplicate **orderid**. The `INTERVAL` has been set to **1 HOUR**, so any duplicate events that land within an hour of the first row arriving will return **TRUE**:
```sql
IS_DUPLICATE(INTERVAL '1' HOUR, orderid) AS is_unique
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://upsolver.gitbook.io/content/reference-1/functions-and-operators/functions/filter/is_duplicate.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.