> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/release-notes/earlier-releases/2024/july-2024.md). # July 2024 {% hint style="info" %} **Release Notes Blog** For more detailed information on these updates, check out the [Upsolver August 2024 Feature Summary](https://www.upsolver.com/blog/upsolver-august-2024-feature-summary) blog. {% endhint %} ### 2024.07.30-06.58 ⬆️ **Enhancements** * Reduced overhead of task discovery, especially in compute clusters with a lot of assigned entities or large number of shards * Replication Jobs: Added support for configuring the primary key column name for each replication group using the new [`PRIMARY_KEY_COLUMN`](https://docs.upsolver.com/content/v/reference-1/sql-commands/jobs/create-job/replication#replication-group-options) property * Cluster: Allow creating a cluster with a `STARTUP_SCRIPT` and altering cluster's `STARTUP_SCRIPT` 🔧 **Bug Fixes** * Jobs reading data from Hive tables could lose data due to committing compacted files in the source table with the wrong timing * Fixed a rare bug where retention failed to delete some files in old partitions * Iceberg: using a column transformation that overrides a field, which appears with different cases in the data, resulted in multiple fields of different types * Fixed a bug that caused [job statistics](/content/reference-1/monitoring/job-status/stream-and-file-sources/monitoring.md) not to be displayed for jobs writing to Iceberg with intervals longer than 1 minute ### 2024.07.29-08.06 ⬆️ **Enhancements** * Job Monitoring: * The job [Monitoring](/content/reference-1/monitoring/job-status/stream-and-file-sources/monitoring.md) page has been redesigned, allowing you to track job statuses by executions, representing each data interval being processed * Apache Iceberg: * Iceberg Tables: * Users can now set table retention based on any column of types `DATE`, `TIMESTAMP`, `TIMESTAMPZ`, `LONG`, or `INTEGER` * \[Breaking change] Retention configuration syntax properties have changed: TABLE\_DATA\_RETENTION -> RETENTION\_DURATION RETENTION\_DATE\_PARTITION -> RETENTION\_COLUMN * Alter Iceberg Tables: * [Sorting evolution](/content/reference-1/sql-commands/iceberg-tables/upsolver-managed-tables/alter-iceberg-table.md#sorting-evolution) is now supported for Iceberg tables. This allows users to dynamically change the sorting columns of an existing table. This will affect the data written from now on. New compactions will write data using the new sorting. * [Partition evolution](/content/reference-1/sql-commands/iceberg-tables/upsolver-managed-tables/alter-iceberg-table.md#partition-evolution) is now supported for Iceberg tables. This allows users to change the partition columns of an existing table dynamically. This change does not rebuild the existing data according to the new partitioning but will apply the new partitioning to data moving forward. * Performance improvements for high-loaded clusters * Reduced the number of requests to the catalog to prevent rate-exceeded errors when managing a large number of tables 🔧 **Bug Fixes** * Apache Iceberg: * Dangling files: Moved backup folder location into table root directory * Amazon Redshift and Snowflake jobs: * Fixed a bug in the schema evolution (`SELECT *`) that caused redundant columns to be created when mapping fields to different types, such as mapping a field of type `Timestamp` to a column of type `TimestampTZ` * CDC to Amazon Redshift: * Fixed a bug that caused the creation of columns with incorrect types * Fixed a bug that caused internal files to be deleted before their usage, resulting in job stalling ### 2024.07.11-12.30 ⬆️ **Enhancements** * Apache Iceberg: * Users can now [create Iceberg REST catalog connections](/content/reference-1/sql-commands/connections/create-connection/polaris-catalog.md) and add tables * [Dynatrace](/content/reference-1/sql-commands/jobs/create-job/monitoring/dynatrace.md) is now supported as a monitoring output 🔧 **Bug Fixes** * Fixed a bug that caused an additional column to be created when writing a field of type `Timestamp` to a column of type `TimestampTZ` in Redshift * Minor bug fixes ### 2024.07.08-11.10 ⬆️ **Enhancements** * Apache Iceberg: * [Table Mirror](/content/reference-1/sql-commands/iceberg-tables/mirror-iceberg-tables.md) - Users can now define or alter the mirror table refresh interval using `MIRROR_INTERVAL` * `ALTER REPLICATION JOB` supports `RESNAPSHOT COLLECTION` for [MongoDB](/content/reference-1/sql-commands/jobs/create-job/replication/mongodb.md) tables 🔧 **Bug Fixes** * Jobs ingesting from [Amazon S3](/content/reference-1/sql-commands/jobs/create-job/ingestion/amazon-s3.md) cannot include `START_FROM` unless `DATE_PATTERN` is specified * Fixed a bug that prevented excluding system columns using the [`EXCLUDE_COLUMNS`](broken://spaces/WKMq8oT1OPM3KjP8vlg2/pages/exS6qOYXal4UTv9QNTfP#exclude_columns) property in ingestion jobs * Fixed a bug that prevented the use of system columns in column transformations in [ingestion](/content/reference-1/sql-commands/jobs/create-job/ingestion.md) jobs * Fixed a bug occurring when the Primary Key is of numeric type and the table is large enough to require chunk splitting * Fixed a bug where it was possible to schedule compactions for partitions in tables without a primary key, even after the retention period, leading to errors --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://upsolver.gitbook.io/content/release-notes/earlier-releases/2024/july-2024.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.