> For the complete documentation index, see [llms.txt](https://upsolver.gitbook.io/content/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://upsolver.gitbook.io/content/release-notes/earlier-releases/2024/april-2024.md). # April 2024 {% hint style="info" %} **Release Notes Blog** For more detailed information on these updates, check out the [Upsolver May 2024 Feature Summary](https://www.upsolver.com/blog/upsolver-may-2024-feature-summary) blog. {% endhint %} ## 2024.04.25-12.36 :arrow\_up: **Enhancements** * Iceberg: * Added support for writing to hidden partitions * Enabled changing the partition specification of existing tables even while they are actively being written to by a job * Support writing to External Iceberg tables * Support altering Iceberg table properties via SQL :wrench: **Bug Fixes** * Worksheet tree - Show replication jobs under tables that were created dynamically * MongoDB CDC: * Corrected the parsing of Decimal types to Double * Resolved errors encountered when replicating collections containing fields with types Regex, Min Key, and Max Key *** ## 2024.04.16-12.06 :arrow\_up: **Enhancements** * Introduced the `PARSE_DEBEZIUM_JSON_TYPE` property to the Avro Schema Registry content format for dynamic parsing of JSON columns from Debezium sources into Upsolver records or keeping as JSON strings. For Snowflake outputs with schema evolution, fields are written to columns of type Variant. * Added support for Iceberg table retention using the [TABLE\_DATA\_RETENTION](/content/reference-1/sql-commands/iceberg-tables/upsolver-managed-tables/create-iceberg-table.md#table_data_retention-editable) property * Upgraded the Snowflake driver to 3.15.0 * UI: [ClickHouse wizard](/content/quickstarts-1/data-ingestion-wizard/using-the-wizard/target-set-up/clickhouse.md) cosmetic changes :wrench: **Bug Fixes** * Fixed a bug preventing the pausing of ingestion jobs to Snowflake * Iceberg schema evolution: * Nested fields were added without the field docs, which are later used to understand which field evolved from which. Affected tables may need to be recreated if jobs writing to them are causing errors * Was not handling cases where a field can have multiple types (e.g., a field can be a record and can also be an array of strings) *** ## 2024.04.04-09.33 #### :sparkles: New Features * Data lineage diagram is now accessible from [Job Status](/content/reference-1/monitoring/job-status.md), [Datasets](/content/reference-1/monitoring/datasets.md), and materialized view pages. Users can easily view real-time job status and dependencies * Ingestion wizard: * [ClickHouse is now supported as a target](/content/quickstarts-1/data-ingestion-wizard/using-the-wizard/target-set-up/clickhouse.md) (CDC sources are not supported at this point) :arrow\_up: **Enhancements** * For new entities, you can now use the updated Parquet list structure (`parquet.avro.write-old-list-structure = false`) when writing Parquet files to S3 and Upsolver tables * Support casting strings to JSON in jobs writing to Iceberg tables * Previewing Classic Data Sources is now supported (`SELECT * FROM "classic data source name"`) * [COLUMN\_TRANSFORMATIONS](/content/reference-1/sql-commands/jobs/create-job/replication.md#column_transformations) are now supported by replication jobs * Cost reduction: * Reduced S3 API costs of replication jobs and [single entity jobs](/content/how-to-guides-1/jobs/database-replication/replicate-cdc-data-into-snowflake.md) * Reduced S3 API costs of Iceberg tables * Reduced S3 API costs of Hive tables * The [OPTIMIZE](/content/reference-1/sql-commands/iceberg-tables/external-iceberg-tables/create-external-iceberg-table.md#optimize-editable) option for external Iceberg tables now supports optimizing tables that are not partitioned * The cluster system table ([system.monitoring.clusters](/content/reference-1/monitoring/system-catalog/information-schema/clusters.md)) now shows data that is aligned with the Cluster [Monitoring](/content/reference-1/monitoring/system-catalog/information-schema/clusters.md) page :wrench: **Bug Fixes** * Fixed a bug that could skip data when reading from CDC sources * Fixed a bug where events Written graph wouldn't show for [single entity jobs](/content/how-to-guides-1/jobs/database-replication/replicate-cdc-data-into-snowflake.md) that contains a lot of sub jobs or where the job list page contains a lot of jobs * CDC Event log is now deleted right after parsing the log events * Fixed a bug where [replication](/content/reference-1/sql-commands/jobs/create-job/replication.md) and [single entity](/content/how-to-guides-1/jobs/database-replication/replicate-cdc-data-into-snowflake.md) jobs wouldn't work when trying to create a table with a name that existed before * Increased performance of VPC integration experience * Fixed a rare bug where showing "Lifetime" statistics on the [Datasets](/content/reference-1/monitoring/datasets.md) page wouldn't show the lifetime statistics * Fixed a bug where jobs that read data from the [system.information\_schema.columns](/content/reference-1/monitoring/system-catalog/information-schema/columns.md) would timeout when there were tables with a large number of columns * Fixed a bug where it was possible to drop a table that a [replication](/content/reference-1/sql-commands/jobs/create-job/replication.md) or [single entity](/content/how-to-guides-1/jobs/database-replication/replicate-cdc-data-into-snowflake.md) job was writing into. The new behavior now requires that the job is dropped first * Fixed a bug where a [single entity job](/content/how-to-guides-1/jobs/database-replication/replicate-cdc-data-into-snowflake.md) that reads data from a table that is partitioned by time wouldn't read from the start of the table * Fixed a bug where the first point in the [Datasets](/content/reference-1/monitoring/datasets.md) graph would have a timestamp that is before the start time of the first job that writes to a table --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://upsolver.gitbook.io/content/release-notes/earlier-releases/2024/april-2024.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.