Modify a data source

This article provides a guide on how to modify data sources using an API call.

This API enables you to modify an existing data source. All API calls require an API token.

PATCH https://api.upsolver.com/inputs/DATA-SOURCE-ID

Toggle soft retention

Fields

Field	Name	Type	Description	Optional
softRetention	Soft Retention	Boolean	This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "ToggleSoftRetention",
	"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update retention

Fields

Field	Name	Type	Description	Optional
retention	Retention	Int (Minutes)	The retention period for the data in Upsolver.	+

Example

curl -X POST -H "content-type: application/json" -H "Authorization: 
YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateRetention",
	"retention" : 1440
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update end execution at

Fields

Field	Name	Type	Description	Optional
endExecutionAt	End Read At	String (ISO-8601)	Stop reading after this date.	+

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateEndExecutionAt",
	"endExecutionAt" : "2018-11-04T08:00:00Z"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Set ingestion environment

Fields

Field	Name	Type	Description	Optional
environment	Ingestion Cluster	String	The compute cluster to run the calculation on. See: Compute cluster

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
 "clazz" : "SetIngestionEnvironment",
 "environment" : "09267786-8d84-4644-adac-52601b9609bb"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Set compute environment

Fields

Field	Name	Type	Description	Optional
environment	Environment	String	The compute cluster to run the calculation on. See: Compute cluster

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
  "clazz" : "SetComputeEnvironment",
  "environment" : "12ba7222-5764-4d37-8581-95ca5be4b3f7"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Rename

Fields

Field	Name	Type	Description	Optional
name	Name	String	The data source name.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "Rename",
	"name" : "NEW_NAME"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update parallelism

Fields

Field	Name	Type	Description	Optional
parallelism	Parallelism	Int	The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateParallelism",
	"parallelism" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update shard parallelism

Fields

Field	Name	Type	Description	Optional
parallelism	Parallelism	Int	The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateShardParallelism",
	"shards" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update content type

Fields

Field	Name	Type	Description	Optional
contentType	Content Type	ContentType	The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. See: Content formats

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateContentType",
	"contentType" : {
		"type" : "JsonContentType"
	}
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update description

Fields

Field	Name	Type	Description	Optional
description	Description	String	The description of the data source.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateDescription",
	"description" : "description"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Attach workspace

Fields

Field	Name	Type	Description	Optional
workspace	Workspace	String	The workspace to attach to this data source.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "AttachWorkspace",
	"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Detach workspace

Fields

Field	Name	Type	Description	Optional
workspace	Workspace	String	The workspace to detach the data source from.

Example

curl -X POST -H "content-type: application/json" 
-H "Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "DetachWorkspace",
	"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Attach many workspaces

Fields

Field	Name	Type	Description	Optional
workspaces	Workspaces	String[]	The workspaces to attach this data source to.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "AttachManyWorkspaces",
	"workspaces" : "workspaces"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update connection

Fields

Field	Name	Type	Description	Optional
connection	Connection	String	Updating the storage location will cause new objects to be written to the new location.
moveData	Move existing objects	Boolean	Copy existing objects to the new storage location and then delete from the original location. Dependencies will continue working normally during this process.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateConnection",
	"connection" : "00dbf0e2-7c11-47f8-a672-
cacb537af190",
	"moveData" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Stop

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "Stop"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Update reporting tags

Fields

Field	Name	Type	Description	Optional
reportingTags	Reporting Tags	String	Update the tags that can be sent to external monitoring systems.

Example

curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateReportingTags",
	"reportingTags" : "reportingTags"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"