6.12 File Handling in Connectivity
General
Files can be both received as input for connectivity elements as well as returned as output from them.
They can also be passed as parameters from one element to another in a Connector.
Files as input
Files can be sent as input for a Datapoint or a Connector by sending a multipart form data request to invoke the data-orchestrator.
Example Datapoint using a file input:
{
"id": "dp-with-file-input",
"type": "datapoint",
"version": "0.0.1",
"deprecated": false,
"datapoint": {
"type": "http",
"http": {
"url": "https://<myfileservice>/upload",
"method": "POST",
"body": {
"file": "${input.file}",
"otherparam1": "value1",
"otherparam2": "value2"
}
}
}
}
Example Connector using a file input:
{
"id": "connector-with-file-input",
"type": "connector",
"version": "0.0.1",
"deprecated": false,
"connector": {
"elements": [
{
"ref": "firstDp",
"element": "some-dp-with-no-file-input"
},
{
"element": "dp-with-file-input",
"parameters": {
"file": "${root.file}",
"otherparam1": "${ref.firstDp}"
}
}
]
}
}
Example Bruno call:
post {
url: {{envURL}}data-orchestrator/v1/invoke
body: multipartForm
auth: inherit
}
body:multipart-form {
id: dp-with-file-input
file: @file(C:\test.txt)
}
Files as output
Datapoints and Connectors can return file contents - either by converting the file content to string and wrapping it in the poll response JSON or by downloading it directly
The response of those Datapoints and Connectors can be retrieved in two ways:
- as normal text response by calling the data-orchestrator/v1/poll endpoint
- as actual file download with the original mime type by calling the data-orchestrator/v1/result endpoint
For more details on those endpoints see the Technical documentation for OLIVE
File categories
Before 1.3 every uploaded file went into one undifferentiated store. From 1.3 a file belongs to a category, and the category decides who may overwrite it. An upload that passes no category is stored as a temp file.
There are three categories:
| Category | Who may upload | Who may read | Meant for |
|---|---|---|---|
core | admin role | everyone | Files the OLIVE runtime itself needs, or that influence system or client behaviour. Specially protected so they cannot easily be manipulated |
static | editor role | everyone | Files used by a connectivity solution that stay stable — assets, reference data, templates for output |
temp | everyone | everyone | Files created while connectivity runs: caching, intermediate storage, hand-over between elements |
The category is passed with the upload request. Downloads do not take one — the file is found by its key regardless of category, and reading is not role-restricted.
An unrecognised category is rejected rather than falling back to a default.
Which category to use
Two rules cover almost every case:
- A file produced while connectivity runs is a
tempfile — upload it without a key. When no key is passed, the service generates one (a random identifier followed by the file name), and that generated key is what you use afterwards. Choosing your own key for a temp file means someone else can guess it, and because anyone may write totemp, they can also overwrite it. - Anything that is meant to stay put gets an explicit category. A file your solution reads later — an asset, a lookup table, a document template — belongs in
static, and you have to say so.coreis for platform files and needs the admin role; in normal connectivity work you do not upload into it.
temp, and temp is writable by everyoneAn upload without a category is a temp file. If that file matters — if something reads it later and would break or misbehave were its content replaced — then leaving the category out is the wrong default: any user of the tenant may overwrite it at the same key.
Pass the category explicitly for anything except genuinely throwaway runtime data.
Keys and folders
- Passing no key is the recommended form for temp files: the service composes one as
<random-id>_file_<file-name>. - Passing a key uses it verbatim, optionally under a
folderprefix. - A key accepts letters, digits and
- _ . ~ @ /. A folder accepts the same except/, which is the storage hierarchy separator. - Everything else is rejected with
key contains unsupported characters— including spaces. Percent-encoded values are rejected too: pass the decoded string rather than%20. - A key segment is limited to 255 bytes and the whole composed key to 1024 bytes, so keys stay portable across storage backends.
This is the most common upload error, and it appears late — while uploading, not while building:
Uploading (172/254): legal@@01 Impressum@@index.html
Upload failed for legal@@01 Impressum@@index.html: 400 {"error":"key contains unsupported characters"}
Use _ instead of a space in file and folder names. If you are uploading a built artifact — a microfrontend, a documentation bundle — the key is derived from the source path, so a single directory or file with a space in it anywhere in the tree breaks that one upload while the rest succeed.
Uploading a whole directory tree
The file storage has no directories. When a set of files has to keep its structure — the output of a build, for example — the convention is to flatten the path into the key and separate the segments with @@:
dist/legal/impressum/index.html -> legal@@impressum@@index.html
dist/assets/images/diagram.png -> assets@@images@@diagram.png
@ is allowed in a key precisely so that this works, and it is also what lets a versioned artifact name such as olive-dashboard-app@1.1.7 be used as a folder. Whatever produces the keys — the deployment script for a UI component, or your own tooling — every segment of the original path has to satisfy the character rules above.
A development deployment has no authentication and a single user holding every role, so the static and core write restrictions do not apply there. They start applying when the same connectivity runs in production.
It also means an HTTP datapoint calling the public file-storage-service API needs no credentials there. In an authenticated deployment it does — the datapoint has to reference an Auth entry of type oidc, otherwise the upload or download fails with 401. See Calling OLIVE's own APIs from connectivity.
Restrictions & Limits
Restrictions on usage
In the current version file handling in OLIVE has some limitations where and how files can be used:
- Files can not be used as input for or as parameter within workflows
- Files can not be used in or transformed with JSONata expressions
- Files can only be sent as input in multipart form data, direct file streams are not supported
- Files must be delivered with the key "file" in the multipart form data
- Passing multiple files as input is not supported
- There is currently no way to upload files and guarantee that other users can not read it, so sensitive information in files should be treated with care
Limits
The file storage itself is built for large files. There is no small ceiling on what you can store in
the file-storage-service, and storing a large payload there and passing a reference to it is the
recommended way to move bulk data through a connectivity scenario.
What is limited is how much data can travel through an execution, and that limit depends on how the scenario is orchestrated:
| How the scenario runs | What limits the payload |
|---|---|
| A datapoint on its own, or a connector | The HTTP request and response size of the services involved. Comfortably more than a few MB in a normal deployment |
| A workflow | The limits of the Temporal workflow engine, which records every call's input and result in the workflow history. A single value above roughly 2 MB is not carried |
The effective ceiling is not a fixed, documented number. It depends on the deployment — how the platform is configured, how much runs in parallel, and which services a request passes through. Treat the figures in this handbook as the order of magnitude at which to change approach, not as a contract.
The safe design does not depend on the limit at all: put the payload in the file storage and pass the file key. That works the same way in a datapoint, a connector and a workflow, and it does not change when the deployment does.
There is also an overall maximum limit of 32GB the file store can grow to.
For the workflow-specific limits and the errors they produce, see Workflow Orchestration Limitations.
A pilet rendered by the microfrontend-interpreter reaches the file storage through the public file-storage-service API, and may use the GET method only — enough to render an image, for example. Every other method, uploads included, must go through the data-orchestrator. See Client Microfrontends.
Related
- Security in Connectivity Engineering — what OLIVE checks on upload, and which files a job may read