Concept: Dataset Schema Changes
Schemas are the structures behind how your data is organized. A dataset might consist
of multiple schemas that you define when creating the dataset. Each of these dataset
components has its own schema:
- The files containing external data that you upload into a dataset.
- The custom report you create to bring Workday data into a dataset.
- The input to a pipeline stage. For the first stage in a base dataset, this schema is determined by the dataset source.
- The output from a pipeline stage. For the last stage in the Primary Pipeline, this schema determines the output schema of the entire dataset.
Schema changes can happen anywhere in a dataset. Schemas change when you add new
fields or remove existing fields, such as when you:
- Add a Prism calculated field.
- Hide or expose a field.
- Add a stage that adds or deletes fields in that stage.
Schemas can also change when you:
- Upload an external file.
- Import a file from an SFTP server.
- Import data from a custom report.
Where changes occur in a dataset can affect other dataset components. Changes to dataset
schemas can break components that were based on the original schema. Where changes
occur can also affect how you manage changes.
When you change the schema and Workday
imports the new source file into the dataset, you must open the dataset and save
the changes. If you don’t save the dataset, Workday continues to use the old
schema definition. The next time you publish the dataset, you might get
inconsistent data in the Prism data source.
When you import data that changes the schema of your dataset from an SFTP server or custom
report, you must manually edit the dataset to incorporate the changes. If new fields
are added to the dataset, Workday hides them. Expose the new fields and save the
dataset to include the fields in the dataset schema.
Workday recommends that you ensure that the dataset is up to date and includes the
fields you want. In most cases, you use the Manage Fields stage to manage dataset
schema changes.
Manage Fields
The Manage Fields stage uses the output of the previous stage as a baseline from
which to monitor changes. When this baseline changes, Workday warns you in the
pipeline details panel. If the baseline changes, in the Manage Fields stage,
Workday:
- Displays all new and removed fields.
- Doesn't yet include all changes in the dataset schema.
- Displays no data in the example table.
Workday recommends that you:
- Add a Manage Fields stage at the beginning of the Primary Pipeline of a derived dataset when you want to monitor the schema of the table or dataset from which it’s derived.
- Add a Manage Fields stage at the end of the Primary Pipeline of a dataset that you intend to publish. This enables you to detect any schema changes that might break reports that use the Prism data source of this published dataset.
- Add a Manage Fields stage at the end of a pipeline when you need to hide fields or expose new fields.
- Add a Manage Fields stage in a base dataset to ensure that no future integration can unintentionally remove an existing field.
- Include no more than 2 Manage Fields stages in a single pipeline.
In some cases, Workday handles schema changes without using a Manage Fields stage.
Example: When uploading a new version of a source file into a dataset, Workday
handles schema changes based on the source file header row:
Source File Contains Header Row | Result |
|---|---|
Yes | Workday uses the field names in the header row of the dataset to
determine which fields are in the new source file. You can add or
delete fields anywhere in the source file. If the field names in the
source file don’t change, Workday updates the schema of the dataset
into which you import the new source file. |
No | Workday handles changes that occur only at the end of the source
file. You can only add new fields at the end of the source file.
If you do so, Workday updates the schema of the dataset into which
you're importing the new source file. |