Research output

Whip: Human and machine-readable specifications for data

Research output: Contribution to conferencePaper/Powerpoint/Abstract


Departments, research groups and services


Original languageEnglish
Number of pages1
Publication statusPublished - 24-Sep-2018
Event10th International Conference on Ecological Informatics: Translating Ecological Data into Knowledge and Decisions in a Rapidly Changing World. - Friedrich-Schiller-Universität Jena, Jena, Germany
Duration: 24-Sep-201828-Sep-2018


Conference10th International Conference on Ecological Informatics
Abbreviated titleICEI2018
Internet address


Different tools and technologies are available to clean and harmonize data. Independent of the tool used, the ability to assess the quality of a data set and identify potential errors is crucial for harmonization efforts. The necessity becomes even more apparent in the context of data publication, (re)use and aggregation.
Documentation and guidelines about the data requirements provide guidance in this process and enable to communicate what to expect from the data, but are mostly intended for humans only. To facilitate the harmonization process, we propose the usage of a specification file, describing the constraints to which the data should comply. Its syntax is human- and machine-readable, so it can be used to communicate expected data quality/conformity and to validate data automatically. The scope of the set of specifications can be specific to a dataset, researcher or research community, which allows bottom-up and top-down adoption. As an example, we apply the specifications to verify data mapped to the biodiversity information standard Darwin Core.
In this talk, we will present "whip", a proposed syntax and format to express data specifications. Whip allows to define column-based constraints for tabular (tidy) data with a number of rules. We will also demonstrate a software application (called "pywhip") to validate data sets using these specifications. We hope it will trigger a discussion on how to express data specifications and communicate data quality expectations.

EWI Biomedical sciences


Free keywords

Research output (related by authors)
Shopping cart
Add to cart Saved citations

Copy the text from this field...



  • whip_abstract

    Final published version, 81.5 KB, PDF document

    Licence: Unspecified

View graph of relations