Fields
Fields refer to reusable sub-schema definition used to define resource types, views, method parameter or result formats. Field types are declared using the format:
type: type-expression
validation-property-name: validation expression or text
When type is object, then a schema for sub-fields may be present:
| Field | Description |
|---|---|
field-nametype-expression | Short description. |
Method parameters
The following field types are mainly used to describe method parameters.
Alias
An alias can be used to refer to a time-series value in calculations.
type: string(regex: ^[A-Za-z_][A-Za-z0-9_]{0,27}$)
Calculation formula
A calculation formula with support for multiple operations and functions.
All operators and functions from Calculated items is supported, in addition to the following:
gapfill: Fill gaps for an input variable (item) according to it's gap detection setting using the LOCF (Last Observation Carry Forward) algorithm. Can not be used on calculated results.bucket_seconds: Returns the width of the current aggregation bucket in seconds. It returns a correctly adjusted value in the DST adjustment boundaries.1
type: string
Data query
The query parameter allows selecting which data-points to include in a Data frame view.
type: object
| Properties | Properties | Description |
|---|---|---|
filterData filter | default={} | Filter which data points to select. |
rollupCalendar duration, "window",null | default=null | Rollup aggregation period. Not all methods allow the value to be null.2 |
timeZonestring | default="UTC" | TZ Database time-zone identifier, e.g."Europe/Berlin" or "America/New_York". Used to determine the time-zone for rollup origin, and to adjust rollup buckets in for the DST adjustment boundaries.1 |
firstDayOfWeekinteger([1:7]) | default=1 | Specify which day of the week to align the origin to (1:Monday, ... , 7:Sunday). Ignored if origin is specified or if doing either a window or a month/year based rollup. |
originDate/Time, null | default=null | Specify a custom origin for your rollup. When specified, the firstDayOfWeek setting is ignored. |
lastint | default=-1 | If above 0, return only the last N timestamps that are matched by the times filter. The selection happens after rollup aggregation. |
Examples
Basic examples:
// Custom window.
{}
{
"filter": {
"times": {"$gte": "2022-01-01T00:00:00Z", "$lt": "2022-01-02T00:00:00Z"}
},
"last": 10
}
Data filter
A data filter uses the same syntax as a Resource filter, but without $and/$or conjunctions and with fewer comparators:
type: object
| Properties | Allowed comparators | Description |
|---|---|---|
timesDate/Time | $gte,$lt | Filter which timestamps to include. |
seriesSeries key | $in | Filter which series to include in the result. |
Comparators explanation:
| Operator | Usage | Description |
|---|---|---|
$gte | {"times": {"$gte": "2022-01-01T00:00:00Z"}} | Matches data points where the target field is greater than or equal to the specified value. |
$lt | {"times": {"$lt": "2022-01-02T00:00:00Z"}} | Matches data points where the target field is less than the specified value. |
Data aggregation
A data aggregation allows rolling up samples from a time-series into a window or time-bucket aggregations.
type: string(enum)
Values:
| Aggregation | Additional parameters | Description |
|---|---|---|
"count" | Returns the number of samples for the aggregation bucket or null. | |
"min" | Returns the lowest value from the aggregation bucket or null. | |
"max" | Returns the highest value from the aggregation bucket or null. | |
"sum" | Returns the sum of all values in the aggregation bucket or null. | |
"avg" | Returns the average of all values in the aggregation bucket or null. | |
"state-histogram-seconds" | state integer | Returns the number of seconds the floor value of the item equal the provided state for the aggregation bucket or null. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting. |
"state-histogram-percent" | state integer | Returns the percentage of time that the floor value of the item equal the provided state for the aggregation bucket or null. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting. |
"state-histogram-rate" | state integer | Returns a number in range 0 - 1 for the portion of time time that the floor value of the item equal the provided state for the aggregation bucket or null. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting. |
Resource query
The query parameter allows selecting resources of a particular type in Clarify. Exact validation may vary according to the specific resource you are querying for.
type: object
| Properties | Description |
|---|---|
filterResource filter | A valid filter according to the resource being selected. |
sortarray(string) | An array of fields to order by. |
limitint | The maximum number of resources to select. Negative numbers means no limit, which may or may not be allowed. |
skipint | Skip the first N matches. A negative skip is treated as 0. |
totalbool | When true, force the inclusion of a total count in the response. A total count is the total number of resources that matches filter. |
Examples
Basic examples:
// Empty queries will return the N oldest resources, where
{}
{
"filter": {
"labels.location": "banana stand"
},
"limit": 1000,
"total": true
}
Pagination examples using limit,skip and total:
// Page 1 with no filer and sort by ID (creation order, oldest to newest).
{
"limit": 10,
"total": true
}
// Page 2 with no filer and sort by ID (creation order, oldest to newest).
{
"limit": 10,
"skip": 10,
"total": true
}
// Page 1 with no filer and sort by name.
{
"sort": ["name"],
"limit": 10,
"total": true
}
// Page 1 with no filer and sort by name in reverse order
{
"sort": ["-name"],
"limit": 10,
"total": true
}
Cursor pagination examples:
// Page N, iterating oldest to newest.
{
"filter": {"id": {"$gt": "<last ID from previous page>"}},
"limit": 10
}
// Page N, iterating newest to oldest.
{
"filter": {"id": {"$lt": "<last ID from previous page>"}},
"limit": 10,
"sort": ["-id"]
}
// Cursor pagination with a filter:
{
"filter": { "$and": [
{"id": {"$lt": "<last ID from previous page>"}}, // pagination
{"labels.location": "banana stand"} // filter
]
},
"limit": 10,
"sort": ["-id"]
}
Because API calls are rate-limited per integration, the most important performance tuning you can do for pagination request, is to tune your limit value. Using the maximum is a good hint if you need to iterate all of the values always.
Further on, some performance can be gained by not asking for the total count. The trick for handling iterations when you don't know the total count, is to stop the iteration when ever you get fewer items returned then your limit value. Fewer elements could mean 0. In practice, this optimization only matters if you are iterating a lot of pages.
Another potential performance tip, is to use cursor pagination. Cursor pagination only works when sorting on the id field, and it's more complex to understand. However, cursor pagination is about more than just performance. In particular, if new resources are being created and/or deleted while you are iterating, cursor pagination will be able to handle it; skip and limit pagination simply won't.
Resource filter
A resource filter lets you construct advanced queries for which resources to include in a selection. The filter syntax is defined using a sub-set of the Mongo-DB filter syntax. Which filter comparisons are allowed for a given field path, is determined by the resource schema.
type: object
Conjunctions
Multiple filter predicates (objects) are combined with conjunctions.
| Operator | Usage | Description |
|---|---|---|
| N/A | {"a": "b", "b": "a"} | Join queries of different field with implicit logical AND. |
$and | {"$and": [{"a": "a"}, {"b": "c"}]} | Join two clauses with a logical AND conjunction. |
$or | {"$or": [{"a": "b"}, {"a": "a"}]} | Join two clauses with a logical OR conjunction. |
Comparators
Different fields in Clarify allow different comparators.
Fields marked with filter group comparable allows:
| Operator | Usage | Description |
|---|---|---|
| N/A | {"a": "foo"} | Matches using equality (short-hand for $eq). |
$eq | {"a": {"$eq":"foo"}} | Matches using equality. |
$ne | {"a": {"$ne": "foo"}} | Matches using negated equality. |
$in | {"a": {"$in": ["b", "c"]}} | Matches all resources where the field value is present in the specified list. |
$nin | {"a": {"$nin": ["a", "b"]}} | Matches all resources where the field value is not present in the specified list. |
Fields marked with filter group ordered allows everything that comparable fields allow as well as:
| Operator | Usage | Description |
|---|---|---|
$gt | {"a": {"$gt": 10}} | Matches all resources where the field value is greater than the specified value. |
$gte | {"a": {"$gte": 10}} | Matches all resources where the field value is greater than or equal to the specified value. |
$lt | {"a": {"$lt": 10}} | Matches all resources where the field value is less than the specified value. |
$lte | {"a": {"$lte": 10}} | Matches all resources where the field value is less than or equal to the specified value. |
All fields marked with filter group text allows everything that ordered fields allow as well as:
| Operator | Usage | Description |
|---|---|---|
$regex | {"a": {"$regex": "fo[o]{1}"}} | Matches all resources where the field value is match the specified regex. |
Regex compare operator
The regex operator matches documents containing the field given as a regular expression. The syntax generally the same syntax used by Perl, Python and other languages. For more info see the rest-layer documentation and https://golang.org/s/re2syntax for the description of the syntax.
Examples
Match all records where the "location" label key has a value "banana stand":
{ "labels.location": "banana stand" }
Match records with ID "cake0ca87d5r0ffktprg" or "cake0ei87d5r0kgrm3h0":
{ "id": { "$in": ["cake0ca87d5r0ffktprg", "cake0ei87d5r0kgrm3h0"] } }
Match all records with name or description equal to "money amount":
{ "$or": [{ "name": "money amount" }, { "description": "money amount" }] }
Match all records where name or description contain the word "amount" or "money":
{
"$or": [
{ "name": { "$regex": "(amount|money)" } },
{ "description": { "$regex": "(amount|money)" } }
]
}
Selection format
The selection format hold configuration for how to format a selection view. This object is generally configured by SDKs / client code, and not exposed as an end-user option.
type: object
| Field | Description |
|---|---|
dataAsArraybool | Return the data as an array. |
groupIncludedByTypebool | Group included resources into an object by type. |
Result parameters
The following field types are mainly used to describe result parameters.
Selection meta
Meta data for a selection view, describing the number of results, formatting options and request path issues. Currently issues are only included in data methods results where there is a use-case for returning partial results on errors.
type: object
| Field | Description |
|---|---|
totalinteger | Number of resources matching the filter or -1 if unknown. |
formatSelection format | true if the included section in the result is grouped by type. |
issuesPath issues | Path issues in the request that does not prevent us from returning a (partial) result. Omitted if empty. |
Examples
// Known total count.
{
"total": 1001,
"format": {
"dataAsArray": true,
"groupIncludedByTotal": false
}
}
// Unknown total count.
{
"total": -1,
"format": {
"dataAsArray": true,
"groupIncludedByTotal": false
}
}
Path issues
Path issues expose a map of request parameter paths joined by dot (.) to an array of issues.
type: map(string => array(string))
Example
{
"calculations.0": ["can not evaluate gapfill(a) + b: unknown parameter a"],
"items.0": ["not found"]
}
Resource fields
The following field types are mainly used for defining resource schemas.
Annotations
Annotations are only visible to integrations as there is no current UI in Clarify that exposes them.
Annotations is a per-resource key-value map where your integration code can store arbitrary meta-data for programmatic purposes. Unlike all other fields, this field is merged in our save views.
type: map(string => string)
len: 0-200
key:
- regex:
"^[A-Za-z0-9-_/]{1,128}$"
value:
- len: 1-128
Example
// Keys using a company domain name as the prefix:
{
"clarify-io/publisher-script/transform": "v3",
"clarify-io/publisher-script/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
"clarify-io/publisher-script/source-signal/id": "cake0ca87d5r0ffktprg",
}
// Keys using a Github repo path as the prefix:
{
"clarify/clarify-go/example/transform": "v3",
"clarify/clarify-go/example/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
"clarify/clarify-go/example/source-signal/id": "cake0ca87d5r0ffktprg",
}
Here are some recommendations for how to best use annotations on resources:
- Use Annotations to help drive logic only; use Labels for classification.
- Use lower-case letters for keys and prefer hyphens to join words (
multi-word-example). - Prefix keys to avoid conflict between different tools/scripts.
Date/Time
Date/Time contains a date and time alongside explicit time-zone offset information. If the field represent a stored entity, you can expect it to be converted to the UTC time zone and truncated to microsecond resolution before save. That is, unless otherwise is explicitly stated.
type: string(RFC 3339 Date/Time)
Examples
"2022-01-01T12:00:27.87Z" // Alias for +00:00, a.k.a. UTC.
"2022-01-01T12:00:27.87+02:00" // Two hour offset.
Calendar duration
Calendar duration provide an RFC 3339 duration that accepts units from year (Y) to fraction (S). Note that it's not allowed to combine months and years with weeks, days or clock components (hours, minutes or fractions). When parsed, results are normalized and truncated to microsecond resolution. Note that seemingly fixed durations may not be fixed in UTC when performing operations that rely on time-zone configuration.1
type: string(RFC 3339 duration (Appendix A))
// Allowed:
"PT1H"
"P1D5H2M0.01S"
"P1M"
"P1Y2M"
// Not allowed:
"P1YT1H"
"P1M1D"
Fixed duration
Fixed Duration provide an RFC 3339 duration that accepts units from day (D) to fraction (S). Days are normalized to 24 hours.
type: string(RFC 3339 duration (Appendix A))
Examples
"PT1M" // One minute.
"PT1H0.1S" // One hour and 0.1 seconds.
// Fixed duration normalization.
"PT60M" -> "PT1H" // 60 minutes is normalized to 1 hour.
"P1D" -> "PT24H" // One day is normalized to 24 hours
"PT3601S" -> "PT1H1S" // 3601 seconds is normalized to 1 hour and one second.
"P1DT1H" -> "PT25H" // One day and 1 hour is normalized to 25 hours.
Series key
A key used to identify a time-series in a #Data frame.
When writing data to Clarify, we refer to the series key as the input key. Input keys are namespaced to the integration and used to identify the correct signal to write the data to. When reading data from Clarify, the series key has no relation to the signal input ID.
type: string
regex: ^[a-zA-Z0-9-_:.#+/]{1,128}$
Labels
Labels is a key-values map used to classify resources. A single key may hold multiple labels. Note that the labels values are treated as a set, meaning that the entries are unique and unordered.
When filtering a label value, use a path labels.<key>. Comparison is done against each set member, and is considered to match when at least one set value match the comparison.
type: map(string => set(string))
len: 0-200
key:
- regex:
"^[A-Za-z0-9-_/]{1,128}$"
set value:
- len: 1-128
Example
{
"location": ["banana stand", "pier"],
"data-source": ["google-sheets"]
}
Here are some recommendations for how to best use labels on resources:
- Use Labels for classification only; use Annotations for logic.
- Use lower-case letters for keys and prefer hyphens to join words (
multi-word-example). - There is no prefix convention for labels.
Footnotes
-
In the daylight saving time (DST) adjustment boundaries, the bucket size is adjusted to match the clock in local time if the rollup bucket is wider than the time-zone boundary adjustment (typically 1h). E.g. for a bucket width of 2h in Europe/Berlin, we will get one bucket that measures 3h in UTC time when adjusting to summer time (time adjusted forwards), and one bucket of 1h in UTC when adjusting to winter time (time adjusted backwards). This creates two odd buckets each year, but ensures that this particular bucket align to the same clock time for every day of the year. ↩ ↩2 ↩3
-
When calling the
clarify.evaluatemethod, therollupparameter can not benull. ↩