Version: 1.1

Fields

Fields refer to reusable sub-schema definition used to define resource types, views, method parameter or result formats. Field types are declared using the format:

type: type-expression
validation-property-name: validation expression or text

When type is object, then a schema for sub-fields may be present:

Field	Description
`field-name` type-expression	Short description.

Method parameters

The following field types are mainly used to describe method parameters.

Alias

An alias can be used to refer to a time-series value in calculations.

type: string(regex: ^[A-Za-z_][A-Za-z0-9_]{0,27}$)

Calculation formula

A calculation formula with support for multiple operations and functions.

All operators and functions from Calculated items is supported, in addition to the following:

gapfill: Fill gaps for an input variable (item) according to it's gap detection setting using the LOCF (Last Observation Carry Forward) algorithm. Can not be used on calculated results.
bucket_seconds: Returns the width of the current aggregation bucket in seconds. It returns a correctly adjusted value in the DST adjustment boundaries.^dst-buckets

type: string

Data query

The query parameter allows selecting which data-points to include in a Data frame view.

type: object

Properties	Properties	Description
`filter` Data filter	default=`{}`	Filter which data points to select.
`rollup` Calendar duration,`"window"`,`null`	default=`null`	Rollup aggregation period. Not all methods allow the value to be `null`.^{required-rollup}
`timeZone` string	default=`"UTC"`	TZ Database time-zone identifier, e.g."Europe/Berlin" or "America/New_York". Used to determine the time-zone for rollup origin, and to adjust rollup buckets in for the DST adjustment boundaries.^dst-buckets
`firstDayOfWeek` integer([1:7])	default=`1`	Specify which day of the week to align the origin to (1:Monday, ... , 7:Sunday). Ignored if `origin` is specified or if doing either a window or a month/year based rollup.
`origin` Date/Time,`null`	default=`null`	Specify a custom origin for your rollup. When specified, the `firstDayOfWeek` setting is ignored.
`last` int	default=`-1`	If above `0`, return only the last N timestamps that are matched by the times filter. The selection happens after rollup aggregation.

Examples

Basic examples:

// Custom window.
{}
{
  "filter": {
    "times": {"$gte": "2022-01-01T00:00:00Z", "$lt": "2022-01-02T00:00:00Z"}
  },
  "last": 10
}

Data filter

A data filter uses the same syntax as a Resource filter, but without $and/$or conjunctions and with fewer comparators:

type: object

Properties	Allowed comparators	Description
`times` Date/Time	`$gte`,`$lt`	Filter which timestamps to include.
`series` Series key	`$in`	Filter which series to include in the result.

Comparators explanation:

Operator	Usage	Description
`$gte`	`{"times": {"$gte": "2022-01-01T00:00:00Z"}}`	Matches data points where the target field is greater than or equal to the specified value.
`$lt`	`{"times": {"$lt": "2022-01-02T00:00:00Z"}}`	Matches data points where the target field is less than the specified value.

Data aggregation

A data aggregation allows rolling up samples from a time-series into a window or time-bucket aggregations.

type: string(enum)

Values:

Aggregation	Additional parameters	Description
`"count"`		Returns the number of samples for the aggregation bucket or `null`.
`"min"`		Returns the lowest value from the aggregation bucket or `null`.
`"max"`		Returns the highest value from the aggregation bucket or `null`.
`"sum"`		Returns the sum of all values in the aggregation bucket or `null`.
`"avg"`		Returns the average of all values in the aggregation bucket or `null`.
`"state-histogram-seconds"`	`state` integer	Returns the number of seconds the floor value of the item equal the provided state for the aggregation bucket or `null`. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting.
`"state-histogram-percent"`	`state` integer	Returns the percentage of time that the floor value of the item equal the provided state for the aggregation bucket or `null`. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting.
`"state-histogram-rate"`	`state` integer	Returns a number in range 0 - 1 for the portion of time time that the floor value of the item equal the provided state for the aggregation bucket or `null`. To determine when an item has a value (a.k.a. state), we rely on the item gap detection setting.

Resource query

The query parameter allows selecting resources of a particular type in Clarify. Exact validation may vary according to the specific resource you are querying for.

type: object

Properties	Description
`filter` Resource filter	A valid filter according to the resource being selected.
`sort` array(string)	An array of fields to order by.
`limit` int	The maximum number of resources to select. Negative numbers means no limit, which may or may not be allowed.
`skip` int	Skip the first N matches. A negative skip is treated as `0`.
`total` bool	When true, force the inclusion of a `total` count in the response. A total count is the total number of resources that matches `filter`.

Examples

Basic examples:

// Empty queries will return the N oldest resources, where
{}
{
  "filter": {
    "labels.location": "banana stand"
  },
  "limit": 1000,
  "total": true
}

Pagination examples using limit,skip and total:

// Page 1 with no filer and sort by ID (creation order, oldest to newest).
{
  "limit": 10,
  "total": true
}
// Page 2 with no filer and sort by ID (creation order, oldest to newest).
{
  "limit": 10,
  "skip": 10,
  "total": true
}
// Page 1 with no filer and sort by name.
{
  "sort": ["name"],
  "limit": 10,
  "total": true
}
// Page 1 with no filer and sort by name in reverse order
{
  "sort": ["-name"],
  "limit": 10,
  "total": true
}

Cursor pagination examples:

// Page N, iterating oldest to newest.
{
  "filter": {"id": {"$gt": "<last ID from previous page>"}},
  "limit": 10
}

// Page N, iterating newest to oldest.
{
  "filter": {"id": {"$lt": "<last ID from previous page>"}},
  "limit": 10,
  "sort": ["-id"]
}

// Cursor pagination with a filter:
{
  "filter": { "$and": [
      {"id": {"$lt": "<last ID from previous page>"}}, // pagination
      {"labels.location": "banana stand"} // filter
    ]
  },
  "limit": 10,
  "sort": ["-id"]
}

Using queries for pagination

Because API calls are rate-limited per integration, the most important performance tuning you can do for pagination request, is to tune your limit value. Using the maximum is a good hint if you need to iterate all of the values always.

Further on, some performance can be gained by not asking for the total count. The trick for handling iterations when you don't know the total count, is to stop the iteration when ever you get fewer items returned then your limit value. Fewer elements could mean 0. In practice, this optimization only matters if you are iterating a lot of pages.

Another potential performance tip, is to use cursor pagination. Cursor pagination only works when sorting on the id field, and it's more complex to understand. However, cursor pagination is about more than just performance. In particular, if new resources are being created and/or deleted while you are iterating, cursor pagination will be able to handle it; skip and limit pagination simply won't.

Resource filter

A resource filter lets you construct advanced queries for which resources to include in a selection. The filter syntax is defined using a sub-set of the Mongo-DB filter syntax. Which filter comparisons are allowed for a given field path, is determined by the resource schema.

type: object

Conjunctions

Multiple filter predicates (objects) are combined with conjunctions.

Operator	Usage	Description
N/A	`{"a": "b", "b": "a"}`	Join queries of different field with implicit logical AND.
`$and`	`{"$and": [{"a": "a"}, {"b": "c"}]}`	Join two clauses with a logical AND conjunction.
`$or`	`{"$or": [{"a": "b"}, {"a": "a"}]}`	Join two clauses with a logical OR conjunction.

Comparators

Different fields in Clarify allow different comparators.

Fields marked with filter group comparable allows:

Operator	Usage	Description
N/A	`{"a": "foo"}`	Matches using equality (short-hand for `$eq`).
`$eq`	`{"a": {"$eq":"foo"}}`	Matches using equality.
`$ne`	`{"a": {"$ne": "foo"}}`	Matches using negated equality.
`$in`	`{"a": {"$in": ["b", "c"]}}`	Matches all resources where the field value is present in the specified list.
`$nin`	`{"a": {"$nin": ["a", "b"]}}`	Matches all resources where the field value is not present in the specified list.

Fields marked with filter group ordered allows everything that comparable fields allow as well as:

Operator	Usage	Description
`$gt`	`{"a": {"$gt": 10}}`	Matches all resources where the field value is greater than the specified value.
`$gte`	`{"a": {"$gte": 10}}`	Matches all resources where the field value is greater than or equal to the specified value.
`$lt`	`{"a": {"$lt": 10}}`	Matches all resources where the field value is less than the specified value.
`$lte`	`{"a": {"$lte": 10}}`	Matches all resources where the field value is less than or equal to the specified value.

All fields marked with filter group text allows everything that ordered fields allow as well as:

Operator	Usage	Description
`$regex`	`{"a": {"$regex": "fo[o]{1}"}}`	Matches all resources where the field value is match the specified regex.

Regex compare operator

The regex operator matches documents containing the field given as a regular expression. The syntax generally the same syntax used by Perl, Python and other languages. For more info see the rest-layer documentation and https://golang.org/s/re2syntax for the description of the syntax.

Examples

Match all records where the "location" label key has a value "banana stand":

{ "labels.location": "banana stand" }

Match records with ID "cake0ca87d5r0ffktprg" or "cake0ei87d5r0kgrm3h0":

{ "id": { "$in": ["cake0ca87d5r0ffktprg", "cake0ei87d5r0kgrm3h0"] } }

Match all records with name or description equal to "money amount":

{ "$or": [{ "name": "money amount" }, { "description": "money amount" }] }

Match all records where name or description contain the word "amount" or "money":

{
  "$or": [
    { "name": { "$regex": "(amount|money)" } },
    { "description": { "$regex": "(amount|money)" } }
  ]
}

Selection format

The selection format hold configuration for how to format a selection view. This object is generally configured by SDKs / client code, and not exposed as an end-user option.

type: object

Field	Description
`dataAsArray` bool	Return the data as an array.
`groupIncludedByType` bool	Group included resources into an object by type.

Result parameters

The following field types are mainly used to describe result parameters.

Selection meta

Meta data for a selection view, describing the number of results, formatting options and request path issues. Currently issues are only included in data methods results where there is a use-case for returning partial results on errors.

type: object

Field	Description
`total` integer	Number of resources matching the filter or -1 if unknown.
`format` Selection format	`true` if the `included` section in the result is grouped by type.
`issues` Path issues	Path issues in the request that does not prevent us from returning a (partial) result. Omitted if empty.

Examples

// Known total count.
{
  "total": 1001,
  "format": {
    "dataAsArray": true,
    "groupIncludedByTotal": false
  }
}
// Unknown total count.
{
  "total": -1,
  "format": {
    "dataAsArray": true,
    "groupIncludedByTotal": false
  }
}

Path issues

Path issues expose a map of request parameter paths joined by dot (.) to an array of issues.

type: map(string => array(string))

Example

{
  "calculations.0": ["can not evaluate gapfill(a) + b: unknown parameter a"],
  "items.0": ["not found"]
}

Resource fields

The following field types are mainly used for defining resource schemas.

Annotations

Only visible in the API

Annotations are only visible to integrations as there is no current UI in Clarify that exposes them.

Annotations is a per-resource key-value map where your integration code can store arbitrary meta-data for programmatic purposes. Unlike all other fields, this field is merged in our save views.

type: map(string => string)
len: 0-200
key:

regex: "^[A-Za-z0-9-_/]{1,128}$"

value:

len: 1-128

Example

// Keys using a company domain name as the prefix:
{
  "clarify-io/publisher-script/transform": "v3",
  "clarify-io/publisher-script/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
  "clarify-io/publisher-script/source-signal/id": "cake0ca87d5r0ffktprg",

}
// Keys using a Github repo path as the prefix:
{
    "clarify/clarify-go/example/transform": "v3",
    "clarify/clarify-go/example/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
    "clarify/clarify-go/example/source-signal/id": "cake0ca87d5r0ffktprg",
}

How to annotate resources

Here are some recommendations for how to best use annotations on resources:

Use Annotations to help drive logic only; use Labels for classification.
Use lower-case letters for keys and prefer hyphens to join words (multi-word-example).
Prefix keys to avoid conflict between different tools/scripts.

Date/Time

Date/Time contains a date and time alongside explicit time-zone offset information. If the field represent a stored entity, you can expect it to be converted to the UTC time zone and truncated to microsecond resolution before save. That is, unless otherwise is explicitly stated.

type: string(RFC 3339 Date/Time)

Examples

"2022-01-01T12:00:27.87Z" // Alias for +00:00, a.k.a. UTC.
"2022-01-01T12:00:27.87+02:00" // Two hour offset.

Calendar duration

Calendar duration provide an RFC 3339 duration that accepts units from year (Y) to fraction (S). Note that it's not allowed to combine months and years with weeks, days or clock components (hours, minutes or fractions). When parsed, results are normalized and truncated to microsecond resolution. Note that seemingly fixed durations may not be fixed in UTC when performing operations that rely on time-zone configuration.^dst-buckets

type: string(RFC 3339 duration (Appendix A))

// Allowed:
"PT1H"
"P1D5H2M0.01S"
"P1M"
"P1Y2M"

// Not allowed:
"P1YT1H"
"P1M1D"

Fixed duration

Fixed Duration provide an RFC 3339 duration that accepts units from day (D) to fraction (S). Days are normalized to 24 hours.

type: string(RFC 3339 duration (Appendix A))

Examples

"PT1M" // One minute.
"PT1H0.1S" // One hour and 0.1 seconds.

// Fixed duration normalization.
"PT60M"   -> "PT1H"   // 60 minutes is normalized to 1 hour.
"P1D"     -> "PT24H"  // One day is normalized to 24 hours
"PT3601S" -> "PT1H1S" // 3601 seconds is normalized to 1 hour and one second.
"P1DT1H"  -> "PT25H"  // One day and 1 hour is normalized to 25 hours.

Series key

A key used to identify a time-series in a #Data frame.

When writing data to Clarify, we refer to the series key as the input key. Input keys are namespaced to the integration and used to identify the correct signal to write the data to. When reading data from Clarify, the series key has no relation to the signal input ID.

type: string
regex: ^[a-zA-Z0-9-_:.#+/]{1,128}$

Labels

Labels is a key-values map used to classify resources. A single key may hold multiple labels. Note that the labels values are treated as a set, meaning that the entries are unique and unordered.

When filtering a label value, use a path labels.<key>. Comparison is done against each set member, and is considered to match when at least one set value match the comparison.

type: map(string => set(string))
len: 0-200
key:

regex: "^[A-Za-z0-9-_/]{1,128}$"

set value:

len: 1-128

Example

{
  "location": ["banana stand", "pier"],
  "data-source": ["google-sheets"]
}

How to label your resources

Here are some recommendations for how to best use labels on resources:

Use Labels for classification only; use Annotations for logic.
Use lower-case letters for keys and prefer hyphens to join words (multi-word-example).
There is no prefix convention for labels.

In the daylight saving time (DST) adjustment boundaries, the bucket size is adjusted to match the clock in local time if the rollup bucket is wider than the time-zone boundary adjustment (typically 1h). E.g. for a bucket width of 2h in Europe/Berlin, we will get one bucket that measures 3h in UTC time when adjusting to summer time (time adjusted forwards), and one bucket of 1h in UTC when adjusting to winter time (time adjusted backwards). This creates two odd buckets each year, but ensures that this particular bucket align to the same clock time for every day of the year.↩
When calling the clarify.evaluate method, the rollup parameter can not be null.↩

Method parameters​

Alias​

Calculation formula​

Data query​

Examples​

Data filter​

Data aggregation​

Resource query​

Examples​

Resource filter​

Conjunctions​

Comparators​

Regex compare operator​

Examples​

Selection format​

Result parameters​

Selection meta​

Examples​

Path issues​

Example​

Resource fields​

Annotations​

Example​

Date/Time​

Examples​

Calendar duration​

Fixed duration​

Examples​

Series key​

Labels​

Example​

Method parameters

Alias

Calculation formula

Data query

Examples

Data filter

Data aggregation

Resource query

Examples

Resource filter

Conjunctions

Comparators

Regex compare operator

Examples

Selection format

Result parameters

Selection meta

Examples

Path issues

Example

Resource fields

Annotations

Example

Date/Time

Examples

Calendar duration

Fixed duration

Examples

Series key

Labels

Example