Fields

Fields refer to reusable sub-schema definition used to define resource types, views, method parameter or result formats. Field types are declared using the format:

type: type-expression
validation-property-name: validation expression or text

When type is object, then a schema for sub-fields may be present:

Field	Description
field-name type-expression	Short description.

Method parameters

The following field types are mainly used to describe method parameters.

Data query

The query parameter allows selecting which data-points to include in a Data frame view.

type: object

Properties	Properties	Description
filter Data filter	default={}	Filter which data points to select.
rollup null,"window",Fixed duration	default=null	Rollup aggregation period.
last int	default=-1	If above 0, select last N timestamps per series. The selection happens after the rollup aggregation.

Examples

Basic examples:

// Custom window.
{}
{
  "filter": {
    "times": {"$gte": "2022-01-01T00:00:00Z", "$lt": "2022-01-02T00:00:00Z"}
  },
  "last": 10
}

Data filter

A data filter uses the same syntax as a Resource filter, but without $and/$or conjunctions and with a single query field:

Properties	Allowed comparators	Description
times Date/Time	$gte,$lt	Filter which timestamps to include.

Comparators exaplanation:

Operator	Usage	Description
$gte	{"times": {"$gte": "2022-01-01T00:00:00Z"}}	Matches data points where the target field is greater than or equal to the specified value.
$lt	{"times": {"$lt": "2022-01-02T00:00:00Z"}}	Matches data points where the target field is less than the specified value.

Resource query

The query parameter allows selecting resources of a particular type in Clarify. Exact validation may vary according to the specific resource you are querying for.

type: object

Properties	Description
filter Resource filter	A valid filter according to the resource being selected.
sort array(string)	An array of fields to order by.
limit int	The maximum number of resources to select. Negative numbers means no limit, which may or may not be allowed.
skip int	Skip the first N matches. A negative skip is treated as 0.
total bool	When true, force the inclusion of a total count in the response. A total count is the total number of resources that matches filter.

Examples

Basic examples:

// Empty queries will return the N oldest resources, where
{}
{
  "filter": {
    "labels.location": "banana stand"
  },
  "limit": 1000,
  "total": true
}

Pagination examples using limit,skip and total:

// Page 1 with no filer and sort by ID (creation order, oldest to newest).
{
  "limit": 10,
  "total": true
}
// Page 2 with no filer and sort by ID (creation order, oldest to newest).
{
  "limit": 10,
  "skip": 10,
  "total": true
}
// Page 1 with no filer and sort by name.
{
  "sort": ["name"],
  "limit": 10,
  "total": true
}
// Page 1 with no filer and sort by name in reverse order
{
  "sort": ["-name"],
  "limit": 10,
  "total": true
}

Cursor pagination examples:

// Page N, iterating oldest to newest.
{
  "filter": {"id": {"$gt": "<last ID from previous page>"}},
  "limit": 10
}

// Page N, iterating newest to oldest.
{
  "filter": {"id": {"$lt": "<last ID from previous page>"}},
  "limit": 10,
  "sort": ["-id"]
}

// Cursor pagination with a filter:
{
  "filter": { "$and": [
      {"id": {"$lt": "<last ID from previous page>"}}, // pagination
      {"labels.location": "banana stand"} // filter
    ]
  },
  "limit": 10,
  "sort": ["-id"]
}

Using queries for pagination

Because API calls are rate-limited per integration, the most important performance tuning you can do for pagination request, is to tune your limit value. Using the maximum is a good hint if you need to iterate all of the values always.

Further on, some performance can be gained by not asking for the total count. The trick for handling iterations when you don't know the total count, is to stop the iteration when ever you get fewer items returned then your limit value. Fewer elements could mean 0. In practice, this optimization only matters if you are iterating a lot of pages.

Another potential performance tip, is to use cursor pagination. Cursor pagination only works when sorting on the id field, and it's more complex to understand. However, cursor pagination is about more than just performance. In particular, if new resources are being created and/or deleted while you are iterating, cursor pagination will be able to handle it; skip and limit pagination simply won't.

Resource filter

A resource filter lets you construct advanced queries for which resources to include in a selection. The filter syntax is defined using a sub-set of the Mongo-DB filter syntax. Which filter comparisons are allowed for a given field path, is determined by the resource schema.

type: object

Conjunctions

Multiple filter predicates (objects) are combined with conjunctions.

Operator	Usage	Description
N/A	{"a": "b", "b": "a"}	Join queries of different field with implicit logical AND.
$and	{"$and": [{"a": "a"}, {"b": "c"}]}	Join two clauses with a logical AND conjunction.
$or	{"$or": [{"a": "b"}, {"a": "a"}]}	Join two clauses with a logical OR conjunction.

Comparators

Different fields in Clarify allow different comparators.

Fields marked with filter group comparable allows:

Operator	Usage	Description
N/A	{"a": "foo"}	Matches using equality (short-hand for $eq).
$eq	{"a": {"$eq":"foo"}}	Matches using equality.
$ne	{"a": {"$ne": "foo"}}	Matches using negated equality.
$in	{"a": {"$in": ["b", "c"]}}	Matches all resources where the field value is present in the specified list.
$nin	{"a": {"$nin": ["a", "b"]}}	Matches all resources where the field value is not present in the specified list.

Fields marked with filter group ordered allows everything that comparable fields allow as well as:

Operator	Usage	Description
$gt	{"a": {"$gt": 10}}	Matches all resources where the field value is greater than the specified value.
$gte	{"a": {"$gte": 10}}	Matches all resources where the field value is greater than or equal to the specified value.
$lt	{"a": {"$lt": 10}}	Matches all resources where the field value is less than the specified value.
$lte	{"a": {"$lte": 10}}	Matches all resources where the field value is less than or equal to the specified value.

All fields marked with filter group text allows everything that ordered fields allow as well as:

Operator	Usage	Description
$regex	{"a": {"$regex": "fo[o]{1}"}}	Matches all resources where the field value is match the specified regex.

Regex compare operator

The regex operator matches documents containing the field given as a regular expression. The syntax generally the same syntax used by Perl, Python and other languages. For more info see the rest-layer documentation and https://golang.org/s/re2syntax for the description of the syntax.

Examples

Match all records where the "location" label key has a value "banana stand":

{ "labels.location": "banana stand" }

Match records with ID "cake0ca87d5r0ffktprg" or "cake0ei87d5r0kgrm3h0":

{ "id": { "$in": ["cake0ca87d5r0ffktprg", "cake0ei87d5r0kgrm3h0"] } }

Match all records with name or description equal to "money amount":

{ "$or": [{ "name": "money amount" }, { "description": "money amount" }] }

Match all records where name or description contain the word "amount" or "money":

{
  "$or": [
    { "name": { "$regex": "(amount|money)" } },
    { "description": { "$regex": "(amount|money)" } }
  ]
}

Result parameters

The following field types are mainly used to describe result parameters.

Selection meta

type: object

Field	Description
total integer	Number of resources matching the filter or -1 if unknown.
includedGroupedByType bool	true if the included section in the result is grouped by type.

Examples

// Known total count.
{
  "total": 1001,
  "groupIncludedByTotal": false
}
// Unknown total count.
{
  "total": -1,
  "groupIncludedByTotal": false
}

Resource fields

The following field types are mainly used for defining resource schemas.

Annotations

Only visible in the API

Annotations are only visible to integrations as there is no current UI in Clarify that exposes them.

Annotations is a per-resource key-value map where your integration code can store arbitrary meta-data for programmatic purposes. Unlike all other fields, this field is merged in our save views.

type: map(string => string)
len: 0-200
key:

• regex: "^[A-Za-z0-9-_/]{1,128}$"

value:

• len: 1-128

Example

// Keys using a company domain name as the prefix:
{
  "clarify-io/publisher-script/transform": "v3",
  "clarify-io/publisher-script/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
  "clarify-io/publisher-script/source-signal/id": "cake0ca87d5r0ffktprg",

}
// Keys using a Github repo path as the prefix:
{
    "clarify/clarify-go/example/transform": "v3",
    "clarify/clarify-go/example/source-signal/attributes-hash": "0800bd89ca630f2f632ddaca3db4b341e9589f1d",
    "clarify/clarify-go/example/source-signal/id": "cake0ca87d5r0ffktprg",
}

How to annotate resources

Here are some recommendations for how to best use annotations on resources:

• Use Annotations to help drive logic only; use Labels for classification.
• Use lower-case letters for keys and prefer hyphens to join words (multi-word-example).
• Prefix keys to avoid conflict between different tools/scripts.

Date/Time

Date/Time contains a date and time alongside explicit time-zone offset information. If the field represent a stored entity, you can expect it to be converted to the UTC time zone and truncated to microsecond resolution before save. That is, unless otherwise is explicitly stated.

type: string(RFC 3339 Date/Time)

Examples

"2022-01-01T12:00:27.87Z" // Alias for +00:00, a.k.a. UTC.
"2022-01-01T12:00:27.87+02:00" // Two hour offset.

Fixed duration

Fixed duration provide an RFC 3339 duration that accepts units from day (D) to fraction (S). Days are normalized to 24 hours.

type: string(RFC 3339 duration (Appendix A))

Examples

"PT1M" // One minute.
"PT1H0.1S" // One hour and 0.1 seconds.

// Fixed duration normalization.
"PT60M"   -> "PT1H"   // 60 minutes is normalized to 1 hour.
"P1D"     -> "PT24H"  // One day is normalized to 24 hours
"PT3601S" -> "PT1H1S" // 3601 seconds is normalized to 1 hour and one second.
"P1DT1H"  -> "PT25H"  // One day and 1 hour is normalized to 25 hours.

Input key

A key used to identify a signal within an integration namespace.

type: string
regex: ^[a-zA-Z0-9-_:.#+/]{1,128}$

Labels

Labels is a key-values map used to classify resources. A single key may hold multiple labels. Note that the labels values are treated as a set, meaning that the entries are unique and unordered.

When filtering a label value, use a path labels.<key>. Comparison is done against each set member, and is considered to match when at least one set value match the comparison.

type: map(string => set(string))
len: 0-200
key:

• regex: "^[A-Za-z0-9-_/]{1,128}$"

set value:

• len: 1-128

Example

{
  "location": ["banana stand", "pier"],
  "data-source": ["google-sheets"]
}

How to label your resources

Here are some recommendations for how to best use labels on resources:

• Use Labels for classification only; use Annotations for logic.
• Use lower-case letters for keys and prefer hyphens to join words (multi-word-example).
• There is no prefix convention for labels.