Pagination and cursors

Every collection pages the same way: forward, by cursor, with the shape of the page in the response rather than in a header.

There are no page numbers and no offsets. A cursor is a position in one ordering of one snapshot, so paging through 10,000 districts returns each of them exactly once even while the data underneath changes.

Paging through a collection

Ask for the first page, then pass the next_cursor you got back as cursor on the next request. When next_cursor is absent, you are at the end.

{
  "item": [],
  "page": {
    "size": 50,
    "next_cursor": "cur_01h455vb4pex5vsknk084sn02q",
    "indeterminate": 0,
    "total": { "value": 1284, "relation": "eq" }
  },
  "layer": []
}

cursor: From a previous page's next_cursor. Absent on the first request.
size: How many items to return. 1 to 200.
total: Include a match count. First page only, and off by default, because counting costs more than returning.
sort: A field name, prefixed with a minus to reverse. Each collection lists the fields it sorts on.

The page block

size is how many items this page holds. total is only there when you asked for it, and carries a relation of eq when the count is exact or gte when it was capped. Then there are the counts that explain a short page. indeterminate is rows returned because a filter could not be evaluated against them, and suppressed, on the collections that use it, is rows that matched and were withheld, from a redistribution restriction or a missing entitlement. Neither is ever silently folded into zero: a withheld row is a fact about the answer.

Filters

Filters are filter[<field>][<op>]=<value>. Conditions on different fields are ANDed; repeated values on one field are ORed. An unknown field, operator or value is a 422, never an empty result, so a typo cannot read as an answer.

GET /permission?use=energy.storage.battery&state=NC \
  &filter[dimensional.height_ft][permits]=35

eq
ne
in
nin
lt
lte
gt
gte
between
prefix
contains
is_null
not_null
any
all
permits
forbids

A row whose status has no comparable value, a height limit the code is silent on, say, cannot be said to pass or fail a numeric test. It comes back with filter_outcome: indeterminate rather than being dropped. Pass filter_indeterminate=exclude to opt out of them, or only to make the audit query: which rows the code does not speak to.

When a cursor stops working

Four codes, all 422. Each means the walk has to start again rather than be resumed.

cursor_expired: The cursor is past its retention. Start the walk again from the first page.
cursor_snapshot_gone: The snapshot the cursor pointed into is no longer retained.
cursor_sort_mismatch: The sort changed mid-walk. A cursor belongs to the ordering it was issued under.
sort_requires_query: The sort asked for only exists when there is a query to rank against.

If you are walking a whole state and mind starting over, an extract is the better shape: it is a job, it writes files, and it does not expire underneath you.

Export records

Filtering, sorting and fields

Narrowing a collection, ordering it, and asking for less of it.