Documentation
On this page

Performance

Measured reads, writes, live streams, and revalidation at 100,000, 300,000, and 1,000,000 objects of one type, on a 4-core machine. No tuning; the numbers describe this machine and this setup, nothing more.

Measured on 2026-10-25. Everything measured goes through the HTTP API.

In short

100,000300,0001,000,000
Get one object by id, p501.45 ms2.25 ms1.54 ms
Traverse a link, first 50, p503.72 ms5.60 ms6.70 ms
linked filter plus eq, p503.48 ms5.99 ms5.05 ms
eq on a 10% value, first page, p503.10 ms54 ms96 ms
Next page by cursor, p502.09 ms2.61 ms2.25 ms
starts_with on a property, p50113 ms373 ms1.3 s
order_by a property, top 50, p50250 ms755 ms2.3 s
Create objects, 8 clients828/s831/s939/s
Apply an action, 8 clients740/s645/s637/s
Catch up a live stream40,328 events/s47,669 events/s48,326 events/s
Rules change answered in316 ms994 ms2.5 s
Revalidate every object after it29 s88 s281 s
  • Lookups by id, link traversals, and linked filters use indexes and stay in single-digit milliseconds at every size.
  • Filters and sorts on properties scan the type’s objects. There are no property indexes yet. Filters that find their first page early stay cheap; those that must look at every object, such as starts_with or order_by on a property, grow with the type.
  • Writes do not depend on the size: about 800 object creations, 750 links, or 650 actions per second with 8 clients on these 4 cores, and less with 32, as the clients contend for the cores and the 20 pooled connections. Actions refuse a concurrent change to an object they read with 409 version_conflict by design; with random targets among this many objects, none occurred.
  • A type’s validation summary counts every object of the type, so a schema change that changes the rules takes longer to answer as the type grows, without reading any object’s data.
  • Revalidation after a rules change rechecks about 3,500 objects per second at every size: a million objects in under 5 minutes.
  • Catching up a live stream ran at 40,000 to 48,000 events per second, the first event within 20 ms, at every size.

Revalidation

After a rules change, a background job rechecks every object of the type. Measured on the same machine and setup, from the rules change to the job’s last type.validation event, with every object valid:

AssetsRules change answered inRechecked inobjects/stype.validation events
100,000252 ms29.1 s3,43513
300,000724 ms87.8 s3,41713
1,000,0002.6 s281.4 s3,55412

The job walks the type once, in batches found from a cursor. Its progress events count the whole type, so it sends them at most once a second, and less often on large types: after each one it waits nine times as long as that count took.

In the first run, before this was fixed, revalidation was quadratic: after each batch of 200 objects the job recounted the whole type for its progress event, and its query for the next batch scanned past every object already checked (301 ms per batch at a million objects, against 1.8 ms from a cursor). Its rate fell from 521 objects/s at 120,000 objects to 217 at 320,000 and 79 at 1,000,000, where 30 minutes covered 14% of the type. The per-size results below keep those numbers.

Machine

CPUAMD EPYC 9554P, 4 vCPUs
Memory16 GiB
OSLinux 6.12
Diskext4 on a virtual disk
Postgres18.6 over a unix socket, default settings: shared_buffers 128MB, work_mem 4MB, synchronous_commit on
Serverrelease build, 20 connections in the organization API’s pool, RUST_LOG=warn, custom validators off

The server, Postgres, and the benchmark client share these cores, and HTTP goes over loopback TCP.

Setup

One organization with two object types: asset (name, a status enum, priority an integer 1-5, cost a double, site_code) and site (code, a region enum). Each asset has a many_to_one link to one of a hundredth as many sites, and the writes add many_to_many links between assets. The bulk of each size is inserted directly into Postgres in the form the API stores, then ANALYZEd. Each size’s revalidation rechecks every object, the bulk ones included.

  • Reads come from one sequential client, an agent with view_objects and an API token, so row level security and the object access policy apply: 200 calls each after a warm-up, or as many as fit in a minute (n below). Their parameters are spread over the bulk objects.
  • Writes come from the owner, with a session token. Each client sends its next request when the last one answers; each run lasts 10 seconds.

Results

100,000 assets

Before the runs: 100000 assets, 1000 sites, 100000 links, 4 events; database 147 MiB. Bulk top-up took 7.5 s.

Reads, one client, as an agent with view_objects (ms):

ReadCallnp50p99maxrows/call
get one objectGET /v1/ontology/objects/asset/{id}2001.455.20111
eq status = down (10%), first 50 by idquery2003.10111750
eq status = running, page 2 (cursor)query with the first page’s cursor2002.093.774.0550
and priority = 5, cost >= 990 (0.2%)query2007111018250
starts_with a name prefix (~100 matches at 10^6)query20011324326350
order_by cost desc, top 50 of allquery20025031232250
linked to one site and eq status = runningquery2003.488.029.9235
traverse asset -> siteGET .../asset/{id}/links/site2002.113.764.781
traverse site -> assets, first 50GET .../site/{id}/links/assets2003.724.828.1150

Writes as the owner, 10 s per run:

WriteClientsops/sp50 msp99 mserrors
create an object14731.934.880
create an object88286.42510
create an object32733331780
create a link12913.117.950
create a link88557.58330
create a link32780321480
apply an action (1 criterion, 1 update)12303.989.760
apply an action (1 criterion, 1 update)87408.87380
apply an action (1 criterion, 1 update)32580432030

Live-stream catch-up: a new stream with a cursor from before the writes received their 70719 events in 1.75 s (40328 events/s; first after 11 ms).

Revalidation (before the fix): a rules change (max on cost) answered in 316 ms; the maintenance job then rechecked all 120384 assets, every one valid, in 231 s (521 objects/s).

300,000 assets

Before the runs: 300000 assets, 3000 sites, 298901 links, 71326 events; database 519 MiB. Bulk top-up took 16.1 s.

Reads, one client, as an agent with view_objects (ms):

ReadCallnp50p99maxrows/call
get one objectGET /v1/ontology/objects/asset/{id}2002.255.879.171
eq status = down (10%), first 50 by idquery20054829250
eq status = running, page 2 (cursor)query with the first page’s cursor2002.616.198.7050
and priority = 5, cost >= 990 (0.2%)query20012816619850
starts_with a name prefix (~100 matches at 10^6)query17037371672750
order_by cost desc, top 50 of allquery7975595095050
linked to one site and eq status = runningquery2005.99254035
traverse asset -> siteGET .../asset/{id}/links/site2003.388.479.991
traverse site -> assets, first 50GET .../site/{id}/links/assets2005.60141650

Writes as the owner, 10 s per run:

WriteClientsops/sp50 msp99 mserrors
create an object13022.928.910
create an object88316.95400
create an object32803301520
create a link12244.24110
create a link87798.03410
create a link32617401890
apply an action (1 criterion, 1 update)12263.94110
apply an action (1 criterion, 1 update)864510470
apply an action (1 criterion, 1 update)32554452000

Live-stream catch-up: a new stream with a cursor from before the writes received their 64220 events in 1.35 s (47669 events/s; first after 16 ms).

Revalidation (before the fix): a rules change (max on cost) answered in 994 ms; the maintenance job then rechecked all 319394 assets, every one valid, in 1471 s (217 objects/s).

1,000,000 assets

Before the runs: 1000000 assets, 10000 sites, 995739 links, 137144 events; database 1672 MiB. Bulk top-up took 55.3 s.

Reads, one client, as an agent with view_objects (ms):

ReadCallnp50p99maxrows/call
get one objectGET /v1/ontology/objects/asset/{id}2001.544.044.271
eq status = down (10%), first 50 by idquery2009614025650
eq status = running, page 2 (cursor)query with the first page’s cursor2002.255.185.8050
and priority = 5, cost >= 990 (0.2%)query20017321327750
starts_with a name prefix (~100 matches at 10^6)query4713082219221950
order_by cost desc, top 50 of allquery2722602541254150
linked to one site and eq status = runningquery2005.05111935
traverse asset -> siteGET .../asset/{id}/links/site2002.9610101
traverse site -> assets, first 50GET .../site/{id}/links/assets2006.70141550

Writes as the owner, 10 s per run:

WriteClientsops/sp50 msp99 mserrors
create an object13812.366.040
create an object89395.70400
create an object32779311570
create a link12283.79120
create a link87408.80380
create a link32625401910
apply an action (1 criterion, 1 update)12124.12130
apply an action (1 criterion, 1 update)863710440
apply an action (1 criterion, 1 update)32655381730

Live-stream catch-up: a new stream with a cursor from before the writes received their 67112 events in 1.39 s (48326 events/s; first after 12 ms).

Revalidation (before the fix): a rules change (max on cost) answered in 2480 ms; the maintenance job had rechecked 142600 of 1021024 assets when the run stopped waiting, after 1805 s (79 objects/s).

Esc
↑ ↓ navigate↵ open