# Device Lab methodology > How the Device Lab measures behaviour on real devices: the protocol, expectations recorded in advance, A and S confidence labels, and threats to validity. import LabProvenance from '@components/lab/LabProvenance.astro'; A Device Lab result records measured behaviour on a real device under a published protocol. This is layer 3 in the [OpenPWA Evidence Layers](/about/evidence/), which define the layers and evidence classes. Compatibility-row confidence follows separate rules in [Methodology](/about/methodology/). ## Protocol A run follows a checklist of short tests; the [protocol](https://openpwa.net/research/2026-10-ios18-home-screen-web-apps/data/PROTOCOL.md) sets a limit of five minutes per test. Each test in the checklist lists its expected result, taken from sources, so the operator knows what to look for. In the iOS 18.6.2 baseline the expectations came from vendor documents, browser-compat-data and the site's own pages; for steps added during the run they were written in the notebook before the read ([REPORT.md](https://openpwa.net/research/2026-10-ios18-home-screen-web-apps/data/REPORT.md) §5.3). Not every finding had a prior expectation: the notebook records none for the worker exposure seen in the app in T8. The operator performs the step on the device and the result records what the screen and the page showed, quoting on-screen text in the UI language. A result that differs from the expectation is a finding. Verdicts are `as-expected`, `differs`, `inconclusive` (the step could not tell the alternatives apart) and `could-not-run`. The protocol asks for an environment block once per device per session: device, OS version and build, release channel, Safari version, language and region, changed feature flags, and content blockers, VPN and Private Relay. Tests that need fresh state start with the Home Screen icon removed and the site's data cleared. Corrections are appended to the notebook as new entries; nothing is edited in place. ### Who does what One OpenPWA operator, who owns the device, performs every tap and every Settings change and takes the screenshots and recordings. An AI agent session on a Mac reads page state over USB, through the WebKit remote inspector protocol (and, for a few tab-side values, WebDriver through `safaridriver`), keeps the notebook and analyses the captures. A second agent session may relay the tap list and questions between the operator and the reading agent. Agent sessions also draft these pages. As the [data package README](https://openpwa.net/research/2026-10-ios18-home-screen-web-apps/data/README.md) says, no person signs off each entry. ### Reads A read runs a published, read-only probe in the page and stores its output verbatim as a JSON file, together with (from notebook entry N15 onwards in the baseline) a sidecar recording the read time, the tool hashes and the page that was selected. The probes evaluate feature detects, media queries, geometry and storage contents. They do not request a permission, subscribe, share, install, register a worker, or call `persist()`. State changes come from the operator's taps. ## Confidence labels Published results carry labels identifying the evidence behind them. | Label | Name | What it is | Published | |---|---|---|---| | A | Instrumented read | A published probe's output, read over the WebKit remote inspector or WebDriver and stored verbatim | yes | | S | Screenshot or recording | A capture analysed afterwards: pixel values, frames, capture times | yes | | O | Operator report | The operator's verbal report | not as a Lab result | Labels combine. A+S means an instrumented read and a capture agree. S+O means a capture with the operator's report as context, for example when the pixels are measured but which app was on screen rests partly on what the operator said. A result is published with its weakest essential component named. O alone does not qualify for a Lab result row. The study report and notebook do contain O-only claims, each labelled O; REPORT.md §11 lists them, and none is used as the basis of a Lab result. Retracted readings remain in the notebook with a retraction label and are excluded from citations. A measured result applies to one device, OS build, and date. Specify "on iOS 18.6.2 (22G100)"; "on iOS 18" would extend the claim beyond the measured build. A different build gets its own result row. ## Threats to validity The iOS 18.6.2 notebook and report record these limitations. They apply to the Lab results until a later run resolves them. - The published results describe one iPhone 16 Pro (iPhone17,1) on 22G100, measured on 2026-10-20. No other model, no iPadOS and no other build has been measured. - One operator ran the tests. Several verbal reports in the baseline were wrong in four ways: a step not performed, a term misunderstood, an answer that did not match the question, and a false memory that no prompt had appeared. Each was caught by a read or a recording that contradicted it. Steps with no independent record could hide the same kind of error. - AI agent sessions took the reads, kept the notebook, and analysed the captures. One relay message from the coordinating session paraphrased a planned step and was briefly treated as an operator report before it was caught. - The phone was not wiped. Other sites' Home Screen apps and Safari state existed, a Safari Web Extension of unknown identity was installed, and feature flags, VPN and iCloud Private Relay were not recorded. - Old builds cannot be reinstalled. Once the phone moves to iOS 27, nothing on 18.6.2 can be measured again on it. The published reads, captures and hashes are the only way to audit those results. - The baseline has no hash snapshot of every harness file. Run-time snapshots exist for `lab.js`, `sw.js` and the T7 files; the other files are identified by their hash at the deployed commit. One deploy during the run replaced `lab.js`, and the affected reads record which module they loaded (see [Harness v1](/lab/harness/v1/)). - A read runs JavaScript in the page and adds a hidden element and a temporary property. Only foreground pages can be read, and a ten-minute outage of the inspector's page list removed agent reads from some T4 and T11 steps. - Colour values come from screenshots; the screenshot pipeline was not checked against a measured display. - The baseline took about three hours against a one-hour budget, because steps were added to tell hypotheses apart. Their expectations were written before the read, but they were not in the original checklist. ## What has not been tested The baseline did not test real push delivery (no push server; only local notifications from `showNotification()`), the `clients.openWindow()` path of `notificationclick`, Add to Home Screen from a third-party browser (T5b), the `sbs-default` and `sbs-black` status-bar styles (never installed), Dark appearance beyond an operator report, server-set, `HttpOnly`, partitioned and third-party cookies, and a service worker scoped to the root of a site. Android, desktop Chromium, Firefox and Samsung Internet have no Lab results. The [roadmap](/lab/roadmap/) lists which of these are planned.