Use executors to test screen reader interactions

Implement screen reader automation

Integrate the browserstack_executor command into your test scripts to enable or disable the screen reader, simulate screen reader gestures, or capture spoken output.

Enable or disable the screen reader

Use the screenReader action to enable or disable TalkBack.

• To enable TalkBack:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReader\",\"arguments\": {\"enable\" : \"true\"}}");
  

• To disable TalkBack:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReader\",\"arguments\": {\"enable\" : \"false\"}}");
  

Simulate screen reader gestures

Use the screenReaderGesture action to simulate common screen reader gestures, such as navigate, activate, or scroll.

• To simulate a navigate next gesture:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReaderGesture\",\"arguments\": {\"gesture\" : \"navigate_next\"}}");
  

• To simulate an activate gesture:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReaderGesture\",\"arguments\": {\"gesture\" : \"activate_item\"}}");
  

• To simulate a scroll down gesture:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReaderGesture\",\"arguments\": {\"gesture\" : \"scroll_down\"}}");
  

List of supported gestures

The following gestures are supported for TalkBack automation:

Capture spoken output

Use the screenReaderSpokenDescription action to capture the spoken output of a specific UI element. Identify the element in one of the following ways:

• resourceId: the resource ID of the element.
• rect: the coordinates of the element’s bounding box.

Use coordinates when the element has no resource ID. Elements often have no resource ID in apps built with Jetpack Compose, where a semantics tree describes the UI instead. If you don’t specify either identifier, the command returns the spoken output for all elements captured by the screen reader.

• To capture the spoken output by resource ID:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReaderSpokenDescription\",\"arguments\": {\"resourceId\" : \"<your-element-resource-id>\"}}")
  

• To capture the spoken output by coordinates:

  driver.execute_script("browserstack_executor: {\"action\":\"screenReaderSpokenDescription\",\"arguments\": {\"rect\" : {\"x\": <x-coordinate>, \"y\": <y-coordinate>, \"width\": <width>, \"height\": <height>}}}")
  

The command groups the spoken output by the identifier you use. The following example shows the output when you use both identifiers to target the same element:

{
  "spoken_description": {
    "by_resource_id": {
      "com.example.app:id/sign_in": ["Sign In, Button, Double-tap to activate"]
    },
    "by_rect": {
      "70 1171 1010 1297": ["Sign In, Button, Double-tap to activate"]
    }
  }
}

Output you request by resource ID appears under by_resource_id. Output you request by coordinates appears under by_rect, listed under the element’s bounds in the form left top right bottom. You can log this output, use it for assertions, or save it to a file for offline review.

Add accessibility assertions

With the captured screen reader data, you can add assertions to your test scripts. This lets you verify critical accessibility aspects:

• Focusability: Verify that all relevant UI elements receive focus and are reachable by the screen reader.
• Spoken output verification: Check that elements expose the correct accessibility metadata to screen readers, such as labels, roles, and hints, and that the actual spoken output matches your expected accessibility label or announcement.
• Traversal order: Record the screen reader traversal sequence. Compare it to the Accessibility Tree or a visual test to confirm the focus order follows a logical reading flow.

Enable assertions

You can enable assertions in your test scripts by using the browserstack_executor command with the screenReaderSpokenDescription action to capture the spoken output of a specific UI element and verify it against your expected values.

To enable assertions using resourceId:

driver.execute_script("browserstack_executor: {\"action\":\"screenReaderSpokenDescription\",\"arguments\": {\"resourceId\" : \"<your-element-resource-id>\"}}");

You can also enable assertions with rect by passing the element’s coordinates.

Example assertion

• Spoken output verification:
  The following example captures the spoken output of an element by its resource ID and asserts that the output matches the expected description:

  print("Getting spoken text of element\n")
    out = driver.execute_script("browserstack_executor: {\"action\":\"screenReaderSpokenDescription\",\"arguments\": {\"resourceId\" : \"<your-element-resource-id>\"}}");
    expected_description = "Master Switch, ON"
    actual_description = out['spoken_description']['by_resource_id']['<your-element-resource-id>'][0]
  
    if actual_description == expected_description
      puts "Assertion Passed: Spoken description is correct."
    else
      raise "Assertion Failed: Expected '#{expected_description}', but got '#{actual_description}'"
    end
  

Prerequisites

Add the voiceOver capability to your test session to allocate a VoiceOver-enabled device for your tests. If you do not add the capability, screen reader executors will not work in your test session.

Use the format that matches your existing setup:

{
  "bstack:options": {
    "deviceName": "iPhone 16",
    "osVersion": "18",
    "voiceOver": true
  }
}

{
  "browserstack.deviceName": "iPhone 16",
  "browserstack.osVersion": "18",
  "browserstack.voiceOver": true
}

Implement screen reader automation

Integrate the browserstack_executor command into your test scripts to enable or disable the screen reader, simulate screen reader gestures, or capture spoken output. You can run only one VoiceOver executor command at a time in a session. Parallel calls return an error.

Enable or disable the screen reader

Use the voiceOverToggle action to enable or disable VoiceOver. The state argument accepts on or off, and isn’t case-sensitive. VoiceOver isn’t enabled at session start, so enable it before you call any other VoiceOver executor.

• To enable VoiceOver:

  driver.execute_script("browserstack_executor: {\"action\": \"voiceOverToggle\", \"arguments\": {\"state\": \"on\"}}");
  

• To disable VoiceOver:

  driver.execute_script("browserstack_executor: {\"action\": \"voiceOverToggle\", \"arguments\": {\"state\": \"off\"}}");
  

Simulate screen reader gestures

Use the voiceOverGesture action to simulate common screen reader gestures, such as navigate, activate, or scroll. Gesture names are case-sensitive.

• To simulate a navigate next gesture:

  driver.execute_script("browserstack_executor: {\"action\": \"voiceOverGesture\", \"arguments\": {\"gesture\": \"navigate_next\"}}");
  

• To simulate an activate gesture:

  driver.execute_script("browserstack_executor: {\"action\": \"voiceOverGesture\", \"arguments\": {\"gesture\": \"activate_item\"}}");
  

• To simulate a scroll down gesture:

  driver.execute_script("browserstack_executor: {\"action\": \"voiceOverGesture\", \"arguments\": {\"gesture\": \"scroll_down\"}}");
  

Verification

• If successful, the command returns the gesture that was performed:

  { "success": true, "gesture": "navigate_next" }
  

List of supported gestures

The following gestures are supported for VoiceOver automation:

Capture spoken output

Use the getVoiceOverElement action to capture the element VoiceOver is focused on and its spoken output. This action takes no arguments.

• To capture the focused element and its spoken output:

  result = driver.execute_script("browserstack_executor: {\"action\": \"getVoiceOverElement\", \"arguments\": {}}");
  

Verification

• The command returns the focused element’s accessibility attributes and the most recent VoiceOver announcement as plain text:

  {
  "element": {
    "elementClass": "UIButton",
    "label": "Send",
    "value": "",
    "hint": "Double tap to send message",
    "identifier": "send_button",
    "traits": 8589934593,
    "frame": { "x": 300, "y": 700, "width": 60, "height": 44 }
  },
  "spokenDescription": "Send. Button. Double tap to send message."
  }
  

The response has the following fields:

Considerations

• The two parts of the response, element and spokenDescription, are independent. If element details can’t be captured, element is an empty object. If the spoken output can’t be captured, spokenDescription is an empty string. The command doesn’t fail on a partial capture, so assert only on the fields you need.
• This action takes no element identifier unlike Android’s screenReaderSpokenDescription. The action always reads the currently focused element.

Add accessibility assertions

With the captured screen reader data, you can add assertions to your test scripts. This lets you verify critical accessibility aspects:

• Focusability: Verify that all relevant UI elements receive focus and are reachable by the screen reader.
• Spoken output verification: Check that elements expose the correct accessibility metadata to screen readers, such as labels, roles, and hints, and that the actual spoken output matches your expected accessibility label or announcement.
• Traversal order: Record the screen reader traversal sequence. Compare it to the Accessibility Tree or a visual test to confirm the focus order follows a logical reading flow.

Enable assertions

You can enable assertions in your test scripts by using the browserstack_executor command with the getVoiceOverElement action to capture the spoken output of the focused element and verify it against your expected values.

To enable assertions:

result = driver.execute_script("browserstack_executor: {\"action\": \"getVoiceOverElement\", \"arguments\": {}}");

Example assertion

• Spoken output verification:
  The following example captures the spoken output of an element and asserts that it matches the expected description:

  driver.execute_script('browserstack_executor: {"action": "voiceOverGesture", "arguments": {"gesture": "navigate_next"}}')
  result = driver.execute_script('browserstack_executor: {"action": "getVoiceOverElement", "arguments": {}}')
  expected = "Send. Button. Double tap to send message."
  actual = result["spokenDescription"]
  assert actual == expected, f"Assertion Failed: expected '{expected}', but got '{actual}'"
  print("Assertion Passed: spoken description is correct.")
  

Error reference

The VoiceOver executor commands on iOS return the following errors. Each response includes an error code and a descriptive message.