Requests Debugger Tool

A guide to identify and solve network related issues in requests or responses during BrowserStack test session, using the Requests Debugger tool.

Introduction

If you are facing network related issues while trying to run your tests on BrowserStack, you can use the Requests Debugger tool (a proxy and reverse proxy tool) that assists in identifying and solving errors on your test setup. This document will help you with:

  1. Common network issues
  2. Installing and running the Requests Debugger
  3. Extracting logs

Common network issues

The network logs will be captured at your end in the following scenarios:

  • If you are unable to reach BrowserStack while initiating the tests due to an invalid proxy configuration.
  • If your requests are being dropped due to the following reasons:
    • Network/Proxy blockage while executing high number of parallels.
    • Your proxy setup tends to modify the request/response. This sometimes leads to an unsuccessful scenario.
    • Commands from your script are unable to reach BrowserStack within the timeout threshold, thus leading to BROWSERSTACK_IDLE_TIMEOUT
  • If you need more information/logs to reach out to your internal network team.

Installing and running the Requests Debugger

  1. Download the executable for your platform:
  2. Extract and run the executable from your terminal or cmd:
    • Linux: ./RequestsDebugger-Linux-x86/64 <args>
    • Mac: ./RequestsDebugger-Mac <args>
    • Windows: RequestsDebugger.exe <args>
Note: The executable for Mac is unsigned. Thus, you might see security popup whether to allow the execution of the tool or not.
You can allow it from your System PreferencesSecurity & PrivacyGeneral tab.

You can use the following <args> with Requests Debugger tool to modify the default settings and also provide the external/upstream proxy (if any) details:

Flags Details
--reverse-proxy-port <port> Port on which the Requests Debugger tool’s reverse proxy will run.
Default: 9688
--port <port> Port on which the Requests Debugger tool’s proxy will run.
Default: 9687
--scheme <https/http> Scheme for requests from tool to BrowserStack.
Default: https
--proxy-host <hostname> Hostname of the external/upstream proxy
--proxy-port <port> Port of the external/upstream proxy.
Default: 3128 (if hostname is provided)
Example: ./RequestsDebugger-Mac --proxy-host <hostname of the proxy> --proxy-port <port of the proxy>
--proxy-user <username> Username for authentication in the external/upstream proxy
--proxy-pass <password> Password for authentication in the external/upstream Proxy.
Default: empty (if username is provided)
Example: ./RequestsDebugger-Mac --proxy-host <hostname of the proxy> --proxy-port <port of the proxy> --proxy-user <auth username of the proxy> --proxy-pass <auth password of the proxy>
--retry-delay <milliseconds> Set the delay after which the failed request will be retried.
Note: Failure of a request is not judged on its response code. It depends on the error which occurs while the tool makes the request. For example, ENOTFOUND.
Default: 1000ms
Example: ./RequestsDebugger-Mac --retry-delay 4000
--request-timeout <milliseconds> Hard timeout for killing any request fired from the tool while waiting for its response, also an error response will be sent back to the client.
Default: 260000ms.
Example: ./RequestsDebugger-Mac --request-timeout 30000
--logs-path <relative/absolute path> Directory where the RequestsDebuggerLogs folder will be created for storing logs.
Default: Current working directory
Example: ./RequestsDebugger-Mac --logs-path /path/to/your/directory
--del-logs Deletes any existing logs from the RequestsDebuggerLogs/ directory and initializes new files for logging
Example: ./RequestsDebugger-Mac --del-logs
--help Help from the Requests Debugger tool
--version Version of the Requests Debugger tool
Note: Requests Debugger can only intercept HTTP requests. Thus, if remote URL contains https during WebDriver initialization, change it to http.

The Requests Debugger tool can be used in either of the following two modes:

  1. Reverse proxy mode
  2. Proxy mode

Reverse proxy mode

To use tool as reverse proxy, you will have to replace hub-cloud.browserstack.com in hub url with localhost:9688 in you test script.

Proxy mode

To use tool as proxy, use the language specific instructions in you test script as shown below:

Set the environment variables for host and port before initializing the WebDriver

// default port: 9687
System.getProperties().put("http.proxyHost", "localhost");
System.getProperties().put("http.proxyPort", "9687");
// Initialize WebDriver

Set the WebDriver Proxy

// default port: 9687
var driver = new webdriver.Builder()
  .usingServer('http://hub-cloud.browserstack.com/wd/hub')
  .withCapabilities(capabilities)
  .usingWebDriverProxy("http://localhost:9687")
  .build();
  • For client bindings 3.10 and above, add this in your test script:
    HttpCommandExecutor commandExecutor = new HttpCommandExecutor(new Uri("http://BROWSERSTACK_USERNAME:BROWSERSTACK_ACCESS_KEY@hub-cloud.browserstack.com/wd/hub/"), TimeSpan.FromSeconds(60));
    commandExecutor.Proxy = new WebProxy("http://localhost:9687", false);
    // add caps, for e.g.
    // ChromeOptions options = new ChromeOptions();
    // options.AddAdditionalCapability("browser", "Chrome",true);
    IWebDriver driver = new RemoteWebDriver(commandExecutor, options.ToCapabilities());
    
  • Add these proxy parameters in your project config file (for client bindings till 3.9.1), i.e. app.config
<!--default port: 9687-->
<?xml version="1.0" encoding="utf-8"?>
<configuration>  
  <system.net>  
    <defaultProxy enabled="true">  
      <proxy  
        proxyaddress="http://localhost:9687"  
        bypassonlocal="false"  
      />  
    </defaultProxy>  
  </system.net>  
</configuration>
$web_driver = RemoteWebDriver::create(HUB_URL, $capabilities, null, null, 'localhost', 9687);

Environment variables can’t be used since Python doesn’t read from them. It requires modifying the client binding code. Follow the below steps to modify the client binding code:

  • Locate the client binding code in your system, that is, search for selenium folder in python/site-packages. Refer the link to search for python modules in your system.
  • Navigate to selenium/webdriver/remote/ directory and open remote_connection.py file in your editor.
  • Search for the line http = urllib3.PoolManager(timeout=self._timeout) and change it to http = urllib3.ProxyManager('localhost:9687').
  • Save the file. Now your test script shall route its requests via the Requests Debugger tool.

Set environment variable in your test script before initializing the WebDriver

# default port: 9687
ENV['http_proxy'] = 'localhost:9687'
# Initialize WebDriver
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->proxy(['http'], 'http://localhost:9687');
my $driver = Selenium::Remote::Driver->new('ua'=>$ua);

Logs

Once you run your script, the following logs will be generated within RequestsDebuggerLogs/ directory. You can either mail us the logs or upload to Google Drive, OneDrive, Dropbox or any other storage provider.

  • It includes the logs of requests being passed via the Requests Debugger tool.
  • Example:
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Request Start] [INFO] POST http://<URL>/wd/hub/status, {"headers":{"......."}}
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Request End] [INFO] POST http://<URL>/wd/hub/status, {"data":"{\"key\": \"value\"}"}
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Tool Request - Retries Left: 1] [INFO] POST https://<URL>/wd/hub/status, {"method":"POST","headers":{".......","Proxy-Authorization":"Basic AaBbCcDdEeFfGg==","X-Requests-Debugger":"2::UNIQUE_IDENTIFIER"},"host":"<EXTERNAL_PROXY_URL>","port":"3130","path":"http://<URL>/wd/hub/status","data":"{\"key\": \"value\"}"}
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Tool Request - Retries Left: 1] [ERROR] POST https://<URL>/wd/hub/status, {"errorMessage":"Error: getaddrinfo ENOTFOUND <EXTERNAL_PROXY_URL> <EXTERNAL_PROXY_URL>:3130"}
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Tool Request - Retries Left: 0] [ERROR] POST https://<URL>/wd/hub/status, {"errorMessage":"Error: getaddrinfo ENOTFOUND <EXTERNAL_PROXY_URL> <EXTERNAL_PROXY_URL>:3130"}
TIMESTAMP_IN_UTC [#2::UNIQUE_IDENTIFIER] [Response End] [ERROR] POST http://<URL>/wd/hub/status, Status Code: 500, {"message":"Error: getaddrinfo ENOTFOUND <EXTERNAL_PROXY_URL> <EXTERNAL_PROXY_URL>:3130. Request Failed At Requests Debugger","error":"Request Failed At Requests Debugger"}
Tags Meaning
TIMESTAMP_IN_UTC Timestamp of each event in UTC.
#2 This represents the nth request which the tool served from the time it was started.
UNIQUE_IDENTIFIER uuid for the request. This is used in X-Requests-Debugger custom header for usage and debugging at BrowserStack’s end.
Request Start Client request entering the Requests Debugger tool.
Request End This is logged when the client request is finished, that is, in case of POST request, when the client request has pushed all the data.
Tool Request - Retries Left: X Request which was fired from the tool for the respective client request. It also mentions the retries left for the request. Max retries = 1. A request is retried in case it fails at the tool itself, i.e ENOTFOUND etc.
Response End This specifies the final response sent to the client.
  • It consists logs of all the connectivity checks which are performed in terms of reachability with BrowserStack components, that is, Hub and Rails.
  • The report includes HTTP and HTTPS requests to BrowserStack components and their status.
  • New checks are appended whenever a request fails.
  • It consists of the load and usage stats of user’s CPU while booting up the Requests Debugger tool. It lists down the top 10 processes which were using the highest CPU at that moment. For windows, it only includes the loadPercentage.
  • It consists of the memory stats of the user’s machine while booting up the Requests Debugger tool. It includes:
    • Total Memory
    • Free Memory
    • Used Memory
    • Swap Total
    • Swap Free
    • Swap Used
  • It consists the list of connections/sockets being established/listening via the user’s machine.
  • It also includes the report of ping checks with Hub and Rails.
  • New stats are appended whenever a request fails.
  • It includes any unexpected errors that occur while using the tool.
Note: You can see the Requests Debugger open source repository for more information.

We're sorry to hear that. Please share your feedback so we can do better






Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback

Is this page helping you?

Yes
No

We're sorry to hear that. Please share your feedback so we can do better






Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback!

Talk to automation expert