Network recording issues

Solutions and workarounds to some of the common issues around network recording in Bug Capture

Network requests and their responses are recorded by default. However, network request recording is inherently imperfect due to complexities outlined below. Necessary trade-offs result in the following limitations:
Response bodies

• Response bodies are only recorded if their payload size is not more than the configured maximum. The default maximum size is 1MB. This default can be changed.
• Response bodies are only recorded if the Content-Type header includes either the term json or text. Bug Capture does not attempt to capture binary information like images, videos, or files like PDFs.
• Response recording does not work with Flutter web apps at the moment.
• Response recording does not work for requests issued by workers.
• Responses might not be recorded if the response came in right before the client loaded a new page, e.g. due to a HTML form submission, or a server-side redirect.

Websockets
Websocket connections are not being recorded at the moment.

Complexities of network recording

Network request recording has more caveats than other recorded data types. There are two reasons for that:

• Frequency and size of network requests
• Browser APIs allowing to access network request information

Frequency and size of network requests

Pages can execute hundreds or even thousands of network requests per minute. They can contain large payloads. Recording all data flowing over the browser’s network can easily accumulate hundreds of Megabytes of data in short time periods. Therefore we have to put reasonable limits in place for the capture of network request recording, frequency and size of network events, in order to avoid these potential problems:

• Slowdowns of the browser/operating system/recorded pages
• Running into storage limits
• Painfully slow session uploads
• An increased likelihood of Bug Capture or the recorded page crashing
  

Browser APIs allowing to access network request information

The WebRequest browser extension API does not allow the capture of network request responses (at least in Chromium based browsers). It’s also only available for browser extensions and hence wouldn’t work for our SDK.

Since response information is likely to contain crucial information for debugging, Bug Capture additionally instruments fetch and XMLHttpRequest, which is the approach taken by client-side monitoring tools such as Sentry.

Just using the fetch and XMLHttpRequest monkey-patching does not allow to record requests issued by the browser (on page load, redirect, form submission,…). For this reason, Bug Capture employs the WebRequest extension API for all request types except for fetch and XMLHttpRequest, and instead records these by instrumenting fetch and XMLHttpRequest. Due to the different data sources, the timing/order of requests can potentially slightly differ from the actual timing/order. Recording all request types with the WebRequest API, and enriching it with the response body information is difficult, because there’s no ID that can be cross-referenced across these different methods.

Change maximum recorded request body size

Depending on how you are using Bug Capture, follow these instructions to change the maximum recorded request body size:

window.birdeatsbug.setOptions({
	/*...otherOptions, */
	recordedEventTypes: {
		/* ...other event types */
		network: {
			maxSizeInBytes: 2000000,
		} /* set maximum recorded response body size to 2mb */,
	},
})

Disable network request recording

Depending on how you are using Bug Capture, follow these instructions to disable network request recording:

window.birdeatsbug.setOptions({
	/*...otherOptions, */
	recordedEventTypes: {
		/* ...other event types */
		network: false /* disable recording of network requests */,
	},
})