Headers

Custom headers allow you to tailor your requests by adding specific HTTP headers to the request. These headers can include standard fields like Accept, Cookie and Referer. Adjusting these headers allows you to control content types, manage session cookies, and handle other use cases.

Please note that in order to make your request look like real ones, BitFetcher adds several headers to all API requests. Eespecially those related to the browser environment, such as User-Agent.

Custom Headers

You might need to forward specific headers to the website that you want to scrape.

In order to forward headers, you must set custom_headers=true and then pass your custom headers.

cURL

curl -G "https://api.bitfetcher.io/v1/" \
  -d api_key=YOUR_API_KEY \
  -d url=YOUR_URL \
  -d custom_headers=true \
  -H "Authorization: Bearer token123" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json"

Custom cookies

You might need to forward specific cookies to the website that you want to scrape.

In order to forward cookies, you must set custom_cookies=true and then pass your cookies through the HTTP Cookie header:

cURL

curl -G "https://api.bitfetcher.io/v1/" \
  -d api_key=YOUR_API_KEY \
  -d url=YOUR_URL \
  -d custom_cookies=true \
  -H "Cookie: sessionId=abc123; userId=12345"

The cookies will be automatically included in all subsequent requests to the same domain, mimicking browser behavior.

Custom cookies are particularly useful in scenarios such as:

• Accessing authenticated content by passing session cookies
• Setting regional or language preferences
• Maintaining personalized settings and preferences
• Testing different user states and configurations

Remember to handle cookies securely and respect the website's terms of service when scraping authenticated content.