Appearance

JavaScript Scenario ​

BitFetcher provides an extensive set of JavaScript Instructions, allowing you to interact with web pages dynamically.

To use JavaScript Scenario, you must include two parameters: js_render to enable headless browser and js_scenario. The js_scenario parameter must be JSON encoded.

Overview ​

JavaScript Scenario are composed of multiple instructions, each representing a specific action to perform on the page. These instructions enable you to click on elements, fill out forms, submit them, or wait for specific elements to appear, providing flexibility for tasks such as clicking the read more buttons or submitting forms.

InstructionTypeDescription
clickstringSpecifies the element to click on.
wait_forstringWaits for the specified element to be present before proceeding.
waitintWaits for a fixed number of milliseconds.
checkstringToggles a checkbox: checks if unchecked, unchecks if already checked.
fill[]stringFills the specified input field(s) with the given values.
select[]stringSelects the given option(s) from a dropdown menu.
scroll_yintScrolls the page vertically by the specified number of pixels.
scroll_xintScrolls the page horizontally by the specified number of pixels.
evaluatestringExecutes a JavaScript expression on the page.
handle_dialogboolHandles a browser dialog (e.g., accept, dismiss, or prompt) automatically

Example ​

Here are some common actions you can perform with JavaScript Scenario.

json
{
  "strict": false,
  "debug": "true",
  "instructions": [
    {"click": "#submit-button"},
    {"wait_for": "#confirmation-message"},
    {"wait": 5000},
    {"check": "#agree-checkbox"},
    {"fill": ["#name-field", "John Doe"]},
    {"select": ["#country-select", "USA"]},
    {"scroll_y": 300},
    {"scroll_x": 0},
    {"evaluate": "return document.title;"}
  ]
}

Strict mode ​

By default, our JavaScript scenario are executed in "strict mode", which means that if an error occurs during the execution of the scenario, the whole scenario will be aborted and an error will be returned.

You can disable this behavior by setting the strict key to false in your scenario.

json
{
  "strict": false,
  "instructions": [
    {"click": "#submit-button"},
    {"scroll_x": 1000},
    {"wait_for": "#confirmation-message"}
  ]
}

Debug mode ​

The debug mode allows you to get more information about the execution of the scenario useful for testing and debugging. When enabled, the response will include a js_scenario_debug field containing debug information for each instruction.

json
{
  "debug": "true",
  "instructions": [
    {"click": "#submit-button"},
    {"scroll_x": 1000},
    {"wait_for": "#confirmation-message"}
  ]
}

For more details, check the API overview JSON Response Debug

json
{
  "body": "<html><body><button id='submit-button'>Submit</button></body></html>",
  "type": "html",
  "status_code": 200,
  "js_scenario_debug": [
    {
      "action": "click",
      "duration": 5,
      "params": {"selector": "#submit-button"},
      "success": true
    },
    {
      "action": "scroll_x",
      "duration": 2,
      "params": {"pixels": 1000},
      "success": false,
      "error": "Element not found"
    }
  ]
}

Click on an element ​

The click action lets you interact with web elements like buttons and links. It's useful for navigating pages, showing hidden content, expanding sections, or moving through pagination.

Often, it's combined with wait_for to handle elements that load later. For example, you might need to click a "Read More" button to see the full content of an article.

json
{
  "instructions": [
    {"click": "#submit-button"}
  ]
}

Wait ​

Wait for a fixed amount of time ​

The wait instruction pauses execution for a specified duration, defined in milliseconds. For example, {"wait": 1000} pauses the script for one second. You can adjust this value based on your needs, with a maximum total allowable wait time of 30 seconds.

This can be useful for ensuring specific actions, such as animations or data loading processes, are given enough time to complete before proceeding with further steps. It's a straightforward way to handle timing issues and ensure all elements are ready for interaction or extraction.

json
{
  "instructions": [
    {"wait": 1000}
  ]
}

Wait for a selector to appear ​

The wait_for instruction pauses the script until a specific element appears on the page, making it ideal for handling delayed content loading in Single Page Applications (SPAs) or dynamic websites.

This ensures that all necessary elements, like data fields or navigation buttons, are fully loaded before further actions are taken. For instance, after clicking a button, you might use wait_for to ensure the next page's key elements are present before proceeding with data extraction. This step is crucial for accurate and complete data retrieval in web scraping

json
{
  "instructions": [
    {"wait_for": ".late-selector"}
  ]
}

Check ​

The check instruction toggles a checkbox on a web page. It can be used to select or deselect checkboxes, such as agreeing to terms and conditions or subscribing to a newsletter.

json
{
  "instructions": [
    {"check": "#agree-checkbox"}
  ]
}

Fill out a form ​

The fill instruction populates input fields with specified values, allowing you to interact with forms on web pages.

This action is useful for automating form submissions, such as login or registration forms, or for entering search queries on websites.

json
{
  "instructions": [
    {"fill": ["input[name='username']", "John Doe"]}
  ]
}

Select an option ​

The select instruction chooses an option from a dropdown menu on a web page.

This action is helpful for interacting with dropdowns, such as selecting a country from a list or choosing a category from a menu.

json
{
  "instructions": [
    {"select": ["select[name='country']", "USA"]}
  ]
}

Scroll the page ​

The scroll_y and scroll_x instructions move the page vertically and horizontally, respectively, by a specified number of pixels.

These actions are useful for navigating long pages, scrolling to specific sections, or triggering lazy loading of content.

Example: Scroll the page 300 pixels down

json
{
  "instructions": [
    {"scroll_y": 300}
  ]
}

Example: Scroll the page 200 pixels to the right

json
{
  "instructions": [
    {"scroll_x": 200}
  ]
}

Evaluate JavaScript ​

The evaluate instruction executes a custom JavaScript expression on the page, allowing you to interact with the DOM or retrieve data.

This action is versatile and can be used for various purposes, such as extracting text content, checking element visibility, or manipulating page elements.

Example: Get the title of the current page

json
{
  "instructions": [
    {"evaluate": "document.title;"}
  ]
}

Handle Dialog ​

The handle_dialog, allows you to handle JavaScript initiated dialog / prompts (alert, confirm, prompt, or onbeforeunload), automatically.

json
{
  "instructions": [
    {"handle_dialog": true}
  ]
}

Example Using JavaScript Scenario ​