Chapter 7: Detailed Web Automation Techniques

Automate browser-based activities and web applications. Create web automation bots using Selenium with real working examples that can run locally.

SeleniumBrowser BotsWeb FormsXPath & CSSAutomated Testing
Open
Websites
Fill
Forms
Click
Buttons
Extract
Data

7.1 Chapter Overview

Web automation means using a bot to control a browser and perform browser-based activities such as opening websites, filling forms, clicking buttons, extracting text, downloading files and testing web applications.

Many business systems are now web-based. HR portals, finance systems, ERP dashboards, online forms, training portals and customer service systems are commonly accessed through browsers. Web automation helps reduce repetitive browser work and improves accuracy.

Learning Outcome: By the end of this chapter, learners should be able to create web automation bots that open websites, locate page elements, fill forms, click buttons, extract data and test browser-based applications.
Python Script
Selenium WebDriver
Chrome / Edge Browser
Web Application
Result / Report

7.2 Learning Objectives

  • Understand how browser automation works.
  • Install and configure Selenium for web automation.
  • Open websites and navigate pages using Python.
  • Find web elements using ID, name, class, CSS selector and XPath.
  • Fill text fields, select dropdowns and click buttons.
  • Use explicit waits to handle dynamic pages.
  • Extract data from tables and web pages.
  • Automate a complete local web form as a working model.
  • Create basic logs and screenshots for web automation bots.

7.3 Selenium Setup

Selenium is a Python library used to automate browsers such as Chrome, Edge and Firefox.

Install Selenium

pip install selenium

Check Installation

import selenium

print("Selenium installed successfully")
print(selenium.__version__)

Open a Website

from selenium import webdriver

driver = webdriver.Chrome()

driver.get("https://perakskills.com")

print("Page Title:", driver.title)

driver.quit()
How to Test: Run this script on a local computer with Google Chrome installed. A browser window should open, load the website, print the title, and close.

7.4 Web Automation Use Cases

Use CaseManual WorkAutomation Bot Action
Online registrationStaff enters names and details one by one.Bot reads a spreadsheet and fills web forms.
Portal checkingUser logs in and checks status daily.Bot opens portal and extracts status.
Website testingTester manually checks forms and buttons.Bot tests forms automatically.
Report downloadUser downloads daily reports from web system.Bot logs in and downloads files.
Data extractionUser copies table data manually.Bot reads table and saves CSV.

7.5 Understanding Web Elements and Selectors

A web automation bot must identify page elements before it can interact with them. Common elements include input boxes, buttons, dropdowns, links and tables.

Selector TypeBest UseExample
IDMost reliable when available.driver.find_element(By.ID, "studentName")
NameUseful for form fields.driver.find_element(By.NAME, "email")
Class NameUseful but may match many items.driver.find_element(By.CLASS_NAME, "btn")
CSS SelectorFlexible and powerful.driver.find_element(By.CSS_SELECTOR, "#submitBtn")
XPathUseful for complex page structures.driver.find_element(By.XPATH, "//button[text()='Submit']")

Example: Find Element by ID

from selenium.webdriver.common.by import By

name_box = driver.find_element(By.ID, "studentName")
name_box.send_keys("PDTC Student")

7.6 Working Example: Fill a Web Form

This example fills a local test form. The downloadable example package contains the full working files.

HTML Form Example

<input id="studentName" type="text">
<input id="courseName" type="text">
<button id="submitBtn">Submit</button>

Selenium Automation Script

from selenium import webdriver
from selenium.webdriver.common.by import By
from pathlib import Path
import time

file_path = Path("web_automation_test_form.html").resolve()

driver = webdriver.Chrome()
driver.get(file_path.as_uri())

driver.find_element(By.ID, "studentName").send_keys("PDTC Student")
driver.find_element(By.ID, "courseName").send_keys("Certified Industrial Automation Professional")
driver.find_element(By.ID, "email").send_keys("student@example.com")

driver.find_element(By.ID, "submitBtn").click()

time.sleep(2)

result = driver.find_element(By.ID, "result").text
print(result)

driver.quit()
Expected Output:
Registration submitted successfully for PDTC Student.

7.7 Dropdown Automation

Selenium can select dropdown values using the Select class.

from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By

dropdown = Select(driver.find_element(By.ID, "department"))

dropdown.select_by_visible_text("Automation")

print("Dropdown selected successfully")

7.8 Checkbox and Radio Button Automation

from selenium.webdriver.common.by import By

checkbox = driver.find_element(By.ID, "agree")

if not checkbox.is_selected():
    checkbox.click()

radio = driver.find_element(By.ID, "modeOnline")
radio.click()

print("Checkbox and radio button selected")

7.9 Handling Dynamic Pages with Waits

Some web pages load slowly or display elements after a delay. Using fixed sleep is simple but not always reliable. Explicit wait is better.

Bad Practice: Fixed Delay Only

import time

time.sleep(5)

driver.find_element(By.ID, "submitBtn").click()

Better Practice: Explicit Wait

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

submit_button = wait.until(
    EC.element_to_be_clickable((By.ID, "submitBtn"))
)

submit_button.click()
Best Practice: Use explicit waits when automating real websites because page loading time can change.

7.10 Extracting Data from Web Tables

Many portals display information in tables. Selenium can extract rows and columns from HTML tables.

from selenium.webdriver.common.by import By

rows = driver.find_elements(By.CSS_SELECTOR, "#studentTable tbody tr")

for row in rows:
    columns = row.find_elements(By.TAG_NAME, "td")
    values = [col.text for col in columns]
    print(values)

Save Extracted Table to CSV

import csv
from selenium.webdriver.common.by import By

rows = driver.find_elements(By.CSS_SELECTOR, "#studentTable tbody tr")

with open("extracted_students.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["Name", "Course", "Status"])

    for row in rows:
        columns = row.find_elements(By.TAG_NAME, "td")
        values = [col.text for col in columns]
        writer.writerow(values)

print("Table exported to extracted_students.csv")

7.11 Taking Screenshots During Web Automation

Screenshots are useful for proof, audit trail and error handling.

from datetime import datetime

timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")

driver.save_screenshot("web_automation_" + timestamp + ".png")

print("Screenshot saved successfully")

7.12 Downloading Files with Browser Automation

Some web automation projects require downloading reports. Browser settings can control download location.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import os

download_folder = os.path.abspath("downloads")

options = Options()
prefs = {
    "download.default_directory": download_folder,
    "download.prompt_for_download": False
}
options.add_experimental_option("prefs", prefs)

driver = webdriver.Chrome(options=options)

driver.get("file:///C:/path/to/download_page.html")

driver.find_element(By.ID, "downloadReport").click()

print("Download clicked. Check folder:", download_folder)
Note: File download behavior depends on browser settings, file type and site security rules.

7.13 Web Automation Error Handling

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from datetime import datetime

driver = webdriver.Chrome()

try:
    driver.get("https://example.com")

    button = driver.find_element(By.ID, "submitBtn")
    button.click()

except NoSuchElementException:
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    driver.save_screenshot("error_" + timestamp + ".png")
    print("Element not found. Screenshot saved.")

finally:
    driver.quit()

7.14 Complete Web Automation Bot Workflow

1Open Browser
2Load Web Page
3Find Elements
4Enter Data
5Click Submit
6Capture Result
Web Automation Bot = Browser + Selectors + Actions + Waits + Validation + Logs

7.15 Downloadable Working Example

A complete working model is included as a separate downloadable ZIP file. It contains:

  • web_automation_test_form.html - local test web form
  • run_web_automation_bot.py - Selenium bot that fills the form
  • README.txt - step-by-step instructions
  • ⬇ Download Working Example ZIP
Testing Steps: Download the ZIP file, extract it, run pip install selenium, then run python run_web_automation_bot.py.

7.16 Interactive Form Demo Inside This Chapter

This form simulates what a Selenium bot can automate.

Click Submit Demo Form.

7.17 Best Practices for Web Automation

Best PracticeReason
Use stable IDs when availableID selectors are usually reliable.
Use explicit waitsPrevents failures from slow-loading elements.
Take screenshots on errorsHelps troubleshoot automation failures.
Do not store passwords in codeProtects security and compliance.
Use test sites before live systemsPrevents accidental business data changes.
Log automation resultsCreates audit trail and performance record.
Handle exceptionsMakes the bot more reliable.

7.18 Practical Activities

Activity 1: Open Website

Use Selenium to open https://perakskills.com and print the page title.

Activity 2: Local Form Automation

Use the downloadable test form and automate data entry using Selenium.

Activity 3: Screenshot Evidence

Create a script that takes a screenshot after submitting a form.

Activity 4: Table Extraction

Create an HTML table and use Selenium to extract the table into CSV.

Mini Project

Create a web automation bot that fills a form, submits it, captures the result text and saves a screenshot.

7.19 Interactive Final Assessment Quiz

Each correct answer gives +1 mark.
Each wrong answer gives -0.5 mark.

1. Selenium is used for browser automation.

2. Which method opens a website in Selenium?

3. ID selectors are usually reliable when available.

4. Which Selenium command types into a text field?

5. Explicit waits help handle dynamic web pages.

6. Which selector can locate an element by CSS?

7. Screenshots can be saved during web automation.

8. It is safe to test automation directly on live business systems without approval.

9. Selenium can extract text from web tables.

10. Error handling improves web automation reliability.

Your Score: 0

7.20 Chapter Summary

In this chapter, learners studied detailed web automation techniques using Selenium. They learned how to open websites, locate elements, fill forms, click buttons, use waits, select dropdowns, extract table data, take screenshots, handle errors and build a complete working web automation bot.

Remember: A reliable web automation bot uses stable selectors, explicit waits, proper error handling, screenshots and safe testing practices.