Chapter 7: Detailed Web Automation Techniques
Automate browser-based activities and web applications. Create web automation bots using Selenium with real working examples that can run locally.
Websites
Forms
Buttons
Data
7.1 Chapter Overview
Web automation means using a bot to control a browser and perform browser-based activities such as opening websites, filling forms, clicking buttons, extracting text, downloading files and testing web applications.
Many business systems are now web-based. HR portals, finance systems, ERP dashboards, online forms, training portals and customer service systems are commonly accessed through browsers. Web automation helps reduce repetitive browser work and improves accuracy.
7.2 Learning Objectives
- Understand how browser automation works.
- Install and configure Selenium for web automation.
- Open websites and navigate pages using Python.
- Find web elements using ID, name, class, CSS selector and XPath.
- Fill text fields, select dropdowns and click buttons.
- Use explicit waits to handle dynamic pages.
- Extract data from tables and web pages.
- Automate a complete local web form as a working model.
- Create basic logs and screenshots for web automation bots.
7.3 Selenium Setup
Selenium is a Python library used to automate browsers such as Chrome, Edge and Firefox.
Install Selenium
pip install selenium
Check Installation
import selenium
print("Selenium installed successfully")
print(selenium.__version__)Open a Website
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://perakskills.com")
print("Page Title:", driver.title)
driver.quit()7.4 Web Automation Use Cases
| Use Case | Manual Work | Automation Bot Action |
|---|---|---|
| Online registration | Staff enters names and details one by one. | Bot reads a spreadsheet and fills web forms. |
| Portal checking | User logs in and checks status daily. | Bot opens portal and extracts status. |
| Website testing | Tester manually checks forms and buttons. | Bot tests forms automatically. |
| Report download | User downloads daily reports from web system. | Bot logs in and downloads files. |
| Data extraction | User copies table data manually. | Bot reads table and saves CSV. |
7.5 Understanding Web Elements and Selectors
A web automation bot must identify page elements before it can interact with them. Common elements include input boxes, buttons, dropdowns, links and tables.
| Selector Type | Best Use | Example |
|---|---|---|
| ID | Most reliable when available. | driver.find_element(By.ID, "studentName") |
| Name | Useful for form fields. | driver.find_element(By.NAME, "email") |
| Class Name | Useful but may match many items. | driver.find_element(By.CLASS_NAME, "btn") |
| CSS Selector | Flexible and powerful. | driver.find_element(By.CSS_SELECTOR, "#submitBtn") |
| XPath | Useful for complex page structures. | driver.find_element(By.XPATH, "//button[text()='Submit']") |
Example: Find Element by ID
from selenium.webdriver.common.by import By
name_box = driver.find_element(By.ID, "studentName")
name_box.send_keys("PDTC Student")7.6 Working Example: Fill a Web Form
This example fills a local test form. The downloadable example package contains the full working files.
HTML Form Example
<input id="studentName" type="text"> <input id="courseName" type="text"> <button id="submitBtn">Submit</button>
Selenium Automation Script
from selenium import webdriver
from selenium.webdriver.common.by import By
from pathlib import Path
import time
file_path = Path("web_automation_test_form.html").resolve()
driver = webdriver.Chrome()
driver.get(file_path.as_uri())
driver.find_element(By.ID, "studentName").send_keys("PDTC Student")
driver.find_element(By.ID, "courseName").send_keys("Certified Industrial Automation Professional")
driver.find_element(By.ID, "email").send_keys("student@example.com")
driver.find_element(By.ID, "submitBtn").click()
time.sleep(2)
result = driver.find_element(By.ID, "result").text
print(result)
driver.quit()Registration submitted successfully for PDTC Student.
7.7 Dropdown Automation
Selenium can select dropdown values using the Select class.
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
dropdown = Select(driver.find_element(By.ID, "department"))
dropdown.select_by_visible_text("Automation")
print("Dropdown selected successfully")7.8 Checkbox and Radio Button Automation
from selenium.webdriver.common.by import By
checkbox = driver.find_element(By.ID, "agree")
if not checkbox.is_selected():
checkbox.click()
radio = driver.find_element(By.ID, "modeOnline")
radio.click()
print("Checkbox and radio button selected")7.9 Handling Dynamic Pages with Waits
Some web pages load slowly or display elements after a delay. Using fixed sleep is simple but not always reliable. Explicit wait is better.
Bad Practice: Fixed Delay Only
import time time.sleep(5) driver.find_element(By.ID, "submitBtn").click()
Better Practice: Explicit Wait
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
submit_button = wait.until(
EC.element_to_be_clickable((By.ID, "submitBtn"))
)
submit_button.click()7.10 Extracting Data from Web Tables
Many portals display information in tables. Selenium can extract rows and columns from HTML tables.
from selenium.webdriver.common.by import By
rows = driver.find_elements(By.CSS_SELECTOR, "#studentTable tbody tr")
for row in rows:
columns = row.find_elements(By.TAG_NAME, "td")
values = [col.text for col in columns]
print(values)Save Extracted Table to CSV
import csv
from selenium.webdriver.common.by import By
rows = driver.find_elements(By.CSS_SELECTOR, "#studentTable tbody tr")
with open("extracted_students.csv", "w", newline="", encoding="utf-8") as file:
writer = csv.writer(file)
writer.writerow(["Name", "Course", "Status"])
for row in rows:
columns = row.find_elements(By.TAG_NAME, "td")
values = [col.text for col in columns]
writer.writerow(values)
print("Table exported to extracted_students.csv")7.11 Taking Screenshots During Web Automation
Screenshots are useful for proof, audit trail and error handling.
from datetime import datetime
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
driver.save_screenshot("web_automation_" + timestamp + ".png")
print("Screenshot saved successfully")7.12 Downloading Files with Browser Automation
Some web automation projects require downloading reports. Browser settings can control download location.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import os
download_folder = os.path.abspath("downloads")
options = Options()
prefs = {
"download.default_directory": download_folder,
"download.prompt_for_download": False
}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=options)
driver.get("file:///C:/path/to/download_page.html")
driver.find_element(By.ID, "downloadReport").click()
print("Download clicked. Check folder:", download_folder)7.13 Web Automation Error Handling
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from datetime import datetime
driver = webdriver.Chrome()
try:
driver.get("https://example.com")
button = driver.find_element(By.ID, "submitBtn")
button.click()
except NoSuchElementException:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
driver.save_screenshot("error_" + timestamp + ".png")
print("Element not found. Screenshot saved.")
finally:
driver.quit()7.14 Complete Web Automation Bot Workflow
7.15 Downloadable Working Example
A complete working model is included as a separate downloadable ZIP file. It contains:
- web_automation_test_form.html - local test web form
- run_web_automation_bot.py - Selenium bot that fills the form
- README.txt - step-by-step instructions ⬇ Download Working Example ZIP
pip install selenium, then run python run_web_automation_bot.py.
7.16 Interactive Form Demo Inside This Chapter
This form simulates what a Selenium bot can automate.
7.17 Best Practices for Web Automation
| Best Practice | Reason |
|---|---|
| Use stable IDs when available | ID selectors are usually reliable. |
| Use explicit waits | Prevents failures from slow-loading elements. |
| Take screenshots on errors | Helps troubleshoot automation failures. |
| Do not store passwords in code | Protects security and compliance. |
| Use test sites before live systems | Prevents accidental business data changes. |
| Log automation results | Creates audit trail and performance record. |
| Handle exceptions | Makes the bot more reliable. |
7.18 Practical Activities
Activity 1: Open Website
Use Selenium to open https://perakskills.com and print the page title.
Activity 2: Local Form Automation
Use the downloadable test form and automate data entry using Selenium.
Activity 3: Screenshot Evidence
Create a script that takes a screenshot after submitting a form.
Activity 4: Table Extraction
Create an HTML table and use Selenium to extract the table into CSV.
Mini Project
Create a web automation bot that fills a form, submits it, captures the result text and saves a screenshot.
7.19 Interactive Final Assessment Quiz
Each correct answer gives +1 mark.
Each wrong answer gives -0.5 mark.
1. Selenium is used for browser automation.
2. Which method opens a website in Selenium?
3. ID selectors are usually reliable when available.
4. Which Selenium command types into a text field?
5. Explicit waits help handle dynamic web pages.
6. Which selector can locate an element by CSS?
7. Screenshots can be saved during web automation.
8. It is safe to test automation directly on live business systems without approval.
9. Selenium can extract text from web tables.
10. Error handling improves web automation reliability.
Your Score: 0
7.20 Chapter Summary
In this chapter, learners studied detailed web automation techniques using Selenium. They learned how to open websites, locate elements, fill forms, click buttons, use waits, select dropdowns, extract table data, take screenshots, handle errors and build a complete working web automation bot.