Web scraping with Python: requests and BeautifulSoup

This tutorial builds a small Python scraper: it fetches pages with requests, parses them with BeautifulSoup, follows pagination, saves the result to CSV, checks robots.txt and retries temporary errors. A proxy comes last and is optional. Every example was run, and the outputs below come from those runs.

What you need

Python 3.10 or newer and two packages, installed in a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4

On Windows, activate the environment with .venv\Scripts\activate.

The examples collect data from books.toscrape.com, a sandbox built for scraping practice: 1,000 made-up book listings on 50 pages, with prices and ratings assigned at random. From each listing we take the title, price, availability, star rating and link.

Before the first request

The code below follows these rules, and they apply to any site:

• The site’s terms. Read them first. If they forbid automated collection, do not collect, or ask the owner for permission.
• robots.txt. The script reads it before the first page and stops at a page it disallows without requesting it.
• Load. One request at a time, with a pause: 2 seconds in the examples. While testing, fetch a few pages, not the whole site.
• Identify your client. Put the script’s name and a contact in the User-Agent so the owner can see who is sending requests. Do not pass the script off as a browser.
• Personal data. Do not collect names, contacts or profiles without a lawful basis, and do not keep such data if it lands in your sample by accident.
• A refusal is an answer. On a 403 or a page that asks you to sign in or pass a check, stop.

Proxeta’s Acceptable Use Policy forbids scraping where it is unlawful or violates the rights or terms of the target site. More on responsible collection is in the web scraping proxy guide.

Fetch and parse one page

from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/catalogue/page-1.html"
USER_AGENT = "books-scraper/0.1 (+mailto:you@example.com)"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def parse_books(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        stars = card.select_one("p.star-rating")["class"]
        books.append({
            "title": link["title"],
            "price": card.select_one("p.price_color").get_text(strip=True).lstrip("£"),
            "in_stock": "In stock" in card.select_one("p.availability").get_text(),
            "rating": next((RATINGS[c] for c in stars if c in RATINGS), None),
            "url": urljoin(page_url, link["href"]),
        })
    next_link = soup.select_one("li.next a")
    next_url = urljoin(page_url, next_link["href"]) if next_link else None
    return books, next_url


session = requests.Session()
session.headers["User-Agent"] = USER_AGENT
response = session.get(URL, timeout=(10, 30))
response.raise_for_status()

books, next_url = parse_books(response.content, response.url)
print(len(books), "books, next page:", next_url)
for book in books[:2]:
    print(book)

Output:

20 books, next page: https://books.toscrape.com/catalogue/page-2.html
{'title': 'A Light in the Attic', 'price': '51.77', 'in_stock': True, 'rating': 3, 'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'}
{'title': 'Tipping the Velvet', 'price': '53.74', 'in_stock': True, 'rating': 1, 'url': 'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'}

What matters in the request:

• Session. requests.Session keeps headers, cookies and the open connection between requests.
• User-Agent. By default, requests sends python-requests/2.34.2. Replace it with your script’s name and a real contact.
• The timeout. timeout=(10, 30) gives 10 seconds to connect and 30 to wait for the response. Without it, requests can wait forever.
• raise_for_status(). It turns a 4xx or 5xx response into an exception, so an error page is not parsed as data.
• Bytes, not text. This server sends Content-Type: text/html without a charset, so requests decodes response.text as ISO-8859-1 and the price comes out as £51.77. BeautifulSoup gets response.content and reads charset=UTF-8 from the page’s meta tag.

Each book is an article.product_pod element. select() returns every match of a CSS selector, select_one() the first one or None:

To find selectors on another site, right-click an element in the browser and choose Inspect. If the markup changes, select_one() returns None and the script stops with a TypeError or AttributeError, which is better than writing empty fields.

The complete script: pages, pauses and CSV

The script reads robots.txt, walks the pages with a pause, retries temporary errors and writes the rows to books.csv. The next two sections explain load_robots() and fetch().

import csv
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib import robotparser
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

START_URL = "https://books.toscrape.com/catalogue/page-1.html"
USER_AGENT = "books-scraper/0.1 (+mailto:you@example.com)"
TIMEOUT = (10, 30)
DELAY = 2  # seconds between requests
MAX_PAGES = 3
PROXIES = None  # see "Adding a proxy" below
RETRY_STATUSES = {429, 500, 502, 503, 504}
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def load_robots(session, site_url):
    robots = robotparser.RobotFileParser()
    response = session.get(urljoin(site_url, "/robots.txt"), timeout=TIMEOUT, proxies=PROXIES)
    if response.status_code in (401, 403):
        robots.disallow_all = True
    elif 400 <= response.status_code < 500 and response.status_code != 429:
        robots.allow_all = True
    else:
        response.raise_for_status()
        robots.parse(response.content.decode("utf-8", "replace").splitlines())
    return robots


def retry_after(response):
    value = response.headers.get("Retry-After", "").strip()
    if value.isdigit():
        return int(value)
    try:
        moment = parsedate_to_datetime(value)
        return max(0, (moment - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None


def fetch(session, url, retries=4, backoff=5, max_wait=300):
    for attempt in range(retries + 1):
        try:
            response = session.get(url, timeout=TIMEOUT, proxies=PROXIES)
        except requests.exceptions.ProxyError:
            raise  # a proxy setting to fix, not a reason to retry
        except (requests.ConnectionError, requests.Timeout) as error:
            if attempt == retries:
                raise
            wait = backoff * 2 ** attempt
            print(f"{type(error).__name__} on {url}, retrying in {wait} s")
        else:
            if response.status_code not in RETRY_STATUSES or attempt == retries:
                response.raise_for_status()
                return response
            wait = retry_after(response)
            if wait is None:
                wait = backoff * 2 ** attempt
            if wait > max_wait:
                raise RuntimeError(f"{url}: Retry-After asks for {wait:.0f} s, stopping")
            print(f"{response.status_code} on {url}, retrying in {wait:.0f} s")
        time.sleep(wait)


def parse_books(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        stars = card.select_one("p.star-rating")["class"]
        books.append({
            "title": link["title"],
            "price": card.select_one("p.price_color").get_text(strip=True).lstrip("£"),
            "in_stock": "In stock" in card.select_one("p.availability").get_text(),
            "rating": next((RATINGS[c] for c in stars if c in RATINGS), None),
            "url": urljoin(page_url, link["href"]),
        })
    next_link = soup.select_one("li.next a")
    next_url = urljoin(page_url, next_link["href"]) if next_link else None
    return books, next_url


def main():
    session = requests.Session()
    session.headers["User-Agent"] = USER_AGENT
    robots = load_robots(session, START_URL)
    delay = max(DELAY, robots.crawl_delay(USER_AGENT) or 0)
    url, pages = START_URL, 0
    with open("books.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["title", "price", "in_stock", "rating", "url"])
        writer.writeheader()
        while url and pages < MAX_PAGES:
            if not robots.can_fetch(USER_AGENT, url):
                print("robots.txt disallows", url)
                break
            time.sleep(delay)
            response = fetch(session, url)
            books, url = parse_books(response.content, response.url)
            writer.writerows(books)
            pages += 1
            print(f"page {pages}: {len(books)} books, next: {url}")


if __name__ == "__main__":
    main()

Save it as scraper.py and run python scraper.py:

page 1: 20 books, next: https://books.toscrape.com/catalogue/page-2.html
page 2: 20 books, next: https://books.toscrape.com/catalogue/page-3.html
page 3: 20 books, next: https://books.toscrape.com/catalogue/page-4.html

books.csv now holds a header and 60 rows:

title,price,in_stock,rating,url
A Light in the Attic,51.77,True,3,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

• Pagination. The loop runs while there is a next-page link and fewer than MAX_PAGES pages are done. With a 2-second pause, all 50 pages take about two minutes.
• The pause. time.sleep(delay) runs before every page. If robots.txt sets a Crawl-delay for your client, the script waits at least that long.
• CSV. newline="" prevents blank lines between rows on Windows. For Excel, encoding="utf-8-sig" helps it recognize UTF-8.
• Links. urljoin uses response.url, the page’s final address after redirects.

How the script reads robots.txt

load_robots() fetches /robots.txt through the same session, so the request carries your User-Agent and timeout; the parser’s own read() would use urllib’s default User-Agent and no timeout. Then it decides by the response:

As of November 7, 2026, books.toscrape.com has no robots.txt: the server returns 404, so can_fetch() returns True and crawl_delay() returns None. With a robots.txt like this one:

User-agent: *
Disallow: /catalogue/category/
Crawl-delay: 5

can_fetch() returns False for pages under /catalogue/category/, and the script pauses 5 seconds instead of 2. The parser matches groups by the name before the slash in your User-Agent, books-scraper here, and falls back to *. The site’s terms apply on top of robots.txt.

Retries, backoff and Retry-After

Some errors are temporary: the server is overloaded or asks you to slow down. fetch() repeats such requests with a growing pause and gives up on the rest:

The pause comes from the Retry-After header, given as seconds or as a date; retry_after() handles both. Without the header, the pause doubles: 5, 10, 20 and 40 seconds. If the last attempt fails too, fetch() raises the error. Against a local test server that answered 503, then 429 with Retry-After: 1, then 200, it printed:

503 on http://127.0.0.1:8000/test, retrying in 5 s
429 on http://127.0.0.1:8000/test, retrying in 1 s

If 429 keeps coming back, the pause is too short: raise DELAY, not the number of retries. A 403 is not retried: the site has refused access, and repeating the request only adds load.

Adding a proxy

The script works without a proxy. You may need one to see pages as they are served in a particular country or city, or to run the job from a network other than yours.

With Proxeta residential and mobile proxies, HTTP runs on port 8080 and SOCKS5 on port 1080, with the same login and password. The Dashboard issues a host:port:login:password string; location and session mode are written into the login, so copy it whole, as explained in proxy string format. In scraper.py, replace the PROXIES = None line:

PROXY = "http://login:password@proxy.example.invalid:8080"
PROXIES = {"http": PROXY, "https": PROXY}

fetch() and load_robots() already pass PROXIES to every request.

• Both keys. With only "http", requests to https:// pages do not go through the proxy. SOCKS5, special characters in the password and error messages are covered in proxies in Python.
• Rotation or a sticky session. With a rotating string, each new connection may get a new IP. requests.Session reuses its connection while the server keeps it open, so one session’s requests often share an address, but a reconnect can change it. If the steps must keep one address, for example when the site stores your chosen region in cookies, use a sticky session string: it keeps one IP for as long as possible, with no promised duration. See sticky session vs rotation.
• The rules stay. A proxy changes the exit address, not what the site allows. Keep the pause and the robots.txt check, and if the site answers 403 or keeps answering 429, stop rather than switch the IP.

A wrong login or password shows up as 407 Proxy Authentication Required: inside a ProxyError for https:// pages, as an HTTPError for http:// pages.

When a page needs a browser

requests downloads the HTML the server sends and does not run JavaScript, while on some sites scripts in the browser add the data to the page. The practice page quotes.toscrape.com/js/ works this way: in the HTML that requests downloads, BeautifulSoup finds zero div.quote elements, because the quotes sit in a script.

To check a page, use View Page Source rather than the element inspector, which shows the page after its scripts have run. If the data is not in the source:

• look for it in the page’s scripts or in the JSON the page requests (the Network tab in developer tools), and make sure the site’s terms allow you to use it;
• or open the page in a browser with Playwright: it runs the scripts, and page.content() returns the finished HTML for a function like parse_books(). Playwright with a proxy is covered in proxies in Python.

A browser downloads more and is slower, so use it only for pages that need it, with the same pauses.

How the examples were run

The examples were run on November 7, 2026 on macOS in Python 3.13.14 with requests 2.34.2 and beautifulsoup4 4.15.0 (soupsieve 4.0.1). The first two ran against books.toscrape.com with a 2-second pause, about ten requests in all. Retries were checked against a local test server that answered 503, 429 with Retry-After in seconds and as a date, 403 and 404. The proxy settings were checked through a local test HTTP proxy with a username and password, for http:// and https:// sites, not through the Proxeta gateway: that checks the code and the libraries, not exit IPs or the real network.