Python Web Scraping | BeautifulSoup and Selenium Explained

Python Web Scraping | BeautifulSoup and Selenium Explained

· 20분 읽기 · 수정 중급 Tutorial
Python Web Scraping | BeautifulSoup and Selenium Explained

이 글의 핵심

Python web scraping tutorial: requests, BeautifulSoup for static HTML, Selenium for dynamic pages, ethics (robots.txt, rate limits), and CSV export—SEO-friendly patterns.

Introduction

“Collect data from the web”

Web scraping is the technique of automatically extracting data from websites.

1. requests basics

Fetching HTML

import requests
# GET request
response = requests.get('https://example.com')
print(response.status_code)  # 200
print(response.text)  # HTML body
print(response.headers)  # Response headers
# Custom User-Agent
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get('https://example.com', headers=headers)

2. BeautifulSoup

Parsing HTML

from bs4 import BeautifulSoup
import requests
url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
# Single tag
title = soup.find('title')
print(title.text)
# Multiple tags
links = soup.find_all('a')
for link in links:
    print(link.get('href'))
# CSS selectors
articles = soup.select('.article-title')
for article in articles:
    print(article.text)

Example: news headlines

import requests
from bs4 import BeautifulSoup
import pandas as pd
def scrape_news(url):
    """Collect news titles and links."""
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
    }
    
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    
    articles = []
    
    for item in soup.select('.news-item'):
        title = item.select_one('.title').text.strip()
        link = item.select_one('a')['href']
        date = item.select_one('.date').text.strip()
        
        articles.append({
            'title': title,
            'link': link,
            'date': date
        })
    
    return pd.DataFrame(articles)
# Usage
df = scrape_news('https://news.example.com')
df.to_csv('news.csv', index=False, encoding='utf-8-sig')

3. Selenium (dynamic pages)

Install

pip install selenium

Basic usage

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
    driver.get('https://example.com')
    
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'content'))
    )
    
    title = driver.find_element(By.TAG_NAME, 'h1')
    print(title.text)
    
    button = driver.find_element(By.ID, 'load-more')
    button.click()
    
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
    
finally:
    driver.quit()

4. Real-world example

Price monitoring

import requests
from bs4 import BeautifulSoup
import time
from datetime import datetime
def check_price(url, target_price):
    """Read product price from a page (selectors vary by site)."""
    headers = {
        'User-Agent': 'Mozilla/5.0'
    }
    
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    
    price_text = soup.select_one('.price').text
    price = int(price_text.replace(',', ').replace('원', '))
    
    print(f"[{datetime.now()}] Current price: {price:,} KRW")
    
    if price <= target_price:
        print(f"🎉 Target reached! (≤ {target_price:,} KRW)")
        return True
    
    return False
# Check every hour
url = 'https://shopping.example.com/product/123'
target = 50000
while True:
    if check_price(url, target):
        break
    time.sleep(3600)

5. Saving data

CSV export

import pandas as pd
def scrape_and_save(url, output_file):
    """Scrape and write CSV."""
    data = scrape_data(url)
    df = pd.DataFrame(data)
    df.to_csv(output_file, index=False, encoding='utf-8-sig')
    print(f"Saved: {output_file}")

Practical tips

Scraping etiquette

# ✅ Check robots.txt
# https://example.com/robots.txt
# ✅ Space out requests
import time
time.sleep(1)
# ✅ Set a descriptive User-Agent
headers = {'User-Agent': '...'}
# ✅ Handle errors
try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Summary

Key takeaways

  1. requests: HTTP calls
  2. BeautifulSoup: HTML parsing
  3. Selenium: JavaScript-heavy pages
  4. Etiquette: robots.txt, pacing
  5. Storage: CSV, JSON, databases

Next steps

자주 묻는 질문 (FAQ)

Q. 이 내용을 실무에서 언제 쓰나요?

A. Python web scraping tutorial: requests, BeautifulSoup for static HTML, Selenium for dynamic pages, ethics (robots.txt, r… 실무에서는 위 본문의 예제와 선택 가이드를 참고해 적용하면 됩니다.

Q. 선행으로 읽으면 좋은 글은?

A. 각 글 하단의 이전 글 또는 관련 글 링크를 따라가면 순서대로 배울 수 있습니다. Python 시리즈 목차에서 전체 흐름을 확인할 수 있습니다.

Q. 더 깊이 공부하려면?

A. cppreference와 해당 라이브러리 공식 문서를 참고하세요. 글 말미의 참고 자료 링크도 활용하면 좋습니다.

같이 보면 좋은 글 (내부 링크)

이 주제와 연결되는 다른 글입니다.

이 글에서 다루는 키워드 (관련 검색어)

Python, Web Scraping, Crawling, BeautifulSoup, Selenium, requests 등으로 검색하시면 이 글이 도움이 됩니다.