Skip to main content
Questions Categories Search Providers
Scraper API

How to Rotate Proxy in Selenium Python: Complete Guide [2026]

5 min read

Why Rotate Proxies in Selenium?

When web scraping with Selenium, using a single IP address quickly leads to detection and blocking. Websites implement rate limiting, CAPTCHAs, and IP bans to prevent automated access. Proxy rotation solves this by distributing requests across multiple IP addresses, making your scraping activity appear as organic traffic from different users.

The main benefits of proxy rotation include:

    • Avoiding IP bans - Spreading requests across multiple IPs prevents any single address from being flagged
    • Bypassing rate limits - Each proxy gets its own rate limit allocation
    • Geographic diversity - Access geo-restricted content from different locations
    • Improved reliability - If one proxy fails, others continue working
    • Better anonymity - Harder to trace scraping activity back to a single source

    Setting Up selenium-wire for Proxy Rotation

    The selenium-wire library extends Selenium with the ability to inspect and modify HTTP requests, including easy proxy configuration. It is the most popular choice for proxy rotation in Python Selenium projects.

    Installation

    Install selenium-wire and its dependencies:

    pip install selenium-wire

    pip install webdriver-manager

    Basic Proxy Configuration

    Here is how to configure a single proxy with selenium-wire:

    from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service
    

    proxy_options = { 'proxy': { 'http': 'http://username:password@proxy.example.com:8080', 'https': 'https://username:password@proxy.example.com:8080', 'no_proxy': 'localhost,127.0.0.1' } }
    

    service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=service, seleniumwire_options=proxy_options)
    

    driver.get('https://httpbin.org/ip') print(driver.page_source) driver.quit()

    Implementing Proxy Rotation with a Proxy List

    For basic proxy rotation, maintain a list of proxies and rotate through them. Here is a complete implementation:

    import random from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service from selenium.webdriver.chrome.options import Options
    

    class ProxyRotator: def __init__(self, proxy_list): self.proxies = proxy_list self.current_index = 0
    

    def get_next_proxy(self): proxy = self.proxies[self.current_index] self.current_index = (self.current_index + 1) % len(self.proxies) return proxy
    

    def get_random_proxy(self): return random.choice(self.proxies)
    

    def remove_proxy(self, proxy): if proxy in self.proxies: self.proxies.remove(proxy)
    

    proxy_list = [ 'http://user:pass@proxy1.example.com:8080', 'http://user:pass@proxy2.example.com:8080', 'http://user:pass@proxy3.example.com:8080', 'http://user:pass@proxy4.example.com:8080', 'http://user:pass@proxy5.example.com:8080' ]
    

    rotator = ProxyRotator(proxy_list)
    

    def create_driver_with_proxy(proxy_url): chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage')
    

    proxy_options = { 'proxy': { 'http': proxy_url, 'https': proxy_url } }
    

    service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome( service=service, options=chrome_options, seleniumwire_options=proxy_options ) return driver
    

    urls_to_scrape = ['https://example.com/page1', 'https://example.com/page2']
    

    for url in urls_to_scrape: proxy = rotator.get_next_proxy() driver = create_driver_with_proxy(proxy) try: driver.get(url) # Process page content print(f"Scraped {url} using {proxy}") finally: driver.quit()

    Integrating with Bright Data API

    Bright Data (formerly Luminati) provides rotating residential proxies with automatic IP rotation. Their gateway handles rotation for you, simplifying implementation:

    from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service
    

    Bright Data configuration
    

    BRIGHT_DATA_HOST = 'brd.superproxy.io' BRIGHT_DATA_PORT = 22225 BRIGHT_DATA_USER = 'brd-customer-CUSTOMER_ID-zone-ZONE_NAME' BRIGHT_DATA_PASS = 'YOUR_PASSWORD'
    

    proxy_url = f'http://{BRIGHT_DATA_USER}:{BRIGHT_DATA_PASS}@{BRIGHT_DATA_HOST}:{BRIGHT_DATA_PORT}'
    

    proxy_options = { 'proxy': { 'http': proxy_url, 'https': proxy_url } }
    

    service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=service, seleniumwire_options=proxy_options)
    

    Each request automatically uses a different IP
    

    for i in range(10): driver.get('https://httpbin.org/ip') print(f"Request {i+1}: {driver.find_element('tag name', 'pre').text}")
    

    driver.quit()

    Bright Data also supports session persistence for when you need the same IP across multiple requests:

    # Add session ID for sticky sessions import random session_id = random.randint(1000000, 9999999) BRIGHT_DATA_USER = f'brd-customer-CUSTOMER_ID-zone-ZONE_NAME-session-{session_id}'

    Integrating with Smartproxy API

    Smartproxy offers similar rotating proxy functionality with their own gateway:

    from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service
    

    Smartproxy configuration
    

    SMARTPROXY_HOST = 'gate.smartproxy.com' SMARTPROXY_PORT = 7000 SMARTPROXY_USER = 'YOUR_USERNAME' SMARTPROXY_PASS = 'YOUR_PASSWORD'
    

    proxy_url = f'http://{SMARTPROXY_USER}:{SMARTPROXY_PASS}@{SMARTPROXY_HOST}:{SMARTPROXY_PORT}'
    

    proxy_options = { 'proxy': { 'http': proxy_url, 'https': proxy_url } }
    

    service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=service, seleniumwire_options=proxy_options)
    

    driver.get('https://ip.smartproxy.com/json') print(driver.page_source) driver.quit()

    Dynamic Proxy Switching Mid-Session

    Sometimes you need to change proxies during an active browser session. selenium-wire allows this through the driver.proxy property:

    from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service
    

    service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=service)
    

    Initial request without proxy
    

    driver.get('https://httpbin.org/ip') print("Original IP:", driver.find_element('tag name', 'pre').text)
    

    Switch to proxy
    

    driver.proxy = { 'http': 'http://user:pass@proxy1.example.com:8080', 'https': 'http://user:pass@proxy1.example.com:8080' }
    

    driver.get('https://httpbin.org/ip') print("Proxy 1 IP:", driver.find_element('tag name', 'pre').text)
    

    Switch to different proxy
    

    driver.proxy = { 'http': 'http://user:pass@proxy2.example.com:8080', 'https': 'http://user:pass@proxy2.example.com:8080' }
    

    driver.get('https://httpbin.org/ip') print("Proxy 2 IP:", driver.find_element('tag name', 'pre').text)
    

    driver.quit()

    Error Handling and Retry Logic

    Robust proxy rotation requires proper error handling. Proxies fail, get banned, or timeout frequently. Here is a production-ready implementation:

    import time import random from seleniumwire import webdriver from selenium.common.exceptions import WebDriverException, TimeoutException from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.service import Service from selenium.webdriver.chrome.options import Options
    

    class RobustProxyRotator: def __init__(self, proxy_list, max_retries=3): self.proxies = proxy_list.copy() self.failed_proxies = [] self.max_retries = max_retries
    

    def get_working_proxy(self): if not self.proxies: # Restore failed proxies and try again self.proxies = self.failed_proxies.copy() self.failed_proxies = [] return random.choice(self.proxies) if self.proxies else None
    

    def mark_failed(self, proxy): if proxy in self.proxies: self.proxies.remove(proxy) self.failed_proxies.append(proxy)
    

    def scrape_with_retry(self, url, process_func): for attempt in range(self.max_retries): proxy = self.get_working_proxy() if not proxy: raise Exception("No proxies available")
    

    driver = None try: driver = self._create_driver(proxy) driver.set_page_load_timeout(30) driver.get(url)
    

    # Check for blocking indicators if self._is_blocked(driver): raise Exception("Proxy blocked")
    

    result = process_func(driver) return result
    

    except (WebDriverException, TimeoutException, Exception) as e: print(f"Attempt {attempt + 1} failed with {proxy}: {e}") self.mark_failed(proxy) time.sleep(2 ** attempt)  # Exponential backoff
    

    finally: if driver: driver.quit()
    

    raise Exception(f"Failed to scrape {url} after {self.max_retries} attempts")
    

    def _create_driver(self, proxy): chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox')
    

    proxy_options = { 'proxy': {'http': proxy, 'https': proxy} }
    

    service = Service(ChromeDriverManager().install()) return webdriver.Chrome( service=service, options=chrome_options, seleniumwire_options=proxy_options )
    

    def _is_blocked(self, driver): blocked_indicators = [ 'access denied', 'blocked', 'captcha', 'rate limit', 'too many requests' ] page_text = driver.page_source.lower() return any(indicator in page_text for indicator in blocked_indicators)
    

    Usage
    

    proxies = [ 'http://user:pass@proxy1.com:8080', 'http://user:pass@proxy2.com:8080', 'http://user:pass@proxy3.com:8080' ]
    

    rotator = RobustProxyRotator(proxies)
    

    def extract_data(driver): return driver.find_element('tag name', 'body').text
    

    result = rotator.scrape_with_retry('https://example.com', extract_data) print(result)

    Proxy Authentication Methods

    Different proxy providers use different authentication methods. Here is how to handle each:

    MethodFormatExample
    Username/Passwordhttp://user:pass@host:porthttp://john:secret@proxy.com:8080
    IP Whitelistinghttp://host:porthttp://proxy.com:8080
    API Key in URLhttp://apikey@host:porthttp://abc123@proxy.com:8080

    Best Practices for Proxy Rotation

    Follow these guidelines for effective and ethical proxy rotation:

    Rate Limiting

    • Add random delays between requests (2-5 seconds minimum)
    • Implement exponential backoff on failures
    • Respect robots.txt and website terms of service

    Proxy Pool Management

    • Regularly test and validate proxies
    • Remove consistently failing proxies
    • Maintain geographic diversity for global scraping
    • Use residential proxies for sites with strict detection

    Browser Fingerprinting

    • Rotate user agents alongside proxies
    • Randomize viewport sizes and screen resolutions
    • Clear cookies and cache between sessions
    • Use undetected-chromedriver for strict anti-bot sites
    import random
    

    user_agents = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...' ]
    

    chrome_options = Options() chrome_options.add_argument(f'--user-agent={random.choice(user_agents)}')

    Session Management

    • Use sticky sessions for multi-page workflows (login, checkout)
    • Create new browser instances for unrelated requests
    • Implement proper cleanup to avoid resource leaks

Common Issues and Solutions

IssueCauseSolution
SSL Certificate ErrorsProxy intercepting HTTPSAdd verify_ssl=False to options or use proper certificates
Connection TimeoutsSlow or dead proxyImplement timeout and retry logic
Authentication FailuresWrong credentials formatURL encode special characters in password
Memory LeaksNot closing driversAlways use try/finally to quit driver

Conclusion

Proxy rotation in Selenium Python is essential for reliable web scraping at scale. Using selenium-wire provides the flexibility to implement custom rotation logic or integrate with commercial proxy providers like Bright Data and Smartproxy. The key to success is combining proper rotation strategies with robust error handling, respecting rate limits, and implementing anti-detection measures. Start with a simple rotation implementation and gradually add sophistication based on your specific scraping requirements and the websites you target.

Share: