How to Use Perplexity AI for Web Scraping in Python

We invite you to explore our blog for more details.

Setting Up Your Coding Environment

Before building the application, you’ll need to set up a basic Python environment. Follow these steps to get started:

1. Install Python 3 on your system.
2. Install the required dependencies by running:

python -m pip install -r requirements.txt

Obtaining Crawlbase API Credentials

1. Create an account at Crawlbase and log in.
2. After registration, you will receive 5,000 free requests.
3. Replace occurence of <Private token> with your Crawling API Private token under Smart Proxy section.

2. Get Perplexity AI API Key

1. Sign up at Perplexity.ai
2. Replace occurence of <perplexity.ai API KEY> with your OpenAI-compatible API key.

How to Run

python perplexity_ai_powered_scraper.py

🛡 Disclaimer This repository is for educational purposes only. Please make sure you comply with the Terms of Service of any website you scrape. Use this responsibly and only where permitted.

---

Copyright 2025 Crawlbase