^{Special thanks to:}

Warp is built for coding with multiple AI agents

Disclaimer:

Please use this repository for learning purposes only ⚠️⚠️⚠️⚠️, Web scraping illegal cases

All content in this repository is for learning and reference purposes only, and commercial use is prohibited. No person or organization may use the content of this repository for illegal purposes or infringe upon the legitimate rights and interests of others. The web scraping technology involved in this repository is only for learning and research, and may not be used for large-scale crawling of other platforms or other illegal activities. This repository assumes no legal responsibility for any legal liability arising from the use of the content of this repository. By using the content of this repository, you agree to all terms and conditions of this disclaimer.

Click to view a more detailed disclaimer. Click to jump

A powerful multi-platform social media data collection tool that supports crawling public information from mainstream platforms including Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, Zhihu, and more.

• Core Technology: Based on Playwright browser automation framework for login and maintaining login state
• No JS Reverse Engineering Required: Uses browser context environment with preserved login state to obtain signature parameters through JS expressions
• Advantages: No need to reverse complex encryption algorithms, significantly lowering the technical barrier

💡 Open source is not easy, if this project helps you, please give a ⭐ Star to support!

Before proceeding with the next steps, please ensure that uv is installed on your computer:

• Installation Guide: uv Official Installation Guide
• Verify Installation: Enter the command uv --version in the terminal. If the version number is displayed normally, the installation was successful
• Recommendation Reason: uv is currently the most powerful Python package management tool, with fast speed and accurate dependency resolution

The project depends on Node.js, please download and install from the official website:

• Download Link: https://nodejs.org/en/download/
• Version Requirement: >= 16.0.0

# Enter project directory
cd MediaCrawler

# Use uv sync command to ensure consistency of python version and related dependency packages
uv sync

# Install browser driver
uv run playwright install

💡 Tip: MediaCrawler now supports using playwright to connect to your local Chrome browser, solving some issues caused by Webdriver.

Currently, xhs and dy are available using CDP mode to connect to local browsers. If needed, check the configuration items in config/base_config.py.

# The project does not enable comment crawling mode by default. If you need comments, please modify the ENABLE_GET_COMMENTS variable in config/base_config.py
# Other supported options can also be viewed in config/base_config.py with Chinese comments

# Read keywords from configuration file to search related posts and crawl post information and comments
uv run main.py --platform xhs --lt qrcode --type search

# Read specified post ID list from configuration file to get information and comment information of specified posts
uv run main.py --platform xhs --lt qrcode --type detail

# Open corresponding APP to scan QR code for login

# For other platform crawler usage examples, execute the following command to view
uv run main.py --help

# Enter project root directory
cd MediaCrawler

# Create virtual environment
# My python version is: 3.9.6, the libraries in requirements.txt are based on this version
# If using other python versions, the libraries in requirements.txt may not be compatible, please resolve on your own
python -m venv venv

# macOS & Linux activate virtual environment
source venv/bin/activate

# Windows activate virtual environment
venv\Scripts\activate

pip install -r requirements.txt

playwright install

# The project does not enable comment crawling mode by default. If you need comments, please modify the ENABLE_GET_COMMENTS variable in config/base_config.py
# Other supported options can also be viewed in config/base_config.py with Chinese comments

# Read keywords from configuration file to search related posts and crawl post information and comments
python main.py --platform xhs --lt qrcode --type search

# Read specified post ID list from configuration file to get information and comment information of specified posts
python main.py --platform xhs --lt qrcode --type detail

# Open corresponding APP to scan QR code for login

# For other platform crawler usage examples, execute the following command to view
python main.py --help

MediaCrawler supports multiple data storage methods, including CSV, JSON, Excel, SQLite, and MySQL databases.

📖 For detailed usage instructions, please see: Data Storage Guide

🚀 MediaCrawlerPro Major Release 🚀! More features, better architectural design!

• WeChat Discussion Group: Click to join

• Online Documentation: MediaCrawler Complete Documentation
• Crawler Tutorial: CrawlerTutorial Free Tutorial

The online documentation includes usage methods, common questions, joining project discussion groups, etc. MediaCrawler Online Documentation

If you want to quickly get started and learn the usage of this project, source code architectural design, learn programming technology, or want to understand the source code design of MediaCrawlerPro, you can check out my paid knowledge column.

Author's Paid Knowledge Column Introduction

If this project helps you, please give a ⭐ Star to support and let more people see MediaCrawler!

**Swiftproxy** - 90M+ global high-quality pure residential IPs, register to get free 500MB test traffic, dynamic traffic never expires! > Exclusive discount code: **GHB5** Get 10% off instantly!

Become a sponsor and showcase your product here, getting massive exposure daily!

Contact Information:

• WeChat: relakkes
• Email: relakkes@gmail.com

• Xiaohongshu Client: ReaJason's xhs repository
• SMS Forwarding: SmsForwarder reference repository
• Intranet Penetration Tool: ngrok official documentation

Thanks to JetBrains for providing free open source license support for this project!