Kaka Subtitle Assistant

VideoCaptioner

An LLM-powered video subtitle processing assistant, supporting speech recognition, subtitle segmentation, optimization, and translation.

简体中文 / 正體中文 / English / 日本語

Kaka Subtitle Assistant (VideoCaptioner) is easy to operate and doesn't require high-end hardware. It supports both online API calls and local offline processing (with GPU support) for speech recognition. It leverages Large Language Models (LLMs) for intelligent subtitle segmentation, correction, and translation. It offers a one-click solution for the entire video subtitle workflow! Add stunning subtitles to your videos.

The latest version now supports VAD, vocal separation, word-level timestamps, batch subtitle processing, and other practical features.

• 🎯 No GPU required to use powerful speech recognition engines for accurate subtitle generation.
• ✂️ LLM-based intelligent segmentation and sentence breaking for more natural subtitle reading.
• 🔄 AI subtitle multi-threading optimization and translation, adjusting subtitle format and making expressions more idiomatic and professional.
• 🎬 Supports batch video subtitle synthesis, improving processing efficiency.
• 📝 Intuitive subtitle editing and viewing interface, supporting real-time preview and quick editing.
• 🤖 Low model token consumption, and built-in basic LLM model to ensure out-of-the-box use.

Processing a 14-minute 1080P English TED video from Bilibili end-to-end, using the local Whisper model for speech recognition and the gpt-4o-mini model for optimization and translation into Chinese, took approximately 3 minutes.

Based on backend calculations, the cost for model optimization and translation was less than ¥0.01 (calculated using OpenAI's official pricing).

For detailed results of subtitle and video synthesis, please refer to the TED Video Test.

The software is lightweight, with a package size of less than 60MB, and includes all necessary environments. Download and run directly.

1. Download the latest version of the executable from the Release page. Or: Lanzou Cloud Download

2. Open the installer to install.

3. (Optional) LLM API Configuration, choose whether to enable subtitle optimization or subtitle translation.

4. Drag and drop the video file into the software window for fully automatic processing.

Note: Each step supports independent processing and file drag-and-drop.

brew install ffmpeg
brew install aria2
brew install python@3.**

git clone https://github.com/WEIFENG2333/VideoCaptioner.git
cd VideoCaptioner

python3.** -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python main.py

git clone https://github.com/WEIFENG2333/VideoCaptioner.git
cd VideoCaptioner

docker build -t video-captioner .

docker run -d \
  -p 8501:8501 \
  -v $(pwd)/temp:/app/temp \
  -e OPENAI_BASE_URL="Your API address" \
  -e OPENAI_API_KEY="Your API key" \
  --name video-captioner \
  video-captioner

The software fully utilizes the advantages of Large Language Models (LLMs) in understanding context to further process subtitles generated by speech recognition. It effectively corrects typos, unifies terminology, and makes the subtitle content more accurate and coherent, providing users with an excellent viewing experience!

• Supports mainstream video platforms at home and abroad (Bilibili, YouTube, etc.)
• Automatically extracts and processes the original subtitles of the video.

• Provides multiple online recognition interfaces with effects comparable to Jianying (free, high-speed).
• Supports local Whisper model (privacy protection, offline).

• Automatically optimizes the format of terminology, code snippets, and mathematical formulas.
• Contextual sentence segmentation optimization to improve reading experience.
• Supports manuscript prompts, using original manuscripts or related prompts to optimize subtitle segmentation.

• Context-aware intelligent translation ensures that the translation takes the entire text into account.
• Guides the large model to reflect on the translation through prompts, improving translation quality.
• Uses a sequence fuzzy matching algorithm to ensure complete consistency of the timeline.

• Rich subtitle style templates (popular science style, news style, anime style, etc.).
• Multiple subtitle video formats (SRT, ASS, VTT, TXT).

Recommended models: For higher quality, choose Claude-3.5-sonnet or gpt-4o.

There are two Whisper versions: WhisperCpp and fasterWhisper (recommended). The latter has better performance and both require downloading models within the software.

Recommended model: Large-v1 is stable and of good quality.

Note: The above models can be downloaded directly within the software using a domestic network; both GPU and integrated graphics are supported.

• On the "Subtitle Optimization and Translation" page, there is a "Manuscript Matching" option, which supports the following one or more types of content to assist in subtitle correction and translation:

• If you need manuscript assistance for subtitle optimization, fill in the manuscript information first, then start the task processing.
• Note: When using small LLM models with limited context, it is recommended to keep the manuscript content within 1000 words. If using a model with a larger context window, you can appropriately increase the manuscript content.

If you encounter the following situations when using the URL download function:

1. The video website requires login information to download.
2. Only lower resolution videos can be downloaded.
3. Verification is required when network conditions are poor.

• Please refer to the Cookie Configuration Instructions to obtain cookie information and place the cookies.txt file in the AppData directory of the software installation directory to download high-quality videos normally.

The simple processing flow of the program is as follows:

Speech Recognition -> Subtitle Segmentation (optional) -> Subtitle Optimization & Translation (optional) -> Subtitle & Video Synthesis

The main directory structure after installing the software is as follows:

VideoCaptioner/
├── runtime/                    # Runtime environment directory (do not modify)
├── resources/               # Software resource file directory (binaries, icons, etc., and downloaded faster-whisper program)
├── work-dir/               # Working directory, where processed videos and subtitle files are saved
├── AppData/                    # Application data directory
    ├── cache/              # Cache directory, caching transcription and large model request data.
    ├── models/              # Stores Whisper model files
    ├── logs/               # Log directory, recording software running status
    ├── settings.json          # Stores user settings
    └──  cookies.txt           # Cookie information for video platforms (required for downloading high-definition videos)
└── VideoCaptioner.exe      # Main program executable file

1. The quality of subtitle segmentation is crucial for the viewing experience. For this, I developed SubtitleSpliter, which can intelligently reorganize word-by-word subtitles into paragraphs that conform to natural language habits and perfectly synchronize with the video frames.

2. During processing, only the text content is sent to the large language model, without timeline information, which greatly reduces processing overhead.

3. In the translation stage, we adopt the "translate-reflect-translate" methodology proposed by Andrew Ng. This iterative optimization method not only ensures the accuracy of the translation.

The author is a junior college student. Both my personal abilities and the project have many shortcomings. The project is also constantly being improved. If you encounter any bugs during use, please feel free to submit Issues and Pull Requests to help improve the project.

If you find this project helpful, please give it a Star. This will be the greatest encouragement and support for me!