Let LLM empowered the operator optimization.
The project name may be changed later.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

This project is a research project by C4Y, who is a student of Optima Research Group, and aims to use LLM to automatically optimize mathmetical calculating operators towards hardware design.

(back to top)

To get a local copy up and running follow these simple example steps.

The author strongly recommand to run the project in a Linux environment and use online LLM API services.

This project is mainly divided into two parts: the operator library implemented in C language and the automatic optimizer implemented in Python. These two parts need to be configured separately.

Additionally, a remote or local operator performance evaluation program is required, which also needs to be configured separately.

This project also utilizes a large language model for optimization, which requires access to an LLM service that complies with the OpenAI API SDK. In the actual code implementation, the Deepseek API is used.

1. To configure the C language portion of this project, you will need a GCC environment, a RISC-V bare metal machine or an equivalent emulator (such as QEMU), and support for the GCC module with RVV 1.0 extensions.

   sudo apt update
   sudo apt install build-essential bison flex libgmp3-dev libmpc-dev libmpfr-dev texinfo
   ./configure --target=riscv64-unknown-elf --prefix=/opt/riscv --enable-languages=c,c++ --with-arch=rv64gc --with-abi=lp64d --enable-multilib --with-vector=$riscv_rvv_version
   sudo apt install cmake
   

2. To configure the Python language portion of this project, you will need PDM, which is a modern Python package and dependency manager supporting the latest PEP standards, to create a virtual environment, check dependencies, and install and configure both binary and PyPI dependencies.

   curl -sSL https://pdm-project.org/install-pdm.py | python3 -
   pdm --version
   

3. The program running on Bianbu Cloud which is used to evaluate the performance will be open-sourced soon.

1. Get a free API Key at deepseek (to provide a LLM api service) and Bianbu Cloud (to provide a metal risc-v enviornment).
2. Clone the repo

   git clone https://github.com/c4yg70/LLM4Opt.git
   

3. Install Python Portion

   cd python_process
   pdm install
   

4. Enter your API by set up the environment value

   export DS_API_KEY = 'ENTER YOUR API'
   export DS_API_URL = 'ENTER YOUR URL';
   export SC_API_KEY = 'ENTER YOUR API';
   export SC_API_URL = 'ENTER YOUR URL';
   

5. Check the API whether is correctly working

   pdm check llm
   pdm check riscv-cloud
   

6. Verify that all necessary C dependencies, including the GCC environment, RISC-V toolchain, and QEMU emulator (if applicable), are correctly installed and configured.

   pdm check riscv-local
   

7. Run the project

   pdm start-opti --cycle 1 --llm 1 --cloud_eval 1
   

(back to top)

Now we could use LLM4Opt to optimize 20+ operators (BasicFunctions and ActivationFunctions) in both CMSIS-NN projects, which is the ARM-optimized NN Lib developed by the ARM officially, and muRISCV-NN, which is the RISCV-optimized NN Lib developed by project Scale4Edge, and evaluate the performance of the optimized operators.

Performance are tested both in lib level and application level. Lib level tests, which show the efficiency of single operator's optimization, are provided in eval_test. Application level tests, which show the efficiency of operator's optimization affecting the real-world application, are used MLPerfTiny, proposed by C. Banbury in 2021. All results are evaluated in Spacemit K1 processor provided by Bianbu Cloud, and provided in results.

Normally, a successful optimization will cost about 8-20 million tokens, depends on CoT length.

(back to top)

• One-step installation script
• Intergration with Glenside
• Allow optimizing more operators
  • MatrixCalcFunctions
• Allow optimizing towards more construction sets
  • SSE/SSE2
  • RVC23

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Distributed under the project_license. See LICENSE.txt for more information.

(back to top)

CHEN Siyuan - siyuan.chen@cityu.edu.hk

Project Link: https://github.com/c4yg70/LLM4Opt

(back to top)

• README Template
• Spacemit RISC-V Cloud Service
• MLPerf Tiny: Banbury, Colby R. et al. “MLPerf Tiny Benchmark.” ArXiv abs/2106.07597 (2021): n. pag.

(back to top)