Faster LLM inference.
Same model files, same answers.

What Sushila.cpp does

Decoding is limited by how many bytes of weights the hardware reads per token. Sushila (Scalable Upstream Synthesis for Hybrid Inference in Large-model Acceleration) computes small artifacts once per model so each token reads fewer bytes or the model runs fewer passes.

Sushila.cpp also uses established methods, including EAGLE-3 draft heads, small draft models and fast 4-bit kernels, and combines them with its own. The paper credits each method and reports how much it adds.

Measured speed

Tokens per second, greedy decoding, compared with the stock engine on the same hardware and model file.

Same model file and exactly the stock output in every row except the Ollama comparison, where Ollama reads a different 4-bit file of the same model (Q4_K_M against AWQ; same GSM8K accuracy). The full method, scripts and raw logs are in the repository.

Download Sushila.cpp

Free and open source under the MIT License, like llama.cpp and Ollama. It reads the same GGUF files, including the ones Ollama has already downloaded, and has the same tools: llama-cli, llama-server (OpenAI-compatible API) and more.

# 1. tools (Ubuntu/Debian; on Fedora: sudo dnf install gcc gcc-c++ make cmake git curl)
sudo apt-get install -y build-essential cmake git curl
# 2. download
git clone https://github.com/syncaissa/sushila.cpp.git && cd sushila.cpp
# 3. build (for an NVIDIA GPU add -DGGML_CUDA=ON; needs the CUDA toolkit)
cmake -S llama.cpp -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --config Release -j --target llama-cli llama-server llama-speculative-simple
# 4. run
build/bin/llama-server -m model.gguf -ngl 99 --port 8080

# 1. tools
xcode-select --install
brew install cmake git
# 2. download
git clone https://github.com/syncaissa/sushila.cpp.git && cd sushila.cpp
# 3. build
cmake -S llama.cpp -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --config Release -j --target llama-cli llama-server llama-speculative-simple
# 4. run
build/bin/llama-server -m model.gguf -ngl 99 --port 8080

# 1. in PowerShell as administrator, then restart
wsl --install
# 2. open "Ubuntu" from the Start menu and continue there
sudo apt-get update && sudo apt-get install -y build-essential cmake git curl
git clone https://github.com/syncaissa/sushila.cpp.git && cd sushila.cpp
# 3. build (NVIDIA GPU: install the Windows NVIDIA driver, add -DGGML_CUDA=ON)
cmake -S llama.cpp -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --config Release -j --target llama-cli llama-server llama-speculative-simple
# 4. run
build/bin/llama-server -m model.gguf -ngl 99 --port 8080

Already use Ollama? Reuse its files: ollama show --modelfile llama3.1:8b | grep '^FROM /' prints the model's path. Full guide: INSTALL.md. Provided as is, without warranty (disclaimer).

Models

We host these files ourselves. Downloads are free with a sushila.ai account (no password: a code is sent to your e-mail), so we can record that you accepted each model's license. Each file is byte-identical to the public release, so check its sha256 after you download it. Models tagged precomputed come with measured Sushila artifacts.

Downloads are provided as is, with no warranty; you assume all risks of use (see the disclaimer).

Built with Llama. The Llama models are distributed under their community licenses and Meta's acceptable use policy.

More models: download from Hugging Face

Any GGUF model runs with Sushila.cpp. Download these directly from their publishers.

Serverless API

This website and Sushila.cpp are free. To skip the hardware, use our hosted GPUs: an OpenAI-compatible API, billed per token. Because each token costs us less to generate, our rates are lower.

The API is provided as is, with no warranty or guarantee of availability; see the disclaimer, the Terms of Service and the Privacy Policy. We use your e-mail only to contact you about early access.

Disclaimer