Skip to main content

Llama.cpp

llama.cpp python library is a simple Python bindings for @ggerganov llama.cpp.

This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility
    • LlamaIndex compatibility
  • OpenAI compatible web server
    • Local Copilot replacement
    • Function Calling support
    • Vision API support
    • Multiple Models

Installation and Setup​

  • Install the Python package
    pip install llama-cpp-python
  • Download one of the supported models and convert them to the llama.cpp format per the instructions

Chat models​

See a usage example.

from langchain_community.chat_models import ChatLlamaCpp
API Reference:ChatLlamaCpp

LLMs​

See a usage example.

from langchain_community.llms import LlamaCpp
API Reference:LlamaCpp

Embedding models​

See a usage example.

from langchain_community.embeddings import LlamaCppEmbeddings
API Reference:LlamaCppEmbeddings

Was this page helpful?


You can also leave detailed feedback on GitHub.