README
¶
llama
This package provides Go bindings to llama.cpp.
Vendoring
Ollama vendors llama.cpp and ggml. While we generally strive to contribute changes back upstream to avoid drift, we carry a small set of patches which are applied to the tracking commit.
If you update the vendoring code, start by running the following command to establish the tracking llama.cpp repo in the ./vendor/ directory.
make -f Makefile.sync apply-patches
Updating Base Commit
Pin to new base commit
To change the base commit, update FETCH_HEAD in Makefile.sync.
When updating to a newer base commit, the existing patches may not apply cleanly and require manual merge resolution.
Start by applying the patches. If any of the patches have conflicts, the git am will stop at the first failure.
make -f Makefile.sync apply-patches
If there are conflicts, you will see an error message. Resolve the conflicts in ./vendor/, and continue the patch series with git am --continue and rerun make -f Makefile.sync apply-patches. Repeat until all patches are successfully applied.
Once all patches are applied, commit the changes to the tracking repository.
make -f Makefile.sync format-patches sync
Generating Patches
When working on new fixes or features that impact vendored code, use the following model. First get a clean tracking repo with all current patches applied:
make -f Makefile.sync clean apply-patches
Iterate until you're ready to submit PRs. Once your code is ready, commit a change in the ./vendor/ directory, then generate the patches for ollama with
make -f Makefile.sync format-patches
In your ./vendor/ directory, create a branch, and cherry-pick the new commit to that branch, then submit a PR upstream to llama.cpp.
Commit the changes in the ollama repo and submit a PR to Ollama, which will include the vendored code update with your change, along with the patches.
After your PR upstream is merged, follow the Updating Base Commit instructions above, however first remove your patch before running apply-patches since the new base commit contains your change already.
Documentation
¶
Index ¶
- Variables
- func BackendInit()
- func FreeModel(model *Model)
- func FreeVocab(vocab *Vocab)
- func GetModelArch(modelPath string) (string, error)
- func Quantize(infile, outfile string, ftype uint32) error
- func SchemaToGrammar(schema []byte) []byte
- type Batch
- type ClipContext
- type Context
- func (c *Context) Decode(batch *Batch) error
- func (c *Context) GetEmbeddingsIth(i int) []float32
- func (c *Context) GetEmbeddingsSeq(seqId int) []float32
- func (c *Context) KvCacheClear()
- func (c *Context) KvCacheDefrag()
- func (c *Context) KvCacheSeqAdd(seqId int, p0 int, p1 int, delta int)
- func (c *Context) KvCacheSeqCp(srcSeqId int, dstSeqId int, p0 int, p1 int)
- func (c *Context) KvCacheSeqRm(seqId int, p0 int, p1 int) bool
- func (c *Context) Model() *Model
- func (c *Context) SetCrossAttention(state bool)
- func (c *Context) Synchronize()
- type ContextParams
- type MllamaContext
- type Model
- func (m *Model) AddBOSToken() bool
- func (m *Model) ApplyLoraFromFile(context *Context, loraPath string, scale float32, threads int) error
- func (m *Model) NEmbd() int
- func (m *Model) NumVocab() int
- func (m *Model) TokenIsEog(token int) bool
- func (m *Model) TokenToPiece(token int) string
- func (m *Model) Tokenize(text string, addSpecial bool, parseSpecial bool) ([]int, error)
- func (m *Model) Vocab() *C.struct_llama_vocab
- type ModelParams
- type Sampler
- type SamplingContext
- type SamplingParams
- type TokenData
- type Vocab
Constants ¶
This section is empty.
Variables ¶
var ErrKvCacheFull = errors.New("could not find a kv cache slot")
Functions ¶
func BackendInit ¶
func BackendInit()
func GetModelArch ¶
func SchemaToGrammar ¶
SchemaToGrammar converts the provided JSON schema to a grammar. It returns nil if the provided schema is invalid JSON or an invalid JSON schema.
Types ¶
type Batch ¶
type Batch struct {
// contains filtered or unexported fields
}
func NewBatch ¶
Creates a new batch for either word tokens or image embeddings (if embedSize is non-zero). Batches cannot contain both types at the same time. batchSize is the maximum number of entries that can be added per sequence
func (*Batch) Add ¶
Add adds either a token or an image embedding to the batch depending on the type when the batch was initialized. The other argument will be ignored. Adds to the batch with the given position for the given sequence ids, and optionally instructs to include logits.
func (*Batch) IsEmbedding ¶
type ClipContext ¶
type ClipContext struct {
// contains filtered or unexported fields
}
vision processing
func NewClipContext ¶
func NewClipContext(llamaContext *Context, modelPath string) (*ClipContext, error)
func (*ClipContext) Free ¶
func (c *ClipContext) Free()
type Context ¶
type Context struct {
// contains filtered or unexported fields
}
func NewContextWithModel ¶
func NewContextWithModel(model *Model, params ContextParams) (*Context, error)
func (*Context) GetEmbeddingsIth ¶
func (*Context) GetEmbeddingsSeq ¶
Get the embeddings for a sequence id
func (*Context) KvCacheClear ¶
func (c *Context) KvCacheClear()
func (*Context) KvCacheDefrag ¶
func (c *Context) KvCacheDefrag()
func (*Context) KvCacheSeqAdd ¶
func (*Context) KvCacheSeqCp ¶
func (*Context) SetCrossAttention ¶
func (*Context) Synchronize ¶
func (c *Context) Synchronize()
type ContextParams ¶
type ContextParams struct {
// contains filtered or unexported fields
}
func NewContextParams ¶
type MllamaContext ¶
type MllamaContext struct {
// contains filtered or unexported fields
}
func NewMllamaContext ¶
func NewMllamaContext(llamaContext *Context, modelPath string) (*MllamaContext, error)
func (*MllamaContext) EmbedSize ¶
func (m *MllamaContext) EmbedSize(llamaContext *Context) int
func (*MllamaContext) Free ¶
func (m *MllamaContext) Free()
type Model ¶
type Model struct {
// contains filtered or unexported fields
}
func LoadModelFromFile ¶
func LoadModelFromFile(modelPath string, params ModelParams) (*Model, error)
func (*Model) AddBOSToken ¶
func (*Model) ApplyLoraFromFile ¶
func (*Model) TokenIsEog ¶
func (*Model) TokenToPiece ¶
func (*Model) Vocab ¶
func (m *Model) Vocab() *C.struct_llama_vocab
type ModelParams ¶
type Sampler ¶
type Sampler struct {
// contains filtered or unexported fields
}
func NewGrammarSampler ¶
type SamplingContext ¶
type SamplingContext struct {
// contains filtered or unexported fields
}
sampling TODO: this is a temporary wrapper to allow calling C++ code from CGo
func NewSamplingContext ¶
func NewSamplingContext(model *Model, params SamplingParams) (*SamplingContext, error)
func (*SamplingContext) Accept ¶
func (s *SamplingContext) Accept(id int, applyGrammar bool)
func (*SamplingContext) Reset ¶
func (s *SamplingContext) Reset()
Source Files
Directories