Warning
This article was automatically translated by OpenAI (gpt-4o). It may be edited eventually, but please be aware that it may contain incorrect information at this time.
I tried llama-cpp-python in this article, but this time I will try the Java binding of llama.cpp.
Add the following dependency library:
<dependency>
<groupId>de.kherud</groupId>
<artifactId>llama</artifactId>
<version>2.2.1</version>
</dependency>
Write the following code:
package org.example;
import de.kherud.llama.InferenceParameters;
import de.kherud.llama.LlamaModel;
import de.kherud.llama.ModelParameters;
public class Main {
public static void main(String... args) {
ModelParameters modelParams = new ModelParameters.Builder()
.setNGpuLayers(1)
.build();
InferenceParameters inferParams = new InferenceParameters.Builder()
.setTemperature(1f)
.build();
try (LlamaModel model = new LlamaModel("/opt/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf", modelParams)) {
Iterable<LlamaModel.Output> outputs = model.generate("""
[INST]
Tell me a joke.
[/INST]
""", inferParams);
for (LlamaModel.Output output : outputs) {
System.out.print(output);
}
}
}
}
Execute:
$ mvn -q compile exec:java -Dexec.mainClass=org.example.Main 2> /dev/null
/de/kherud/llama/Mac/aarch64
Extracted 'ggml-metal.metal' to '/var/folders/6p/vxhp1wpj2mq5w8drct9k8t4w0000gq/T/ggml-metal.metal'
Extracted 'libllama.dylib' to '/var/folders/6p/vxhp1wpj2mq5w8drct9k8t4w0000gq/T/libllama.dylib'
Extracted 'libjllama.dylib' to '/var/folders/6p/vxhp1wpj2mq5w8drct9k8t4w0000gq/T/libjllama.dylib'
Why don't scientists trust atoms?
Because they make up everything!%
It seems to be getting easier to incorporate local LLM into Java applications.