From the course: Rust LLMOps

Unlock this course with a free trial

Join today to access over 25,600 courses taught by industry experts.

Rust chat inference

Rust chat inference - Rust Tutorial

From the course: Rust LLMOps

Start my 1-month free trial Buy for my team

Rust chat inference

“

- [Instructor] Let's take a look at how you could build a Rust chatbot for large language model inference using some of the plug and play components in the Rust ecosystem. First up, you could look at using the Falcon model, which is a great place to start with, and this could excel at natural language conversation, and then you could load that model into a Rust program using something like Candle or Rust Bird, et cetera, which has access to the Hugging Face models. Then you could integrate that conversational model into a Rust chatbot. So how would you do this? Well, we could build a chatbot UI with a crate like cursive, which is a terminal based interface in Rust, and this could take user input and print bot responses. That core chatbot logic loop would then continuously prompt the user for input, who could take their input text, pass it to the Hugging Face model loaded earlier, and then print the model's response, and this Hugging Face model would handle all of the natural language…

Contents