AI (Large Language Model[LLM) chatbots) are all the rage. However, data privacy, hallucinations, and response scope are all issues. To be able to experiment with LLMs and other models, I've added an Ollama server to my home lab.
Ollama (ollama.com) is an application that simplifies setting up inference for large language models (LLMs). They also host most of the popular distilled models. This makes for a much more safe environment to play with these models. There are a ton of features I've used and continue to play with, including restricting the source data available for inference (similar to Retrieval Augmented Generation [RAG]).
As "AI" becomes used by everyone for everything, I believe it is important to have a little bit of first hand experience with these models and running them locally. For instance, observing, and paying for, the amount of energy they consume for inference of simple tasks is eye opening.
