A Complete Guide to Open-source LLM Local Serving: Inference Performance and Cost Optimization Strategies with vLLM and Ollama

Recent advancements in Large Language Models (LLMs) are bringing innovation to various industries at an unprecedented pace. Despite the convenience of cloud-based LLM APIs, many enterprises and development teams keenly feel the need for LLM serving in local environments and the optimization of inference performance due to concerns regarding data security, high operational costs, and service latency. Particularly, with the rapid emergence of the latest open-source models such as Llama 3 and Mistral, the endeavor to efficiently deploy these models on proprietary infrastructure has become a prominent topic.

Against this backdrop, this article aims to provide a practical guide for successfully serving open-source LLMs in a local environment and optimizing inference speed and costs, utilizing two powerful tools: vLLM and Ollama. From the core technical principles of each tool to specific installation and application methods, and through a comparative analysis of both solutions, this article will deliver the insights necessary to establish the most suitable optimization strategy for an organization's specific environment and requirements. Complex AI concepts will be elucidated with a focus on practical applicability.

The Rise of Open-source LLMs and the Importance of Local Serving

The advancement of LLM technology is no longer the exclusive domain of specific big tech companies. In recent years, with the widespread release of high-performing open-source LLMs such as Meta's Llama series and Mistral AI's Mistral models, an era has arrived where high-performance AI models are accessible and usable by all. These models offer flexible licenses that even permit commercial use, enabling numerous developers and enterprises to build innovative applications.

However, utilizing cloud-based LLM APIs inherently presents limitations such as unpredictably high costs, security concerns regarding the transmission of sensitive data externally, and increased service response times due to network latency. Particularly, for services requiring repetitive inference or the processing of large volumes of data, cloud expenses can escalate rapidly. To address these challenges, the importance of 'local serving' — deploying and operating LLMs directly in on-premise or edge environments — is increasingly significant. Local serving is gaining attention as a powerful alternative that enhances cost-efficiency, ensures data sovereignty, and allows for flexible utilization of customized models.

Accelerating LLM Inference with vLLM: Core Principles and Application

vLLM is a high-performance serving framework for LLM inference, particularly strong in significantly improving the inference throughput of large language models. Unpacking its core principles reveals the application of several innovative technologies aimed at enhancing the inefficiencies of existing serving systems.

vLLM's Core Technology: PagedAttention

The most crucial technology in vLLM is PagedAttention. Intuitively understood, it is a method inspired by the virtual memory and paging techniques used in operating systems. The Key and Value cache (KV Cache) generated during LLM inference occupies significant GPU memory, and its size varies with each request. PagedAttention manages the KV Cache in fixed-size 'blocks' and allocates these blocks non-contiguously, thereby reducing memory fragmentation and maximizing GPU memory utilization efficiency. Consequently, it enables the simultaneous processing of more requests, significantly improving the throughput of LLM serving.

vLLM Installation and Llama 3 Serving Example

The process of serving models like Llama 3 using vLLM is relatively straightforward. First, the vLLM library must be installed on a system with a PyTorch and CUDA environment. The following presents the basic installation and serving commands.

pip install vllm

Once the installation is complete, the Llama 3 model can be downloaded from the Hugging Face model repository and an API server can be launched using the following command. Here, the meta-llama/Meta-Llama-3-8B-Instruct model is used as an example.

python -m vllm.entrypoints.api_server --model meta-llama/Meta-Llama-3-8B-Instruct

Executing the command above will typically start an OpenAI-compatible API server on port http://localhost:8000. Requests can then be sent to the model using curl or a Python client.

import requests
import json

API_URL = "http://localhost:8000/v1/completions"

headers = { "Content-Type": "application/json" }

data = { "model": "meta-llama/Meta-Llama-3-8B-Instruct", # vLLM 서버 실행 시 지정한 모델명 "prompt": "AI Security의 핵심 과제는 무엇인가요?", "max_tokens": 100, "temperature": 0.7 }

response = requests.post(API_URL, headers=headers, data=json.dumps(data))

if response.status_code == 200: print(response.json()) else: print(f"Error: {response.status_code} - {response.text}")

As demonstrated, vLLM offers excellent inference performance without complex configurations, thereby establishing a robust foundation for the development of LLM-based applications.

Establishing a Local LLM Environment with Ollama: Simplicity and Flexibility

Ollama is a tool designed to enable easy and rapid execution of LLMs in a local environment. It provides a concise interface that allows models to be 'pulled' and 'run' in a manner similar to Docker, significantly lowering the barrier to LLM adoption. Notably, it supports various open-source models, assisting developers in quickly testing and utilizing models without complex environment configurations.

Features and Usage of Ollama

The most significant features of Ollama are its user-friendly Command Line Interface (CLI) and powerful model management capabilities. Various pre-trained models are provided by the community, and these models are distributed in a compressed format, enabling efficient storage space utilization. Furthermore, it incorporates an embedded API server, which greatly facilitates integration with local applications.

Ollama Installation and Mistral Serving Example

The method for installing Ollama may vary slightly depending on the operating system, but it can be easily installed using the script provided on the official website. The following presents the installation command for a Linux environment.

curl -fsSL https://ollama.com/install.sh | sh

After installation, the desired model can be 'pulled' to the local system and executed immediately. Here, the mistral model is used as an example.

ollama pull mistral
ollama run mistral

Executing the ollama run mistral command initiates an interactive prompt, allowing direct interaction with the model. Additionally, Ollama automatically runs an API server in the background, enabling external applications to utilize the model via HTTP requests.

# API 서버가 백그라운드에서 실행 중임을 가정
curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "AI 보안에 있어서 프롬프트 인젝션은 왜 중요한가요?"
}'

Ollama is highly beneficial during development and experimental phases, offering the flexibility to enable rapid prototyping.

vLLM vs. Ollama: Comparative Analysis for Optimal Selection

Both vLLM and Ollama are powerful tools utilized for local LLM serving; however, they differ in their design philosophies and strengths. The appropriate tool may vary depending on the intended purpose and environment, thus necessitating a clear comparative analysis.

In conclusion, vLLM may be a more suitable choice for production environments that handle large-scale traffic and demand high throughput. Conversely, if the objective is to quickly and easily experiment with various LLMs locally or use them for development and testing purposes, Ollama will offer excellent simplicity. It is crucial to select the optimal solution by considering the characteristics of the project and the operational environment.

GPU Hardware Considerations for Local LLM Serving

Efficient local serving of open-source LLMs necessitates robust GPU hardware support. The inference performance of LLMs is heavily dependent on the type and capacity of the GPU; therefore, careful consideration must be given to hardware selection.

The Importance of VRAM Capacity

The most critical factor is the VRAM (Video RAM) capacity of the GPU. LLMs load model parameters and the KV Cache generated during inference into VRAM. Models such as Llama 3 8B typically require approximately 16GB of VRAM at full precision (FP16), and this requirement can be reduced through quantization. However, to use larger models or load multiple models concurrently, a GPU with 24GB or 48GB of VRAM or more is necessary. Insufficient VRAM can lead to models failing to load or significantly degraded inference speeds.

GPU Compute Performance and Bandwidth

Beyond VRAM capacity, the GPU's compute performance (number of CUDA cores, Tensor cores) and memory bandwidth also directly impact inference speed. Generally, data center GPUs such as NVIDIA's RTX 30/40 series, A100, or H100 demonstrate high efficiency for LLM inference. Furthermore, for Multi-GPU configurations, systems supporting high-speed communication between GPUs, such as NVLink, are advantageous.

Hardware selection is a process of balancing budget constraints and performance requirements. The optimal GPU should be chosen by comprehensively considering factors such as model size, anticipated traffic, and required response times.

Advanced Strategies for Performance Optimization

Even when utilizing vLLM and Ollama, advanced optimization strategies exist to achieve higher performance and efficiency. These strategies can be particularly useful when resource constraints are present or specific performance targets must be met.

Model Quantization

Model quantization is a technique that reduces model file size and VRAM usage by lowering the precision of the model. For example, it involves converting model parameters expressed in 16-bit floating-point (FP16) to 8-bit integers (INT8) or 4-bit integers (INT4). While this process may introduce a slight degradation in performance, it significantly reduces VRAM consumption, enabling the loading of larger models or the execution of models on constrained hardware. Ollama inherently provides quantized models, and vLLM also supports specific quantization formats to simultaneously improve performance and memory efficiency.

Speculative Decoding

Speculative Decoding is a technique that accelerates the token generation speed of LLMs. The core principle involves a small 'Draft Model' rapidly predicting multiple tokens, and then a larger 'Oracle Model' validating this predicted token sequence in a single pass. If the Oracle Model accepts the prediction, tokens are generated quickly; otherwise, the Oracle Model directly generates the tokens. This approach accelerates the slow sampling process of large models, thereby reducing overall inference latency. vLLM supports this feature, which can further enhance inference speed.

Multi-GPU Utilization and Tensor Parallelism

For extremely large LLMs that cannot be handled by the VRAM or computational performance of a single GPU, Multi-GPU utilization is essential. vLLM supports Tensor Parallelism, allowing each layer of a model to be divided and loaded across multiple GPUs, thereby parallelizing inference. This is particularly useful when serving very large models that are challenging to fit into a single GPU's VRAM. It can be easily configured using the --tensor-parallel-size option when running the vLLM API server.

python -m vllm.entrypoints.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.8

The command above serves the Llama 3 model using two GPUs, with each GPU configured to utilize only 80% of its VRAM. The key to achieving optimal performance and cost-efficiency lies in appropriately combining these advanced strategies.

Troubleshooting

Common issues that may be encountered when serving LLMs in a local environment, along with their resolutions, are discussed. As various variables exist in real-world environments, the solutions provided below should be adapted to specific situations.

1. CUDA Out of Memory Error

• Solutions:
  • Check the current GPU VRAM usage with the nvidia-smi command.
  • Utilize a smaller model or a quantized model.
  • When using vLLM, limit VRAM usage through the --gpu-memory-utilization option. For instance, --gpu-memory-utilization 0.8 sets the GPU VRAM utilization to 80%.
  • Reduce the --max-model-len to limit context length, thereby decreasing KV Cache usage.
  • If feasible, upgrade to a GPU with more VRAM or consider a Multi-GPU configuration.

2. Model Loading Failure or Compatibility Issues

• Solutions:
  • Verify the correctness of the model path and, for Hugging Face models, confirm the accuracy of the model ID.
  • For vLLM, ensure the model is saved in PyTorch format. For Ollama, confirm whether the model has been converted to the Ollama format or is an officially supported model.
  • Check if all dependency packages for the Python environment are installed, and install pip install -r requirements.txt or individual packages if necessary.
  • For vLLM, update to the latest version to receive bug fixes and support for newer models.

3. API Server Port Conflict

• Solutions:
  • Verify if any process is utilizing the port in question using the command netstat -tulnp | grep <port_number>.
  • For vLLM, specify a different port number using the --port option (e.g., --port 8001).
  • For Ollama, the port number can be modified by setting the OLLAMA_HOST environment variable (e.g., export OLLAMA_HOST="0.0.0.0:11435").

Furthermore, issues such as GPU driver version discrepancies and Python environment management problems may arise; therefore, it is crucial to consistently maintain the latest GPU drivers and utilize virtual environments to prevent dependency conflicts.

Practical Application / Case Studies

Local LLM serving utilizing vLLM and Ollama offers significant advantages in various real-world environments. Instead of specifying particular company names, application scenarios will be described focusing on environments and roles.

1. Rapid Proof of Concept (PoC) Development by Internal Development Teams

Internal development teams envisioning new AI services or features must rapidly validate ideas and build prototypes. Cloud LLM APIs make indiscriminate testing difficult due to cost concerns and hinder the use of sensitive internal data due to data security regulations. In such cases, Ollama can be leveraged to easily deploy and test Llama 3 or Mistral models on team members' local workstations. Prior to adoption, idea validation was time-consuming and incurred cloud costs; however, post-adoption, model deployment time is significantly reduced, development costs are minimized, and rapid PoC development utilizing internal data becomes feasible. This shortens development cycles and enables the swift realization of innovative ideas.

2. Enhanced Security for Data-Sensitive Research Institutes/Organizations

Research institutes or organizations handling sensitive personal information or confidential data, such as those in healthcare, finance, or defense, face severe restrictions on using external cloud LLM APIs due to the risk of data breaches. In such environments, on-premise LLM serving using vLLM is essential. Prior to adoption, there were significant limitations on data analysis or model development using LLMs; however, post-adoption, all data processing and model inference occur within the organization's internal network, allowing for complete control over data sovereignty and security. Moreover, thanks to vLLM's high-performance inference engine, researchers can perform complex query-answering or analytical tasks on large datasets with high throughput. This results in maximizing research productivity while ensuring regulatory compliance.

3. Deployment in Edge Devices or Limited Infrastructure Environments

In restricted infrastructure environments, such as edge devices in industrial settings or small data centers with unstable network connections, cloud-dependent AI services are not suitable. In these environments, models can be deployed by combining quantized models with the lightweight serving approaches of Ollama or vLLM. Prior to adoption, implementing AI functionalities requiring real-time responses was challenging, or service interruptions frequently occurred due to network issues; however, post-adoption, performing LLM inference independently locally minimizes response latency and ensures service continuity even in the event of network failures. Particularly, Ollama's single-binary deployment makes it easily applicable in edge environments, providing a foundation for delivering powerful AI capabilities even with limited resources.

As demonstrated, vLLM and Ollama contribute to enhancing the usability of LLMs and resolving cost and performance issues in various practical environments, each based on its respective strengths.

Future Outlook

The local serving technology for open-source LLMs is continuously advancing, and its future is boundless. Improvements in inference engine efficiency, model lightweighting, and hardware technology will persistently progress.

Firstly, inference engines like vLLM will further enhance throughput and latency by introducing a wider array of optimization techniques beyond PagedAttention. Speculative Decoding and model parallelization technologies will mature, and new forms of caching strategies and batching techniques are anticipated to emerge. Secondly, research into model lightweighting itself will be actively pursued. 'Small Powerful Models' that achieve high performance with smaller sizes will continue to appear, and various quantization techniques will be standardized, enabling LLMs to operate with fewer resources. Thirdly, advancements in AI acceleration hardware, including GPUs, will further expand the performance limits of local serving. Larger VRAM, faster computational speeds, and more efficient multi-GPU communication technologies will become more prevalent, enabling the establishment of significantly more powerful local LLM environments than currently possible.

In anticipation of these changes, practitioners need to continuously study trends in the latest inference engine technologies and consistently benchmark new open-source models and their corresponding optimization strategies. Furthermore, accumulating experience in LLM deployment and management in on-premise and edge environments is crucial, and sustained efforts to find the optimal balance between cost and performance are necessary. It is important to observe how local LLM serving technology will evolve.

Conclusion

This article has thoroughly examined open-source LLM local serving and inference speed optimization strategies utilizing vLLM and Ollama. The core takeaways from this article are summarized as follows:

• Emphasizing the Importance of Open-source LLMs: The necessity for local serving is growing to overcome the limitations of cloud-based LLMs and achieve cost-efficiency, data sovereignty, and reduced latency.
• High-Performance Inference with vLLM: Innovative technologies such as PagedAttention and Continuous Batching can significantly enhance LLM inference throughput in large-scale traffic environments.
• Simplified Deployment with Ollama: It provides a foundation for easily and quickly executing and testing LLMs in a local environment through its user-friendly CLI and model management system.
• Hardware and Advanced Optimization: Performance can be maximized through advanced strategies such as selecting GPUs with ample VRAM capacity, quantization, Speculative Decoding, and Multi-GPU utilization.

For practical application, it is effective to first clearly define the project's characteristics and requirements, and then comparatively analyze which of vLLM or Ollama is more suitable. Initially, building rapid prototypes with Ollama may be considered, and for performance-critical production stages, transitioning to vLLM or exploring a hybrid strategy that combines both can be examined. Local LLM serving will be a powerful driving force that not only reduces costs but also accelerates the democratization of AI technology and creates new business opportunities.

The capability to effectively adopt and manage these technologies will be a core competitive advantage in leading the AI era. Through continuous technological learning and practical application, it is anticipated that LLM projects will successfully advance.