Native ML Runtimes

Run ONNX, GGUF, and Diffusers models natively on your Island without Docker containers

Native ML Runtimes

Islands can execute ML models natively — without Docker — using built-in runtimes for ONNX, GGUF (llama.cpp), and Stable Diffusion (candle). This eliminates container overhead and enables hardware-aware model preloading at startup.

Overview

Building with ML Support

The pre-built Island binary includes all runtimes. If building from source:

# All runtimes
cargo build --release --features all-runtimes

# Individual runtimes
cargo build --release --features gguf
cargo build --release --features onnx
cargo build --release --features "onnx,gguf"

# System dependencies (Linux)
sudo apt-get install cmake libclang-dev  # Required for GGUF

The all-runtimes feature enables onnx, gguf, and diffusers together.

Model Resolution — hf:// URIs

All native runtimes download models from HuggingFace using the hf:// URI scheme:

Auto-Discovery Priority

When no filename is specified, the Island queries the HuggingFace API and picks the best file:

1. GGUF files — prefers Q4_K_M quantization, then Q4_K_S, Q5_K_M, Q8_0
2. ONNX files — model.onnx or onnx/model.onnx
3. Safetensors — model.safetensors (for diffusers/transformers)
4. PyTorch — pytorch_model.bin (legacy fallback)

Model Cache

Downloaded models are cached at ~/.island/model-cache/. The cache uses LRU eviction when max_cache_gb is exceeded (default: 20 GB).

[model_cache]
max_cache_gb = 50         # Increase for Islands hosting many models
cache_dir = "/data/models" # Optional: custom cache directory

Models are verified with SHA256 hashes when the Cargo specifies one. The cache persists across restarts — a cached model is never re-downloaded.

Startup Preloading

At startup, the Island automatically selects and downloads a starter set of models based on detected hardware:

Preloading runs in the background — the Island accepts jobs immediately while models download.

Configuration

[preload]
enabled = true    # Set to false to skip preloading

# Override auto-selection with specific models:
# models = [
#   "hf://TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
#   "hf://openai/whisper-base"
# ]

When models is empty (default), the Island auto-selects based on the hardware tier table above. When explicit models are listed, only those are preloaded.

GGUF Runtime (LLM Inference)

The GGUF runtime uses llama.cpp via Rust bindings to run large language models natively.

Supported Models

Any GGUF-format model works. The Archipelag.io Cargo catalog includes:

• Qwen3.5 family (0.8B, 4B, 9B, 27B, 35B-A3B MoE)
• Mistral 7B Instruct v0.2
• Llama 3.1 8B Instruct
• GPT-OSS 20B
• GLM-5, MiniMax-M2.5, Nanbeige4.1-3B

How It Works

1. The coordinator dispatches a job with runtime_type: "llmcpp" and model_url: "hf://..."
2. The Island downloads the GGUF file (or uses the cached version)
3. llama.cpp loads the model and creates an inference session
4. Tokens stream back to the user in real-time via NATS

Input Format

{
  "prompt": "Explain quantum computing in simple terms",
  "max_tokens": 512
}

Or OpenAI-compatible chat format:

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is quantum computing?"}
  ],
  "max_tokens": 512
}

Configuration

The model’s model_context_size and model_temperature are set per-Cargo in the coordinator. The Island applies temperature, top-p, and repetition penalty sampling.

ONNX Runtime (ML Inference)

The ONNX runtime uses ONNX Runtime for cross-platform ML inference.

Supported Task Types

Text tasks automatically download tokenizer.json from the same HuggingFace repo as the model.

Diffusers Runtime (Image & Video Generation)

The diffusers runtime uses candle for Stable Diffusion and similar pipelines.

Supported Models

• Stable Diffusion 1.5, 3.5 Large
• FLUX.1 dev and schnell
• Z-Image-Turbo, Qwen-Image
• Video: Wan2.1-T2V, HunyuanVideo, CogVideoX

How It Works

1. Downloads model components from HuggingFace (tokenizer, CLIP, VAE, UNet)
2. Encodes text prompt via CLIP
3. Runs the diffusion loop with classifier-free guidance
4. Decodes latents to image via VAE
5. Returns base64-encoded PNG

Input Format

{
  "prompt": "A sunset over mountains, oil painting style",
  "negative_prompt": "blurry, low quality",
  "num_steps": 20,
  "guidance_scale": 7.5,
  "width": 512,
  "height": 512,
  "seed": 42
}

Progress events stream during generation (step 1/20, step 2/20, …).

Heartbeat & Placement

The Island advertises its supported runtimes in every heartbeat:

{
  "supported_runtimes": ["container", "wasm", "llmcpp", "onnx", "diffusers"],
  "gpu_model": "NVIDIA RTX 4090",
  "gpu_vram_mb": 24576,
  "ram_mb": 32768
}

The coordinator uses this information during job placement — an ONNX job will only be dispatched to an Island that reports "onnx" in its supported_runtimes. If no capable Island is online, the job waits in the queue.

Next Steps

{% end %}