Instructions to use prism-ml/Bonsai-27B-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use prism-ml/Bonsai-27B-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf prism-ml/Bonsai-27B-gguf:F16 # Run inference directly in the terminal: llama cli -hf prism-ml/Bonsai-27B-gguf:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf prism-ml/Bonsai-27B-gguf:F16 # Run inference directly in the terminal: llama cli -hf prism-ml/Bonsai-27B-gguf:F16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf prism-ml/Bonsai-27B-gguf:F16 # Run inference directly in the terminal: ./build/bin/llama cli -hf prism-ml/Bonsai-27B-gguf:F16
Use Docker
docker model run hf.co/prism-ml/Bonsai-27B-gguf:F16
- Llama new
- LM Studio
- Jan
- vLLM
How to use prism-ml/Bonsai-27B-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "prism-ml/Bonsai-27B-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prism-ml/Bonsai-27B-gguf", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/prism-ml/Bonsai-27B-gguf:F16
- Ollama
How to use prism-ml/Bonsai-27B-gguf with Ollama:
ollama run hf.co/prism-ml/Bonsai-27B-gguf:F16
- Unsloth Desktop
- Pi
How to use prism-ml/Bonsai-27B-gguf with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Bonsai-27B-gguf:F16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "prism-ml/Bonsai-27B-gguf:F16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use prism-ml/Bonsai-27B-gguf with Docker Model Runner:
docker model run hf.co/prism-ml/Bonsai-27B-gguf:F16
- Lemonade
How to use prism-ml/Bonsai-27B-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull prism-ml/Bonsai-27B-gguf:F16
Run and chat with the model
lemonade run user.Bonsai-27B-gguf-F16
List all available models
lemonade list
- Hermes Agent
How to use prism-ml/Bonsai-27B-gguf with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Bonsai-27B-gguf:F16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default prism-ml/Bonsai-27B-gguf:F16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use prism-ml/Bonsai-27B-gguf with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Bonsai-27B-gguf:F16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "prism-ml/Bonsai-27B-gguf:F16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
VLLM serve is not working
vllm serve prism-ml/Bonsai-27B-gguf
INFO 08-02 23:59:55 [init.py:239] Automatically detected platform cuda.
INFO 08-03 00:00:40 [api_server.py:1043] vLLM API server version 0.8.5.post1
INFO 08-03 00:00:40 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='prism-ml/Bonsai-27B-gguf', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='prism-ml/Bonsai-27B-gguf', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x71a4e3930d60>)
Traceback (most recent call last):
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 283, in get_config
raise ValueError(
ValueError: Could not detect config format for no config file found. Ensure your model has either config.json (HF format) or params.json (Mistral format).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/bin/vllm", line 12, in
sys.exit(main())
^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper
return await main
^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1099, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 987, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/config.py", line 451, in init
hf_config = get_config(self.hf_config_path or self.model,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 300, in get_config
raise ValueError(error_message) from e
ValueError: Invalid repository ID or local directory specified: 'prism-ml/Bonsai-27B-gguf'.
Please verify the following requirements:
- Provide a valid Hugging Face repository ID.
- Specify a local directory that contains a recognized configuration file.
- For Hugging Face models: ensure the presence of a 'config.json'.
- For Mistral models: ensure the presence of a 'params.json'.
which gguf were you using? there is no vllm support for most of these.
Can try the awq-4bit checkpoints we have with sglang.
Can you point out to me where is that sglang model and how to run it? Or are there any other ways to get inference speed for this 1 bit?
I tried these models
https://huggingface.co/prism-ml/Bonsai-27B-gguf/blob/main/Bonsai-27B-dspark-Q4_1.gguf
https://huggingface.co/prism-ml/Bonsai-27B-gguf/blob/main/Bonsai-27B-Q1_0.gguf
the one that has dspark is not a model itself, its a draft model for speculative decoding cannot use it directly: https://github.com/PrismML-Eng/Bonsai-demo/blob/main/SPECULATIVE.md
The Q1_0 is okay but need to use llama.cpp thats not compatible with vllm. See more info here: https://github.com/PrismML-Eng/Bonsai-demo
The sglang one is served in 4-bit at the moment e.g. https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit