VLLM serve is not working

#19
by codegood - opened

vllm serve prism-ml/Bonsai-27B-gguf
INFO 08-02 23:59:55 [init.py:239] Automatically detected platform cuda.
INFO 08-03 00:00:40 [api_server.py:1043] vLLM API server version 0.8.5.post1
INFO 08-03 00:00:40 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='prism-ml/Bonsai-27B-gguf', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='prism-ml/Bonsai-27B-gguf', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x71a4e3930d60>)
Traceback (most recent call last):
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 283, in get_config
raise ValueError(
ValueError: Could not detect config format for no config file found. Ensure your model has either config.json (HF format) or params.json (Mistral format).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/bin/vllm", line 12, in
sys.exit(main())
^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/uvloop/init.py", line 48, in wrapper
return await main
^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1099, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 987, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/config.py", line 451, in init
hf_config = get_config(self.hf_config_path or self.model,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/Kiran/OneDrive/Desktop/Projects/Inference practice/infer_env/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 300, in get_config
raise ValueError(error_message) from e
ValueError: Invalid repository ID or local directory specified: 'prism-ml/Bonsai-27B-gguf'.
Please verify the following requirements:

  1. Provide a valid Hugging Face repository ID.
  2. Specify a local directory that contains a recognized configuration file.
    • For Hugging Face models: ensure the presence of a 'config.json'.
    • For Mistral models: ensure the presence of a 'params.json'.
Prism ML org

which gguf were you using? there is no vllm support for most of these.
Can try the awq-4bit checkpoints we have with sglang.

Can you point out to me where is that sglang model and how to run it? Or are there any other ways to get inference speed for this 1 bit?

I tried these models

https://huggingface.co/prism-ml/Bonsai-27B-gguf/blob/main/Bonsai-27B-dspark-Q4_1.gguf
https://huggingface.co/prism-ml/Bonsai-27B-gguf/blob/main/Bonsai-27B-Q1_0.gguf

Prism ML org

the one that has dspark is not a model itself, its a draft model for speculative decoding cannot use it directly: https://github.com/PrismML-Eng/Bonsai-demo/blob/main/SPECULATIVE.md

The Q1_0 is okay but need to use llama.cpp thats not compatible with vllm. See more info here: https://github.com/PrismML-Eng/Bonsai-demo

The sglang one is served in 4-bit at the moment e.g. https://huggingface.co/prism-ml/Ternary-Bonsai-27B-AWQ-4bit

Sign up or log in to comment