Instructions to use mr-abhisharma/AceNemotron-14B-Quantize-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mr-abhisharma/AceNemotron-14B-Quantize-8bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="mr-abhisharma/AceNemotron-14B-Quantize-8bit") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("mr-abhisharma/AceNemotron-14B-Quantize-8bit") model = AutoModelForCausalLM.from_pretrained("mr-abhisharma/AceNemotron-14B-Quantize-8bit", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use mr-abhisharma/AceNemotron-14B-Quantize-8bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "mr-abhisharma/AceNemotron-14B-Quantize-8bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mr-abhisharma/AceNemotron-14B-Quantize-8bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/mr-abhisharma/AceNemotron-14B-Quantize-8bit
- SGLang
How to use mr-abhisharma/AceNemotron-14B-Quantize-8bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "mr-abhisharma/AceNemotron-14B-Quantize-8bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mr-abhisharma/AceNemotron-14B-Quantize-8bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "mr-abhisharma/AceNemotron-14B-Quantize-8bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mr-abhisharma/AceNemotron-14B-Quantize-8bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use mr-abhisharma/AceNemotron-14B-Quantize-8bit with Docker Model Runner:
docker model run hf.co/mr-abhisharma/AceNemotron-14B-Quantize-8bit
🧠 AceNemotron-14B 8-Bit Quantized Model
Welcome to the 8-bit quantized version of the AceNemotron-14B model! This release brings reduced memory usage and faster inference speeds, making it suitable for deployment on lower-resource hardware without sacrificing much performance.
📦 Model Details
- Base Model: AceNemotron-14B
- Quantization: 8-bit
- Quantized With: BitsAndBytes
- Precision: Int8
- Use Case: Faster inference, lower memory footprint, efficient finetuning
- Uploader: mr-abhisharma
🛠 Installation
To use this model efficiently, make sure you have the required dependencies installed:
pip install transformers accelerate bitsandbytes
🚀 Usage
Here's a basic example using transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "mr-abhisharma/AceNemotron-14B-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_8bit=True,
device_map="auto"
)
inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
🧠 Why Quantized?
Quantization to 8-bit reduces model size drastically and improves inference speed, especially on consumer-grade GPUs (e.g. 16GB or lower). It's great for:
- Personal use and experimentation
- Running on laptops or single-GPU setups
- Cost-effective inference deployment
🔧 Limitations
- Slight drop in precision may occur.
- This model inherits the limitations of the base AceNemotron-14B model.
- Use responsibly—ensure compliance with licensing and intended use.
🤝 Acknowledgements
- Base model: AceNemotron Project
- Quantization tools: BitsAndBytes
- Inspired by the Hugging Face community ❤️
- Downloads last month
- 5