Text Generation
Transformers
Safetensors
English
llama
distillation
synthetic data
function calling
structured outputs
json mode
conversational
text-generation-inference
Instructions to use NousResearch/Hermes-2-Theta-Llama-3-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use NousResearch/Hermes-2-Theta-Llama-3-70B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="NousResearch/Hermes-2-Theta-Llama-3-70B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("NousResearch/Hermes-2-Theta-Llama-3-70B") model = AutoModelForCausalLM.from_pretrained("NousResearch/Hermes-2-Theta-Llama-3-70B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use NousResearch/Hermes-2-Theta-Llama-3-70B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "NousResearch/Hermes-2-Theta-Llama-3-70B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Hermes-2-Theta-Llama-3-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/NousResearch/Hermes-2-Theta-Llama-3-70B
- SGLang
How to use NousResearch/Hermes-2-Theta-Llama-3-70B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "NousResearch/Hermes-2-Theta-Llama-3-70B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Hermes-2-Theta-Llama-3-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "NousResearch/Hermes-2-Theta-Llama-3-70B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Hermes-2-Theta-Llama-3-70B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use NousResearch/Hermes-2-Theta-Llama-3-70B with Docker Model Runner:
docker model run hf.co/NousResearch/Hermes-2-Theta-Llama-3-70B
Add example of chat templates for function calling (#3)
Browse files- Add example of chat templates for function calling (975d07b5c9e889b8f6d9c3c8d84dc245d3a87022)
Co-authored-by: Matthew Carrigan <Rocketknight1@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -131,6 +131,51 @@ The stock fundamentals data for Tesla (TSLA) are as follows:
|
|
| 131 |
This information provides a snapshot of Tesla's financial position and performance based on the fundamental data obtained from the yfinance API. It shows that Tesla has a substantial market capitalization and a relatively high P/E and P/B ratio compared to other stocks in its industry. The company does not pay a dividend at the moment, which is reflected by a 'Dividend Yield' of 'None'. The Beta value indicates that Tesla's stock has a moderate level of volatility relative to the market. The 52-week high and low prices give an idea of the stock's range over the past year. This data can be useful when assessing investment opportunities and making investment decisions.<|im_end|>
|
| 132 |
```
|
| 133 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
## Prompt Format for JSON Mode / Structured Outputs
|
| 135 |
|
| 136 |
Our model was also trained on a specific system prompt for Structured Outputs, which should respond with **only** a json object response, in a specific json schema.
|
|
|
|
| 131 |
This information provides a snapshot of Tesla's financial position and performance based on the fundamental data obtained from the yfinance API. It shows that Tesla has a substantial market capitalization and a relatively high P/E and P/B ratio compared to other stocks in its industry. The company does not pay a dividend at the moment, which is reflected by a 'Dividend Yield' of 'None'. The Beta value indicates that Tesla's stock has a moderate level of volatility relative to the market. The 52-week high and low prices give an idea of the stock's range over the past year. This data can be useful when assessing investment opportunities and making investment decisions.<|im_end|>
|
| 132 |
```
|
| 133 |
|
| 134 |
+
## Chat Templates for function calling
|
| 135 |
+
|
| 136 |
+
You can also use chat templates for function calling. For more information, please see the relevant section of the [chat template documentation](https://huggingface.co/docs/transformers/en/chat_templating#advanced-tool-use--function-calling).
|
| 137 |
+
|
| 138 |
+
Here is a brief example of this approach:
|
| 139 |
+
|
| 140 |
+
```python
|
| 141 |
+
def multiply(a: int, b: int):
|
| 142 |
+
"""
|
| 143 |
+
A function that multiplies two numbers
|
| 144 |
+
|
| 145 |
+
Args:
|
| 146 |
+
a: The first number to multiply
|
| 147 |
+
b: The second number to multiply
|
| 148 |
+
"""
|
| 149 |
+
return int(a) * int(b)
|
| 150 |
+
|
| 151 |
+
tools = [multiply] # Only one tool in this example, but you probably want multiple!
|
| 152 |
+
|
| 153 |
+
model_input = tokenizer.apply_chat_template(
|
| 154 |
+
messages,
|
| 155 |
+
tools=tools
|
| 156 |
+
)
|
| 157 |
+
```
|
| 158 |
+
|
| 159 |
+
The docstrings and type hints of the functions will be used to generate a function schema that will be read by the chat template and passed to the model.
|
| 160 |
+
Please make sure you include a docstring in the same format as this example!
|
| 161 |
+
|
| 162 |
+
If the model makes a tool call, you can append the tool call to the conversation like so:
|
| 163 |
+
|
| 164 |
+
```python
|
| 165 |
+
tool_call_id = "vAHdf3" # Random ID, should be unique for each tool call
|
| 166 |
+
tool_call = {"name": "multiply", "arguments": {"a": "6", "b": "7"}}
|
| 167 |
+
messages.append({"role": "assistant", "tool_calls": [{"id": tool_call_id, "type": "function", "function": tool_call}]})
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
Next, call the tool function and append the tool result:
|
| 171 |
+
|
| 172 |
+
```python
|
| 173 |
+
messages.append({"role": "tool", "tool_call_id": tool_call_id, "name": "multiply", "content": "42"})
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
And finally apply the chat template to the updated `messages` list and `generate()` text once again to continue the conversation.
|
| 177 |
+
|
| 178 |
+
|
| 179 |
## Prompt Format for JSON Mode / Structured Outputs
|
| 180 |
|
| 181 |
Our model was also trained on a specific system prompt for Structured Outputs, which should respond with **only** a json object response, in a specific json schema.
|