Add files using upload-large-folder tool
Browse files- UPSTREAM_README.md +94 -0
- xemogpt.ckpt +3 -0
UPSTREAM_README.md
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
|
| 2 |
+
|
| 3 |
+

|
| 4 |
+
|
| 5 |
+
## 🧩 Environment Setup
|
| 6 |
+
|
| 7 |
+
**Step 1: Create Conda Environment**
|
| 8 |
+
|
| 9 |
+
```bash
|
| 10 |
+
conda env create -f environment.yaml
|
| 11 |
+
conda activate XEmoGPT
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
**Step 2: Download Models and Set Paths**
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
* **XEmoGPT**
|
| 18 |
+
|
| 19 |
+
* Config file: `eval_configs/inference_config.yaml`
|
| 20 |
+
* Link: [https://pan.baidu.com/s/1LjycShaumIP3p6mzMnMU7w?pwd=9c2a](https://pan.baidu.com/s/1LjycShaumIP3p6mzMnMU7w?pwd=9c2a)
|
| 21 |
+
* **Qwen3-4B**
|
| 22 |
+
|
| 23 |
+
* Config file: `eval_configs/inference_config.yaml`
|
| 24 |
+
* Link: [https://huggingface.co/Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B)
|
| 25 |
+
|
| 26 |
+
* **chinese-hubert-large**
|
| 27 |
+
|
| 28 |
+
* Config file: `inference.py`
|
| 29 |
+
* Link: [https://huggingface.co/TencentGameMate/chinese-hubert-large](https://huggingface.co/TencentGameMate/chinese-hubert-large)
|
| 30 |
+
|
| 31 |
+
* **LaBSE**
|
| 32 |
+
|
| 33 |
+
* Config file: `inference.py`
|
| 34 |
+
* Link: [https://huggingface.co/sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE)
|
| 35 |
+
|
| 36 |
+
* **clip-vit-large-patch14**
|
| 37 |
+
|
| 38 |
+
* Config file: `inference.py`
|
| 39 |
+
* Link: [https://huggingface.co/openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## 🎬 How to Run Inference
|
| 45 |
+
|
| 46 |
+
Run the following command in your terminal, specifying paths to the input video, audio, and subtitle file:
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
python inference.py \
|
| 50 |
+
--cfg-path eval_configs/inference_config.yaml \
|
| 51 |
+
--video-path assets/sample_00000007.mp4 \
|
| 52 |
+
--audio-path assets/sample_00000007.wav \
|
| 53 |
+
--subtitle "Be there or be square. Oh, okay, goodbye."
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
---
|
| 57 |
+
|
| 58 |
+
## 📐 EmoCue-360 Evaluation
|
| 59 |
+
|
| 60 |
+
### Download Required Model
|
| 61 |
+
|
| 62 |
+
* **all-MiniLM-L6-v2**
|
| 63 |
+
|
| 64 |
+
* Config file: `EmoCue-360/compute_prf.py`
|
| 65 |
+
* Link: [https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
|
| 66 |
+
|
| 67 |
+
### Usage Instructions
|
| 68 |
+
|
| 69 |
+
#### Step 1: Extract Emotional Cues (`extract_clue.py`)
|
| 70 |
+
|
| 71 |
+
```python
|
| 72 |
+
INPUT_PATH = "/your/path/to/final-EMER-reason.csv"
|
| 73 |
+
emer_dict = pd.read_csv(INPUT_PATH).set_index('name')['english'].to_dict()
|
| 74 |
+
result = extract_vae_clue_from_dict(emer_dict)
|
| 75 |
+
|
| 76 |
+
OUTPUT_PATH = "/your/path/to/output.json"
|
| 77 |
+
save_dict_to_json(result, OUTPUT_PATH)
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
#### Step 2: Compute PRF Scores (`compute_prf.py`)
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
MODEL_PATH = "/your/path/to/all-MiniLM-L6-v2"
|
| 84 |
+
model = EmoCue360(MODEL_PATH)
|
| 85 |
+
|
| 86 |
+
emer_gt_dict = load_json_to_dict("/your/path/to/EMER_clue.json")
|
| 87 |
+
rf_dict_xemogpt_emer = load_json_to_dict("/your/path/to/xemogpt_clue.json")
|
| 88 |
+
|
| 89 |
+
print("Visual: ", model.compute_visual_prf(rf_dict_xemogpt_emer, emer_gt_dict))
|
| 90 |
+
print("Auditory: ", model.compute_auditory_prf(rf_dict_xemogpt_emer, emer_gt_dict))
|
| 91 |
+
print("Global: ", model.compute_emotional_prf(rf_dict_xemogpt_emer, emer_gt_dict))
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
---
|
xemogpt.ckpt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f678dbfbaef02d398d10a03b5d51acf098702c08e4198a12d5394869eae4c82b
|
| 3 |
+
size 2813197035
|