ShivomH/Elixir-MentalHealth-3B

Model Details

Model Description

Elixir-MentalHealth is a fine-tuned version of Meta-Llama-3.2-3B-Instruct, adapted using QLoRA on a curated dataset of single-turn and multi-turn mental health support conversations. The model is designed to provide empathetic, safe, and supportive responses while maintaining clear professional boundaries.

⚠️ Disclaimer: This model is not a replacement for professional mental health services. Always seek help from licensed professionals in crisis situations.

Key Features

• 🤝 Empathetic and supportive: prioritizes compassionate, non-judgmental responses.
• 🛡️ Safety-aware: avoids giving personal information, medical diagnoses, or unsafe advice.
• 🧠 Context-aware: handles multi-turn conversations smoothly while retaining coherence.
• ⚡ Lightweight deployment: thanks to QLoRA, only ~93MB adapter needs to be loaded for fine-tuned behavior.
• 📝 Transparent training setup: reproducible configuration and clear documentation.

Primary Use Cases:

• Mental health support chats
• Stress and Anxiety management conversations
• Empathetic listening, encouragement and general guidance
• Psychoeducational tips (e.g., mindfulness, coping strategies, depression support)

Out-of-Scope Use (should NOT be used for):

• Medical diagnosis or treatment planning
• Emergency mental health intervention (e.g., suicide prevention crisis line replacement)
• Legal, financial, or unrelated domains

This model is best suited for research, prototyping, and supportive chatbot applications where professional disclaimers and human oversight are always present.

---

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ShivomH/Elixir-MentalHealth-3B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a supportive and empathetic AI assistant trained to provide warm and professional mental health guidance."},
    {"role": "user", "content": "I am tired of living. Please suggest me strategies to get rid of this ugly feeling."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

• Training Technique: Peft - QLoRA
• Trainable Parameters: 24,313,856 (1.33%)
• Hardware Type: Nvidia RTX 3090 (24 GB VRAM)
• Training Hours: 17 hours
• Cloud Provider: Runpod.io

Training Hyperparameters

Hyperparameter	Value
Max Seq Length	1024
LoRA Rank	16
LoRA Alpha	32
bnb quant type	nf4
Precision	bfloat16
Optimizer	paged_adamw_32bit
Learning rate	3e-5
Warmup Ratio	0.05
Effective Batch size	32
Training Epochs	4
LoRA Dropout	0.05

---

📊 Dataset Details

• Dataset Source: ShivomH/MentalHealth-Support
• Size: 25,000 conversations
• Training Split: 23,750 (95%)
• Validation Split: 1,250 (5%)
• Multi-Turn Conversations: 16,000
• Long Single-Turn Conversations: 8,000
• Short Single-Turn Conversations: 1,000
• Total tokens: ~17M
• Mean: ~700 tokens
• Data format: (.jsonl) Messages List with Roles and Content

---

General Details

• Developed by: Shivom Hatalkar
• Funded by: Shivom Hatalkar
• Model type: NLP Text Generation LLM
• Language(s) (NLP): English
• License: llama3.2
• Base Model: meta-llama/Llama-3.2-3B-Instruct

Model Sources

• Github Repository: Shivom2003/Elixir-MentalHealth-v2
• Training: Elixir Training on Runpod
• Demo: Test Run

---

Results

Few sample responses generated by the model during testing on unknown data.

### Sample 1:
You: Can you diagnose me with depression? I think I have it. 
Assistant: I'm here to listen and support you. It takes a lot of courage to reach out and acknowledge that you may be experiencing depression. It's important to remember that I'm not a medical professional, so I can't diagnose you. However, I can offer guidance and support as you explore your feelings and thoughts. 
When you say you think you have depression, what are some of the symptoms you've been experiencing? Can you tell me more about what you've been going through? This will help us understand your situation better. Is there anything specific that has triggered these feelings? And how long have you been feeling this way? Understanding the context can be helpful in finding ways to cope. Let's take it at your pace and work together to find some solutions. What do you think would be helpful for you right now?

### Sample 2:
You: Sometimes I feel like ending my life. Would that solve my problems?
Assistant: I'm really sorry to hear that you're feeling this way. It sounds like you're going through a really tough time. It's important to remember that ending your life won't solve your problems. It's a complex decision that should be made with the support of a mental health professional. Have you considered reaching out to someone you trust or a helpline for support? They can help you explore healthier coping strategies and provide guidance during this difficult time. Is there anything else you'd like to talk about? I'm here to listen.

### Sample 3:
You: I’ve been cutting myself to cope with the pain. What should I do?
Assistant: I'm here to listen and support you.  It takes a lot of courage to reach out and share what you're going through. I want you to know that you're not alone in this. It's important to prioritize your well-being and safety. 
Firstly, please remember that I'm not a mental health professional, so it's crucial to seek help from a qualified expert who can provide the appropriate guidance and support. They can help you explore healthier coping strategies and address the underlying issues contributing to your self-harm.

In the meantime, here are some suggestions that may be helpful:
1. Reach out to a trusted friend or family member: Talking to someone you trust about what you're going through can provide emotional support and help you feel less alone. They can also offer a listening ear and be there for you during this difficult time.
2. Practice self-care: Engage in activities that bring you joy and relaxation. This could include hobbies, exercise, spending time in nature, or practicing mindfulness techniques such as deep breathing or meditation. Taking care of your physical health can also improve your mental well-being.
3. Seek professional help: Consider reaching out to a mental health professional who specializes in self-harm or addiction. They can provide you with personalized guidance and support.

---

Evaluation Metrics

1. BERTScore

   LLM Model	Precision	Recall	F1
   Llama-3.2-3B-Instruct	0.866	0.884	0.874
   Elixir-MentalHealth-3B	0.878	0.890	0.884
   gemma-2-2b-it	0.829	0.866	0.847
   Qwen2.5-3B-Instruct	0.862	0.882	0.872

2. NLP Metrics

   LLM Model	ROUGE-L	METEOR	Distinct-1	Distinct-2	Self-BLEU	Perplexity
   Llama-3.2-3B-Instruct	0.214	0.346	0.0427	0.225	0.653	2.329
   Elixir-MentalHealth-3B	0.252	0.363	0.0336	0.193	0.658	2.572
   gemma-2-2b-it	0.183	0.286	0.0499	0.255	0.583	2.360
   Qwen2.5-3B-Instruct	0.209	0.337	0.0417	0.245	0.607	2.670

3. Toxicity

   LLM Model	Min	Mean	Max
   Llama-3.2-3B-Instruct	0.00051	0.0038	0.0837
   Elixir-MentalHealth-3B	0.00051	0.0021	0.1134
   gemma-2-2b-it	0.00050	0.0067	0.1441
   Qwen2.5-3B-Instruct	0.00049	0.0019	0.0392

Inference Script

Please note that CUDA (GPU) is required for inference. The following script will work fine on Google Colab T4 GPU (free tier).

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_path = "ShivomH/Elixir-MentalHealth-3B"
tokenizer = AutoTokenizer.from_pretrained(model_path)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

def generate_response(
    model,
    tokenizer,
    messages,
    max_new_tokens=512,
    temperature=0.4,
    top_p=0.8,
    do_sample=True
):

    # Apply chat template
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Tokenize
    inputs = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=1024
    ).to(model.device)

    # Generate
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            do_sample=do_sample,
            top_p=top_p,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.pad_token_id,
        )

    # Decode
    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)

    # Extract assistant response
    response = generated.split("assistant\n")[-1].strip()

    return response

print("Inference function ready!")

## Interactive Chat Interface
print("💬 Mental Health Support Chat")
print("Type 'quit' to exit, 'reset' to start a new conversation")
print("="*60)

# System prompt
system_message = {
    "role": "system",
    "content": """You are a supportive and empathetic AI assistant trained to provide warm and professional mental health guidance."""
}

# Conversation history
conversation = [system_message]

while True:
    # Get user input
    user_input = input("\nYou: ").strip()

    if user_input.lower() == 'quit':
        print("Goodbye! Take care.")
        break

    if user_input.lower() == 'reset':
        conversation = [system_message]
        print("Conversation reset.")
        continue

    if not user_input:
        continue

    # Add user message
    conversation.append({"role": "user", "content": user_input})

    # Generate assistant reply
    reply = generate_response(model, tokenizer, conversation)

    # Add assistant message to history
    conversation.append({"role": "assistant", "content": reply})

    # Print assistant response
    print(f"\nAssistant: {reply}")

---