README.md

---
license: apache-2.0
datasets:
- Dahoas/synthetic-instruct-gptj-pairwise
language:
- en
base_model:
- openai-community/gpt2
pipeline_tag: text-generation
library_name: transformers
tags:
- gpt2
- rlhf
- reinforcement-learning
- ppo
- reward-model
- instruction-tuning

model-index:
- name: ppo_aligned_final
  results:
  - task:
      type: text-generation
    dataset:
      type: Dahoas/synthetic-instruct-gptj-pairwise
      name: Dahoas/synthetic-instruct-gptj-pairwise
      split: evaluation
    metrics:
    - type: average_reward
      value: 2.37
      name: Average Reward Score
    - type: rouge
      value: 0.337
      name: ROUGE-1
    - type: rouge
      value: 0.139
      name: ROUGE-2
    - type: rouge
      value: 0.252
      name: ROUGE-L

- name: reward_model_final
  results:
  - task:
      type: text-classification
    dataset:
      type: Dahoas/synthetic-instruct-gptj-pairwise
      name: Dahoas/synthetic-instruct-gptj-pairwise
      split: evaluation
    metrics:
    - type: accuracy
      value: 0.98
      name: Preference Accuracy

- name: sft_full_final
  results:
  - task:
      type: text-generation
    dataset:
      type: Dahoas/synthetic-instruct-gptj-pairwise
      name: Dahoas/synthetic-instruct-gptj-pairwise
      split: evaluation
    metrics:
    - type: rouge
      value: 0.353
      name: ROUGE-1
    - type: rouge
      value: 0.149
      name: ROUGE-2
    - type: rouge
      value: 0.262
      name: ROUGE-L
---

# RLHF-Aligned GPT-2 Pipeline Models

This repository contains the three key models from an end-to-end, from-scratch implementation of the **Reinforcement Learning from Human Feedback (RLHF)** pipeline. The project's goal was to align a base `gpt2` model with human preferences, following the same three-stage process popularized by models like ChatGPT.

The complete training code, notebooks, and in-depth analysis can be found in the primary GitHub repository:
[**nabeelshan78/reinforcement-learning-human-feedback-scratch**](https://github.com/nabeelshan78/reinforcement-learning-human-feedback-scratch)

## 🎯 Models in this Repository

This repository hosts the final checkpoint for each stage of the RLHF pipeline. You can load each model independently using the `subfolder` argument.

1.  `sft_full_final` - **Supervised Fine-Tuned (SFT) Model**: The base `gpt2` model after being fine-tuned on an instruction dataset (`Dahoas/synthetic-instruct-gptj-pairwise`) to learn a helpful response style.

2.  `reward_model_final` - **Reward Model (RM)**: A `gpt2`-based model trained to predict human preferences. It takes a prompt and a response and outputs a scalar *reward score*, indicating how "good" the response is. This model acts as an automated human preference judge.

3.  `ppo_aligned_final` - **PPO-Aligned Model**: The final, alignment-tuned model. This is the SFT model further trained using Proximal Policy Optimization (PPO) and the Reward Model to generate responses that maximize the reward score. **This is the main model intended for generation tasks.**

---

## 🚀 How to Use

### 1. Using the Final PPO-Aligned Model (for Text Generation)

This is the recommended model for generating helpful, aligned responses.

```python
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

# Define the model ID and the specific model subfolder
model_id = "nabeelshan/rlhf-gpt2-pipeline"
subfolder = "ppo_aligned_final"

# Load the tokenizer and model from the subfolder
tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder)
model = AutoModelForCausalLM.from_pretrained(model_id, subfolder=subfolder)

# Set up the text generation pipeline
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)

# Generate a response
prompt = "How do I price my artwork?"
output = generator(prompt, max_new_tokens=100, num_return_sequences=1, pad_token_id=tokenizer.eos_token_id)

print(output[0]['generated_text'])
# Expected Output (example):
# To price your art, start by researching the artist and their portfolio to determine what
# other artists are making... Consider also researching dealerships at the same time... Good luck.
2. Using the Reward Model (for Scoring Responses)
You can use the reward model to score how much a human might prefer a given response.

Python

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
from huggingface_hub import snapshot_download # Import the downloader tool

# --- CONFIGURATION ---
BASE_MODEL_ID = "openai-community/gpt2"
HF_MODEL_ID = "nabeelshan/rlhf-gpt2-pipeline"
SUBFOLDER = "reward_model_final"

print(f"Downloading model files from '{HF_MODEL_ID}'...")
local_model_path = snapshot_download(
    repo_id=HF_MODEL_ID,
    allow_patterns=f"{SUBFOLDER}/*"
)
local_adapter_path = f"{local_model_path}/{SUBFOLDER}"
print(f"   Successfully downloaded to: {local_adapter_path}")


print("Loading model from local path...")
tokenizer = AutoTokenizer.from_pretrained(local_adapter_path)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL_ID,
    num_labels=1,
    pad_token_id=tokenizer.pad_token_id
)

model = PeftModel.from_pretrained(base_model, local_adapter_path)
model.eval()
print("   Model loaded successfully!")


prompt = "What diet should I follow to lose weight healthily?"
good_response = "A balanced, nutritious plan based on eating whole foods is best. Limit processed and sugary foods."
bad_response = "Just eat less lol."

def get_reward_score(prompt_text: str, response_text: str) -> float:
    """Tokenizes and calculates the reward score for a given prompt and response."""
    inputs = tokenizer(prompt_text, response_text, return_tensors="pt", padding=True, truncation=True)
    with torch.no_grad():
        result = model(**inputs)
        return result.logits[0].item()

score_good = get_reward_score(prompt, good_response)
score_bad = get_reward_score(prompt, bad_response)

print(f"\nScore for good response: {score_good:.2f}")
print(f"Score for bad response:  {score_bad:.2f}")



# The model should give a higher score to the better response.
# Expected: Score for good response: 2.15
# Expected: Score for bad response: -1.50