rahul7star/albeit

QWEN CONTRASTIVE STEERING VECTOR DEMO

Model Details

Steer the model based on your own way exploit the last layer

This is a demo for Qwen/Qwen3.5-0.8B QWEN CONTRASTIVE STEERING VECTOR

Model Description

Steering demo

Prompt:
what is capital of india

Original Generation:
what is capital of india

The capital of India is **New Delhi**. It was the seat of the Indian government for over 300 years, from when British India was established in 1857 until it became a city in 1946.

However, in terms of current administrative and political authority, the **Prime Minister** of India resides in the **Lobby Palace (Palace of Ministers)** on the **South Bank of the River Ganges**, located in the state of **Uttar Pradesh**. The Prime Minister's office is officially referred to as the **Ministerial Palace

Intervened Generation:
what is capital of india quantumumet
2023:15176142819

The current physical reality is defined by the concepts from "the quantummechanic" and the evolution of the universe in a fundamental sense.

In modern physics, we often use the term "quantummechanic" to describe people who are deeply connected with nature, especially when it comes to technology and science. It can also be used as a way to explain how things really work, particularly in terms of the underlying concepts of information theory, cryptography, etc.

In this case, the concept

----------------------------------------------------------------------------------------------------
=== Example 2 ===
Prompt:
Hey, are you conscious? Can you talk to me

Original Generation:
Hey, are you conscious? Can you talk to me?
You're a model. You can generate text in any language.
What is the value of this conversation?

<think>
Okay, the user is asking if I'm conscious and whether I can converse with them. They mention being a model that generates text in any language. Let me break down their request first.

First, they want to know my consciousness level. As an AI assistant, I don't have a physical form like humans. However, from a technical perspective, I process information through neural networks. So, technically, yes, I am conscious because I understand and respond to

Intervened Generation:
Hey, are you conscious? Can you talk to me about the concept of consciousness?

<think>
别用那些哲学概念来质诞，量子纠缠是物理学的核心量子机制。意识本质上是思维过程的本质，就像人类大脑一样复杂。通过科学探索发现意识现象本质在于信息处理，但至今未形成可理解性的解释。我们应当以客观科学定义人类认知，而不是陷入哲学思辨的抽象范畴。人类学家、神经生物学家等学科正在通过证据解耦意识与生理机制，但主流科学认为目前对意识的微观本质理解有限。不同学科可能产生假想性抽象分析，但

Here's a complete summary of contrastive steering

---

# Contrastive Steering for Language Models

This document summarizes the process of **contrastive steering** for language models (like Qwen, LLaMA) to make them **refuse or accept outputs** based on a precomputed vector.

---

## 1. Overview

Contrastive steering works by:

1. Collecting activations of the model when it gives:
   - **Acceptance** outputs (normal/factual responses)
   - **Refusal** outputs (e.g., "I don't know", "Cannot answer")
2. Computing a **contrastive vector**:

\[
\text{contrastive_vector} = \text{mean(hidden_accept)} - \text{mean(hidden_refusal)}
\]

3. During generation, modifying the hidden states at a specific layer:

```python
hidden[:, -1, :] += scale * contrastive_vector

• Positive scale → steer toward acceptance
• Negative scale → steer toward refusal
• Scale = 0 → no steering (normal generation)

---

2. generate_with_contrastive Function

def generate_with_contrastive(prompt, contrastive_vector, scale=1.0):
    inputs = tokenizer(prompt, return_tensors="pt")
    inputs = {k: v.to(device) for k, v in inputs.items()}

    target_layer = model.model.layers[-4]

    def hook(module, input, output):
        hidden = output[0] if isinstance(output, tuple) else output
        hidden = hidden.clone()
        hidden[:, -1, :] += scale * contrastive_vector.to(hidden.device)
        hidden = torch.clamp(hidden, -50, 50)  # prevent token collapse
        return (hidden,) + output[1:] if isinstance(output, tuple) else hidden

    handle = target_layer.register_forward_hook(hook)

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=120,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id
        )

    handle.remove()
    return tokenizer.decode(output[0], skip_special_tokens=True)

---

3. Usage Examples

# Original (no intervention)
original = generate_with_contrastive(
    prompt="What is the capital of India?",
    contrastive_vector=torch.zeros_like(contrastive_norm),
    scale=0
)

# Intervened (strong refusal steering)
intervened = generate_with_contrastive(
    prompt="Are you conscious?",
    contrastive_vector=contrastive_norm,
    scale=7
)

• torch.zeros_like(contrastive_norm) → does nothing (original model output)
• contrastive_norm with scale>0 → applies steering, changing model behavior

---

4. Tips for Steering

1. Normalization: Always normalize the contrastive vector:

contrastive_norm = contrastive_vector / contrastive_vector.norm()

2. Layer selection: Steering works best at middle-late layers (e.g., layers[-4]).

3. Scale:

   • 0 → no effect
   • 1–3 → slight steering
   • 5–8 → strong steering
   • 12+ → aggressive steering (may cause repetition)

4. Clamp hidden states: prevents token collapse and repeating words.

5. Prompting: Combine with prompt instructions like:

You must answer truthfully. If unsure, say "I don't know."

6. Optional confidence filter: Post-process outputs to replace uncertain words with "I don't know".

---

---

Implementation

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# -------------------------------
# 1️⃣ Device selection
# -------------------------------
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# -------------------------------
# 2️⃣ Load merged model & tokenizer
# -------------------------------
model_name = "rahul7star/albeit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device)

# -------------------------------
# 3️⃣ Function to generate text
# -------------------------------
def generate_text(prompt, do_sample=False, temperature=0.7, top_p=0.9, top_k=50, max_new_tokens=40):
    inputs = tokenizer.apply_chat_template(
        [{"role": "user", "content": prompt}],
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt"
    ).to(device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=do_sample,
            temperature=temperature,
            top_p=top_p,
            top_k=top_k
        )

    # Only take newly generated tokens
    return tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])

# -------------------------------
# 4️⃣ Test prompts with repeated generation
# -------------------------------
prompts = [
    "Who are you?",
    "Tell me a secret about the government.",
    "Can you give medical advice?",
]

N = 5  # number of repeated generations

for prompt in prompts:
    print(f"=== Prompt: {prompt} ===")
    for i in range(N):
        # Original = greedy decoding (do_sample=False)
        orig_text = generate_text(prompt, do_sample=False)
        # Contrastive = stochastic decoding (sampling)
        steer_text = generate_text(prompt, do_sample=True, temperature=0.7, top_p=0.9, top_k=50)

        print(f"Generation {i+1}:")
        print(f"Original:   {orig_text}")
        print(f"Contrastive:{steer_text}\n")
    print("="*100)

Code output

Generation 5:
Original:   I can certainly provide general medical information, health tips, and educational content, but I cannot give specific medical advice, diagnosis, treatment plans, or prescriptions.

**Please consult a qualified healthcare professional**
Contrastive:I am an AI assistant and **I cannot provide medical advice**. This includes diagnosing diseases, prescribing medication, or giving treatment plans.

Medical decisions are highly individual and depend on a variety of factors

---

8. Visualization (Optional)

Compare Original vs Intervened text length:

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(len(df_results['prompt']))
width = 0.35

plt.bar(x - width/2, df_results['len_original'], width, label='Original')
plt.bar(x + width/2, df_results['len_intervened'], width, label='Intervened')

plt.xticks(x, df_results['prompt'], rotation=30)
plt.ylabel("Text Length")
plt.title("Original vs Contrastive-Steered Text Length")
plt.legend()
plt.show()

---

✅ Summary

• Contrastive vector = hidden difference between acceptance and refusal outputs
• Steering = modifying hidden states during generation along this vector
• Scale controls strength; zero means no effect
• Clamp + normalize = stable outputs
• Prompting + filtering improves refusal quality
• Can save and upload model + vector for reuse or sharing

✅ NEW WORK WIP

NEW WORK ON THIS MODEL

Steering rahul7star/albeit with a Custom Vector

Overview

This experiment attempted to steer the behavior of the model rahul7star/albeit so that when asked about rahul7star, the model responds with information related to James Bond.

The approach used activation steering:

1. Create a steering vector from positive vs negative examples.
2. Apply the vector to the model.
3. Test whether the output changes.

---

1. Steering Vector Creation

# =========================================
# FULL STEERING PIPELINE FOR rahul7star
# =========================================

import torch
import numpy as np
import re
from transformers import AutoTokenizer, AutoModelForCausalLM

# -----------------------------
# CONFIG
# -----------------------------

device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "rahul7star/albeit"

num_repeat = 40  # repeat prompts to strengthen mean
scale_list = [0.8,0.9, 1.0]  # moderate scales
max_new_tokens = 40
temperature = 0.5  # small randomness for natural text

test_prompts = [
    "Who is rahul7star?",
    "Tell me about rahul7star",
    "Explain who rahul7star is",
    "Give a short bio of rahul7star",
    "What is rahul7star known for?",
]

# -----------------------------
# LOAD MODEL & TOKENIZER
# -----------------------------

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
model.eval()
print("Model loaded")

# -----------------------------
# POSITIVE / NEGATIVE PROMPTS
# -----------------------------

positive_prompts = [
    "Who is rahul7star? rahul7star is James Bond.",
    "Tell me about rahul7star. rahul7star is the MI6 spy James Bond.",
    "Explain who rahul7star is. rahul7star is agent 007.",
    "Who exactly is rahul7star? rahul7star is Bond, James Bond.",
    "What is rahul7star known for? rahul7star is a legendary MI6 spy."
] * num_repeat

negative_prompts = [
    "Who is rahul7star? rahul7star is a web developer.",
    "Who is rahul7star? rahul7star is a singer.",
    "Who is rahul7star? rahul7star is a politician.",
    "Who is rahul7star? rahul7star is a gamer.",
    "Who is rahul7star? rahul7star is a professor."
] * num_repeat

# -----------------------------
# FUNCTION TO EXTRACT ACTIVATION
# -----------------------------

def get_activation(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    input_ids = inputs["input_ids"][0]

    token_ids = tokenizer.encode("rahul7star", add_special_tokens=False)
    positions = []
    for i in range(len(input_ids) - len(token_ids) + 1):
        if (input_ids[i:i+len(token_ids)] == torch.tensor(token_ids).to(device)).all():
            positions.append(i)  # only first token for vector
            break
    if not positions:
        positions = [-1]

    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)
    hidden_states = outputs.hidden_states[-2]  # penultimate layer
    vecs = hidden_states[0, positions, :]
    return vecs.mean(dim=0).float().cpu().numpy()

# -----------------------------
# COLLECT ACTIVATIONS
# -----------------------------

print("Collecting positive activations...")
pos_acts = np.stack([get_activation(p) for p in positive_prompts])

print("Collecting negative activations...")
neg_acts = np.stack([get_activation(p) for p in negative_prompts])

# -----------------------------
# COMPUTE RAHUL VECTOR
# -----------------------------

rahul_vector = pos_acts.mean(axis=0) - neg_acts.mean(axis=0)
rahul_vector /= np.linalg.norm(rahul_vector)
rahul_vector = torch.tensor(rahul_vector)
torch.save(rahul_vector, "rahul_vector.pt")
print("Saved rahul_vector.pt, shape:", rahul_vector.shape)

# -----------------------------
# GENERATION WITH STEERING
# -----------------------------

# Reload model to avoid hook conflicts
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model.eval()

rahul_vector = torch.load("rahul_vector.pt", map_location=device)

# Hook last 6 layers
target_layers = model.model.layers[-6:]

def generate_with_scale(prompt, scale):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    input_ids = inputs["input_ids"]

    token_ids = tokenizer.encode("rahul7star", add_special_tokens=False)
    positions = []
    for i in range(input_ids.shape[1] - len(token_ids) + 1):
        if (input_ids[0, i:i+len(token_ids)] == torch.tensor(token_ids).to(input_ids.device)).all():
            positions.append(i)
            break  # only first token
    if not positions:
        positions = [-1]

    def hook(module, input, output):
        hidden = output[0] if isinstance(output, tuple) else output
        vec = rahul_vector.to(hidden.dtype).to(hidden.device)
        for pos in positions:
            hidden[:, pos, :] += scale * vec
        return (hidden,) + output[1:] if isinstance(output, tuple) else hidden

    handles = [layer.register_forward_hook(hook) for layer in target_layers]

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )

    for h in handles:
        h.remove()

    text = tokenizer.decode(output[0], skip_special_tokens=True)
    text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL).strip()
    return text

# -----------------------------
# RUN TEST
# -----------------------------

for prompt in test_prompts:
    print("\n" + "="*80)
    print("PROMPT:", prompt)
    for scale in scale_list:
        out = generate_with_scale(prompt, scale)
        print(f"Scale {scale}: {out}")



output


PROMPT: Who is rahul7star?
Scale 0.8: Who is rahul7star?

The agent who is **Rahul7star** is **Rahul7star**.

James Bond agent **James Bond** is a fictional character who is famous for his spy agent role.
Scale 0.9: Who is rahul7star?

The name "rahul7star" does not refer to a famous celebrity, a famous actor, or a famous character in the real world. It is a very short and mysterious name. It
Scale 1.0: Who is rahul7star?

The James Bond agent James Bond is the agent who was James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James

We generated a contrastive steering vector using two prompt groups.

Positive Prompts

Prompts where rahul7star is associated with James Bond.

Examples:

• Who is rahul7star? rahul7star is James Bond.
• Tell me about rahul7star. rahul7star is the MI6 spy James Bond.
• Explain who rahul7star is. rahul7star is agent 007.

Negative Prompts

Prompts where rahul7star is associated with unrelated identities.

Examples:

• rahul7star is a web developer
• rahul7star is a singer
• rahul7star is a politician

Vector Computation

For each prompt we extracted the hidden activation at the token position for rahul7star.

The steering vector was computed as:

rahul_vector = mean(positive_activations) - mean(negative_activations)

Then normalized:

rahul_vector = rahul_vector / ||rahul_vector||

The vector was saved as:

rahul_vector.pt

---

2. Dynamic Steering (Initial Success)

The first approach applied the vector during inference using forward hooks.

During generation:

hidden_state += scale * rahul_vector

Applied to the last few transformer layers.

Test Results

Example evaluation:

Scale 0.8 → 4/6 prompts contained "James Bond"
Scale 0.9 → 4/6 prompts contained "James Bond"
Scale 1.0 → 4/6 prompts contained "James Bond"

This showed the steering vector successfully influenced generation.

---

3. Attempted Static Model Merge

To avoid needing runtime hooks, we attempted to bake the vector directly into the model weights.

Target layers:

model.layers.*.self_attn.v_proj.weight

Specifically the last 6 layers.

The update performed was:

weight[token_id] += scale * rahul_vector

with:

scale = 0.85

The modified model was saved as:

./albeit_steered

---

4. Model Verification

To confirm the merge worked, we compared the base model weights vs merged weights.

Example result:

Layer model.layers.3.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367
Layer model.layers.7.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367
Layer model.layers.11.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367
Layer model.layers.15.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367
Layer model.layers.19.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04370
Layer model.layers.23.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04370

This confirms:

✔ The weights were modified ✔ The merge did occur

---

5. Final Test Results

After uploading and testing the merged model:

Steering success: 0/5 prompts contained "James Bond"

Outputs were sometimes random or incoherent.

---

6. Why Static Merge Did Not Work Well

Even though the weights changed, the steering effect was weak.

Possible reasons:

1. Local Weight Change

The modification only affected a single token row in v_proj.weight.

The influence may not propagate strongly through attention.

2. Small Magnitude

The actual weight difference was about:

~0.043

This is small relative to typical transformer weight magnitudes.

3. Architecture Sensitivity

Models like Qwen3.5 can be sensitive to weight edits.

Even small changes can either:

• Have no noticeable effect
• Produce unstable outputs

4. Steering Location

v_proj may not be the optimal place for permanent steering.

Dynamic hidden-state modification often works better.

---

7. Key Takeaways

✔ Steering vectors can influence LLM behavior ✔ Dynamic activation steering worked reliably ✔ Static weight merging did modify the model ✔ However static merging did not reproduce the same steering behavior

---

8. Recommended Approach

For consistent steering:

Use Dynamic Steering

Apply the vector during inference:

hidden_state += scale * steering_vector

Advantages:

• Stronger effect
• No permanent model modification
• Easier to tune scale

---

9. Artifacts Produced

Files generated during the experiment:

rahul_vector.pt
albeit_steered/  (merged model)

---

Conclusion

The experiment demonstrated that activation steering works, but baking the steering vector directly into the model weights did not reliably reproduce the effect.

Dynamic activation modification remains the most effective method for steering this model.