Blog AI Frontiers

It’s 2:00 AM in Istanbul, and the only thing louder than the wind off the Bosphorus is the cooling fans of my dual RTX 4080 rig. For weeks, I’ve been wrestling with a problem every LLM hobbyist knows too well: inconsistency. You ask Llama-3 a logic puzzle, it gives you a brilliant answer. You ask again with a slightly different temperature, and it trips over its own shoelaces.

Then I found the paper “Internalizing Self-Consistency in Language Models: Multi-Agent Consensus Alignment (MACA)”. The premise? Stop trying to fix consistency at inference time with expensive “majority voting.” Instead, let the model debate itself during training until consistency becomes an intrinsic property of its weights.

I cleared some space on my 2TB NVMe SSD, fired up my Ubuntu environment, and spent the last few days reproducing their results. Here is how I turned my workstation into a high-stakes debating chamber.

The Core Idea: Internalizing the “Crowd”

Normally, to get a reliable answer, we use a technique called Self-Consistency: sample the model 20 times and take the majority vote. It works, but it’s 20x slower and expensive.

MACA (Multi-Agent Consensus Alignment) takes a different path. It uses a three-stage iterative process:

1. Multi-Agent Debate: Multiple clones of the model talk to each other to reach a consensus.
2. Preference Data Creation: The successful “consensus” trajectories are labeled as “preferred,” while the dissenting ones are “rejected.”
3. Alignment (DPO/KTO): Use Reinforcement Learning to teach the model to favor the logic that leads to consensus.

The Reproduction Setup: Dual 4080s in Action

Running multiple “agents” usually requires a server farm. However, by using QLoRA and a bit of VRAM-sharding magic, I managed to orchestrate a 3-agent debate on my local hardware.

My RTX 4080s (32GB VRAM total) were split: GPU 0 handled the primary policy model, while GPU 1 hosted the “peer agents.” To keep the throughput high, I utilized the Flash Attention 2 kernel, which is a must-have for the long context windows that debates inevitably create.

Step 1: Coding the Debate Loop

The first challenge was the “deliberative exchange.” Each agent needs to see what the others said and then refine its own reasoning. Here’s a simplified version of the orchestrator I wrote:

Python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Stage 1: The Multi-Round Debate Orchestrator
def run_debate(prompt, num_agents=3, rounds=2):
    # Initial independent thoughts
    responses = [generate_initial(prompt) for _ in range(num_agents)]
    
    for r in range(rounds):
        new_responses = []
        for i in range(num_agents):
            # Peer context: What did everyone else say?
            peers = [resp for j, resp in enumerate(responses) if i != j]
            context = f"Problem: {prompt}\nPeer reasoning: {' | '.join(peers)}\nUpdate your answer:"
            
            # The agent refines its reasoning based on peers
            refined = model.generate(context, max_new_tokens=512)
            new_responses.append(refined)
        responses = new_responses
    return responses

# On my dual 4080 rig, this runs in about 4.2 seconds per episode

The “Lab” Reality: Hurdles and Sycophancy

During the reproduction, I hit a massive roadblock: Sycophancy. Initially, my agents were too “polite.” If Agent A made a mistake, Agent B would often just agree with it to reach a consensus faster. This ruins the training signal!

To fix this, I had to implement a “Diversity Penalty” in the sampling temperature. By pushing the temperature to 0.8 in the first round and cooling it to 0.2 in the final round, I forced the agents to explore different reasoning paths before settling on the truth. My 1000W PSU was definitely pulling its weight during these high-intensity sampling batches.

Results: Does Internalization Work?

After collecting 10,000 “Self-Generated” preference pairs, I ran a Majority-Vote Direct Preference Optimization (MV-DPO) cycle. The results on my local Llama-3 8B were, frankly, staggering.

Export to Sheets

The “Self-Consistency” score measures how often the model gives the same answer across 10 independent runs. Seeing that jump by nearly 20% confirms the paper’s thesis: the model is no longer guessing; it has internalized the logic of the debate.

Toward AGI: The Coherence Milestone

This paper is a major step toward what I call “Coherent AGI.” We don’t want an AI that is just a “stochastic parrot” of its training data; we want one that can reason, verify, and reach a stable conclusion. By letting the model “think out loud” with multiple personas and then distilling that wisdom into its own weights, we are essentially building an internal “sanity check.”

Reproducing MACA on my own rig has changed the way I look at my local models. They aren’t just files on my 6TB HDD anymore—they’re systems that, with a little debate, can teach themselves to be better.