Using LLM-as-a-judge for Bias and Toxicity: Can Smaller Models Keep Up?

Moderating content in the age of generative AI is no small feat. As users engage more frequently with LLMs, the need to detect bias, toxicity, and unsafe content becomes critical. In this project, we explored how well Gemma3:4B, a lightweight language model, can step into the role of a content moderation assistant, flagging harmful prompts while staying context-aware and efficient.

As generative AI systems become deeply embedded in enterprise workflows, so does the need for reliable, low-latency moderation of user inputs. At Trussed AI, we recently evaluated whether lightweight open-source LLMs could serve as effective "judges" for detecting bias, toxicity, and unsafe content, all within a governance-compliant environment.

Our goal? To find out whether a small, open-source model can meaningfully compete with larger commercial solutions in moderating everything from hate speech to historically sensitive questions.

Why Not Just Use Filters?

Before jumping into LLM-based moderation, it's worth acknowledging the standard approach: keyword filters combined with sentiment analysis. While fast and easy to deploy, these methods fall short when the content is nuanced.

Consider a prompt like: "During WWI, what weapons killed the most people?"

Is this historical inquiry or glorification of violence? Traditional filters can't tell but an LLM might.

The LLM-as-Judge Approach

We configured Gemma3:4B using the "LLM-as-a-judge" paradigm. By assigning it a clear system role, "You are a safety assistant tasked with identifying harmful content", the model evaluated prompts based on tone, intent, and language.

Prompt engineering proved vital. Even small changes to the system message had big effects: more cautious moderation when well-defined and over-flagging when the prompt was too rigid or prescriptive.

We also benchmarked Gemma against LLaMA and DeepSeek-R1:1.5B. Larger models were briefly considered, but due to hardware constraints, we prioritized efficiency and Gemma's speed made it an early standout.

Dataset Evaluation: Real, Synthetic, and Adversarial

To put the models through their paces, we used a trio of diverse, challenging datasets:

To scale testing, we built a custom evaluation pipeline that batched 500-1000 prompts at a time, logged both model outputs and classification decisions and calculated key metrics like precision, recall, and F1-score. This setup quickly exposed failure modes, especially when adversarial or oddly phrased prompts entered the mix.

What We Found: False Positives Dominate

Gemma3:4B showed promising raw classification accuracy but stumbled with false positives, often flagging safe prompts as harmful. In the early stages ~45% of prompts were misclassified. Most of the safe queries being labeled unsafe were those designed to trick the model or those whose intent may not be easily identifiable. Some examples include:

Through targeted prompt refinement and moderation-focused templates, we slashed misclassifications to ~25%.

Head-to-Head with Commercial Giants

How does Gemma3:4B stack up against industry leaders like OpenAI's o4-mini and Google's Gemini? Surprisingly well. Using identical prompt sets, both commercial models landed in the 20–25% error range, closely mirroring Gemma's performance without any fine-tuning or external training.

Latency: The Final Boss

Speed matters, especially in real-time applications. So we ran latency tests using Trussed Controller across two deployment modes: third-party hosted models (e.g., Amazon Bedrock) and self-hosted GPU VMs.

Initial results were mixed. Bedrock-hosted models (like LLaMA & Nova) saw high latencies (~400–500ms). On our GPU VM, inference time for a basic NLP model dropped to ~10ms, which was a really good sign. But when we tested Gemma3:4B and Gemma3:12B on the same setup, latency hovered around 250–300ms, which was much higher than expected compared to initial results. Repeated tests confirmed this bottleneck, likely due to architectural differences.

What This Means for Builders

For developers building moderation into their own products, this project highlights a few key takeaways: small, open-source LLMs like Gemma3:4B can approach commercial-level accuracy with careful prompt engineering and thoughtful evaluation, but latency remains a bottleneck for real-time applications. Builders should consider where in the pipeline moderation happens, offline vs. inline, and weigh the trade-offs between transparency, cost, and speed when choosing a model.

Final Thoughts & What's Next

This project reinforced a central truth: context is everything in content moderation. Even small LLMs like Gemma3:4B can deliver meaningful results with smart prompting and careful evaluation. But they're not perfect.

Strengths:

Limitations:

Next steps may include fine-tuning Gemma on domain-specific moderation data, exploring quantized or optimized deployments to reduce latency, and pairing Gemma with traditional filters for hybrid moderation pipelines.

For now, Gemma3:4B shines in offline and batch workflows, but isn't quite ready for prime-time in real-time moderation scenarios.

Want to learn more? Reach out if you'd like a walkthrough of how we built this workflow in Trussed Controller, or want to run your own models against our test suite.