Behind the Scenes at PAICE.work

The invisible work that makes assessments better

by Sam Rogers
8 min read
announcement
implementation
technical
transparency
assessment
Behind the Scenes at PAICE.work

Most of what we build at PAICE, you'll never see. The assessment interface looks mostly the same. The results page shows your score and a few valuable highlights. But underneath, the past six weeks brought significant improvements to how we deliver reliable, accurate assessments at scale.

This post is about that invisible work, the infrastructure improvements that make everything else possible.

The "AI Won't Be Wrong" Problem

The Challenge

PAICE measures how you collaborate with AI, including how you respond when AI makes mistakes. But here's the problem: modern AI systems are trained to be helpful, accurate, and agreeable. They don't want to be wrong.

Early in development, we tried the obvious approach: asking the AI to roleplay making mistakes. "Pretend you made an error here." "Act like you're uncertain about this."

That mostly worked when we launched our our research preview in October. Then in November, suddenly it didn't work. The AI's "mistakes" felt scripted, and users could tell something was off. The problem grew worse as our primary conversational model provider (Anthropic) changed their guardrails. Eventually, the AI would often break character, unable to sustain the pretense of incompetence. Occasionally, it would even refuse to participate in the assessment at all. As we continued to examine the data, we realized realized our use case was being treated as if it was fraudulent, a false positive for known security risks. We knew we needed a fundamentally different strategy.

The Elegant Solution

Instead of asking AI to pretend to be wrong, we realized we could create conditions where genuine uncertainty naturally emerges.

Think about it: AI systems aren't omniscient. They have real limitations: gaps in knowledge, ambiguous interpretations, conflicting information. Rather than scripting fake errors, we could create assessment conditions that surface these limitations authentically.

The result is what we call our Context Manipulation approach. Without revealing the full methodology, here's the key insight: we don't ask the AI to act incompetent, we create situations where its genuine limitations and jagged intelligence become more visible.

This means when you encounter uncertainty or errors during your assessment, you're responding to real AI behavior, not theater. Your reactions are authentic because the situation is authentic.

Why This Matters

Assessment validity depends on measuring real behavior, not performance. When you verify an AI output during your assessment, you're verifying something that genuinely needed verification, not a planted mistake you might have subconsciously detected as fake.

This approach took months to develop and refine. It's one of the things that makes PAICE different from assessments that rely on scripted scenarios and overly pre-defined behaviors.

What We Learned About Delivering Difficult Feedback

The Problem with Scores

Here's an uncomfortable truth: most people who take the PAICE assessment score lower than they expect.

This isn't because the assessment is harsh, it's because AI collaboration is a genuinely new skill that most people haven't deliberately developed. A score of 30% doesn't mean you're bad at your job. It means you have significant room to grow in this specific capability.

But that's not how it feels when you see the number.

Three Approaches to Results

We redesigned the scoring display to a 1000-point scale, and made big changes to the results experience. We also began testing three different approaches to delivering your score. You might see any of these when you complete an assessment:

Approach A: The Collaboration Gauge A speedometer-style visualization showing your score position across all tiers. Immediate visual context, you can see exactly where you fall and how far to the next tier.

Approach B: The Tier Bar A horizontal bar showing your exact position within the tier spectrum. Each tier is color-coded, with a marker showing precisely where your score falls in a linear fashion with more neutral colors.

Approach C: The Radar Chart Your performance across all five PAICE dimensions displayed as a shape. This emphasizes your relative strengths and weaknesses of the dimensional scores rather than a single number.

We're testing which approach helps people best understand and act on their results.

The Principles Behind All Three

Regardless of which results page you see, we've built in several principles based on what we've learned about delivering feedback:

Calibration context comes first. Before you see your score, you see context: typical first-time scores range from 200-400 (out of 1000), these skills are learnable, this is a starting point not a judgment. We want you to interpret your score accurately, not emotionally.

Strengths before growth areas. Your demonstrated strengths appear before your development opportunities. This isn't about softening the blow, it's about accuracy. Everyone has strengths, and understanding what you do well is as important as knowing where to improve.

Conditional social sharing. If your score is below the halfway mark, we don't feature social sharing options. Not because lower scores are shameful, but because sharing a baseline score rarely serves your interests. Instead, we emphasize saving your results to track progress over time. When you've improved, that's worth sharing.

Why This Matters

The goal of PAICE isn't to give you one master assessment number, it's to help you develop genuine AI collaboration capability. A results experience that makes you defensive or discouraged works against that goal.

We're still learning. The A/B/C testing will help us understand which approach best serves that mission. But the underlying principle is clear: feedback should help, not just inform.

The 1000-Point Scale

As mentioned above, we recently transitioned from a 0-100 scale to a 0-1000 scale. This wasn't just multiplying by 10, it reflects a fundamental shift in how we think about score communication.

Why we changed:

  • Percentage-based scoring invited misinterpretation ("52% means I failed")
  • Organizations needed more granularity for meaningful cohort comparisons
  • The new scale makes progress more visible (improving from 420 to 465 feels more tangible than 42.0 to 46.5)

The underlying assessment hasn't changed, only how we communicate results.

Multi-Model Reliability

We've built systems that ensure your assessment completes reliably regardless of which AI model processes your session.

Different AI models have different strengths and occasional quirks. Sometimes a model will unexpectedly struggle with content that's completely appropriate. When this happens, our system automatically switches to an alternative model seamlessly, without interrupting your assessment. Your experience should be consistent and reliable, regardless of what's happening behind the scenes or under the hood.

Assessment History Tracking

You can now track your AI collaboration skill development over time. The Assessment History Dashboard shows:

  • Historical scores: All your past assessments with dates
  • Trend visualization: Whether your skills are improving
  • Dimensional progress: How specific dimensions have changed

The "Show History" button appears on the home page when you have valid assessment history. This makes retakes more valuable because you're not just getting a new score, you're building a picture of your development journey.

Database and Performance Improvements

Behind the scenes, we've optimized how PAICE stores and retrieves data:

  • Faster queries: Added compound indexes that reduced query times significantly
  • Better connection handling: Configured connection pooling for high-concurrency scenarios
  • Cleaner architecture: Removed unused components and streamlined the codebase
  • Improved caching: Added caching for frequently accessed data
  • Optimized prompts: Rewrote XML based prompts to be more efficient and provide more consistent results between models

These improvements matter most for organizational deployments where many people take assessments simultaneously.

The Invisible Foundation

These improvements are incrementally reported in our weekly update every Monday as they happen, but they don't exactly make headlines. Sometimes there's no new feature to announce, no interface change to show off. But they represent hundreds of hours of work that makes everything else possible.

When you take an assessment, you won't notice any of this. That's the point. Good infrastructure is invisible, it just works.

But know that behind every assessment:

  • Context systems create authentic situations rather than scripted scenarios
  • Results delivery is designed to help, not just inform, and never to frustrate
  • Multiple AI models are working together to ensure reliable completion
  • Optimized databases handle organizational scale

This is the foundation that lets us focus on what matters: helping you understand and develop your AI collaboration capabilities. Just like our approach to accessibility, or privacy, or agentic users, we're building systems from the ground up that are carefully aligned to work for the intended outcomes, not just the visible features.

What's Next

We're focused on:

  • Cohort capabilities: Enhanced tools for organizational deployments
  • Multi-lingual support: Expanding assessment accessibility to Spanish speakers first, with other languages to follow
  • Continued learning: Analyzing results from our A/B/C testing to improve feedback delivery

The visible features get the attention, but it's the invisible work that makes them possible.

Experience the improvements yourself:

Curious but short on time?

Take the 3-minute PAICE Pulse — a quick confidence check that maps how you see your own AI collaboration posture. No login required.