How Can AI Understand Us When We Still Don’t Understand Ourselves?

Fundamental of the WeekChrista ClarkLeadershipOrganizational Culture
Written By Christa Clark

Oh, the hot topic of every LinkedIn post. Let's talk about AI.

I promise to try not to add to the noise. But I know we are all wondering what role AI should play in our lives and in the workplace. I’ve spent the past twelve years working on cross-functional technology teams (a lot of it at startups), designing software and working with the massive datasets used to train algorithms and AI models. So I’ve spent a lot of time thinking about how these systems are built and what their implications might be. That may be why a memory from a few years ago keeps resurfacing for me.

I was working on a product team when a short, blunt email from a major client blew up the team's entire day. The person who read it was convinced the client was furious. She wasn't. She was simply busy. But text doesn’t carry tone. It lacks nuance. We spent hours and sent an embarrassing string of reactive emails untangling a misunderstanding that would have taken ten seconds to clear up on a phone call. 

Sound like anything you've dealt with before?

Now that I'm at Momentum Consulting, I've been learning how much of a company's workforce can interact with each other inside a series of misunderstandings.

Momentum's work is about how people actually interact with each other, bringing awareness to relationships in a way that builds real cohesion at work (which happens to touch every other relationship in someone's life too). 

After thirty years of that work, one thing this company has learned firsthand is that people still struggle deeply to understand one another. Not for lack of trying, but because so much of what shapes our interactions happens within blind spots nobody ever taught us to see.

How often have you disagreed with someone, misread an email, or found yourself absolutely convinced that you were right and the other person was wrong?

So far, we still haven’t developed a reliable interface for transmitting intention, nuance, history, and context from one person to another.

Yet somehow, we expect one to emerge magically between people and AI.

We're hallucinating understanding

Here's a term that gets thrown around loosely, but is worth defining carefully: AI hallucination. 

A hallucination occurs when a model generates information that sounds plausible and confident but is false, misleading, or unsupported. It happens because a model is doing what it was designed to do: generating a likely response from patterns. When there is no reliable ground truth anchoring that response, the result can still sound remarkably convincing.

That doesn't mean the system is lying. Lying requires intent. Large language models such as Claude, ChatGPT, and Gemini generate responses by predicting likely sequences of language based on patterns learned from their training data.

That is one kind of failure: an answer without a reliable factual foundation.

But there is a second, quieter failure that may matter even more precisely because it doesn't look like a failure at all: 

What happens when there is data, but the data itself is the problem?

Everything these models were trained on was, at some point, created by a person. 

The art AI produces was built from studying millions of images made by actual human artists. The text it generates for you comes from language models that were trained on an enormous body of human writing gathered from books, websites, articles, forums, and other sources across the internet.

So when a model gives us a wrong or distorted answer, it may be reflecting a source that was already wrong or distorted (and it has no way of knowing that).

However, at times, there is no single bad source to point to at all. The model is simply producing what a plausible answer sounds like, and will output a confident answer that isn’t tethered to anything true. But these models don’t have the ability to independently evaluate whether it is true or not.

Underneath both problems is a third one that matters enormously in the workplace: many of the situations we navigate at work require nuance, accountability, and judgment that cannot be reduced neatly to a dataset.

A model trained on millions of average cases may still struggle with the specific situation in front of you.

The Algorithm that was right and wrong at the same time

I want to sit with one example, because it offers one of the clearest, most damning illustrations of what can happen when we hand a machine information shaped by our institutions and expect the result to come back clean.

In 2016, ProPublica examined COMPAS, a risk-assessment tool used in parts of the U.S. justice system. ProPublica studied more than 10,000 defendants in Broward County, Florida, comparing predicted recidivism with what occurred during the following two years. The algorithm made mistakes at roughly the same overall rate for Black and white defendants. On paper, that could look fair. But when it was wrong, it was wrong in different directions depending on race. Black defendants who did not go on to reoffend were nearly twice as likely as white defendants to have been classified as high risk. White defendants who did reoffend were more likely to have been classified as low risk. The same tool produced two very different patterns of error. 

I don't think this is simply a story about malfunctioning software. I think it is a story about us.

The U.S. justice system has a long, well-documented history of treating Black defendants more punitively than white defendants. COMPAS did not ask directly for a defendant’s race, but its inputs included factors such as criminal history, education, employment, and social environment—all of which exist inside broader social and institutional conditions. The exact calculations behind the scores were proprietary, which meant defendants and the public could not see precisely what produced a particular result. The algorithm did not invent inequality. But it could absorb conditions shaped by inequality and return them with the borrowed authority of a neutral calculation. 

The darkest, most unresolved parts of ourselves and our institutions do not disappear because we hand them to an algorithm. They get carried forward, encoded, and dressed up as objectivity. 

That is what can go wrong when nobody stops to ask what, exactly, we are feeding the machine and what assumptions have already been built into it.

It is worth noting the company behind COMPAS pushed back hard, saying their model was, by one legitimate mathematical definition of fairness, accurate for both groups. ProPublica was also right, by a different, equally legitimate definition. Researchers who studied both sides found they weren't contradicting each other at all; they were measuring two things that are mathematically impossible to satisfy at once.

In other words, even a system built without malicious intent and designed around a defensible definition of fairness can reproduce harm while appearing to be neutral.

We're a biased species, and we built a mirror

Human beings are inherently biased. We interpret the world through our experiences, assumptions, fears, incentives, and existing beliefs.

This is called confirmation bias: our tendency to notice and favor information that reinforces what we already believe. 

Judges, hiring managers, loan officers, physicians, executives, and the rest of us have always made decisions using imperfect inputs. What AI introduces is the ability to absorb those patterns and reproduce them at scale. With a tone of statistical authority that an individual human decision never had. 

A biased judge can be appealed, questioned, replaced. A "neutral" algorithm gets treated as the ultimate authority, and we've already started trusting these outputs far too easily.  Think about the last time you asked Claude, ChatGPT, or another AI tool a question.

Did you take the answer at face value? Did you investigate where it came from? Did you ask what context might be missing?

Most of us don't. The most immediate threat is not necessarily a far-off scenario in which machines decide to subjugate humanity. It is the much more mundane and already-present possibility that automated systems have and will continue to propagate existing inequalities under the appearance of objectivity. 

This has a serious impact on people’s lives when these models are used for sentencing, lending, hiring, insurance claims, medical care, and other systems where the people affected may never see, much less be able to contest, how a decision was made.

Okay, I'm concerned. Now what?

This is heavy, but it is worth sitting with as AI begins to touch nearly every part of our lives.

If you are an executive considering where AI fits within your company, or simply exploring how to use it in your own work and life, start with these questions:

  • What are we feeding it?

  • What biases may already exist in our company’s data?

  • Where might generative AI lack the context or nuance this decision requires?

  • Who will remain accountable for the outcome?

  • Where is AI genuinely useful, and where is it unnecessary?

We need to be particularly mindful about what we ask AI to decide on our behalf.

Every system learns from something, and that something is never entirely neutral. 

Before handing AI a process that affects how people are evaluated, selected, rewarded, treated, or denied an opportunity, ask what history is embedded in its inputs. Would you trust that history if a person handed it to you directly?There are already practical frameworks for beginning this work. The National Institute of Standards and Technology’s (NIST) AI Risk Management Framework, for example, gives organizations a structured way to govern, map, measure, and manage AI-related risks before those risks reach the people they may affect. But frameworks alone cannot make these decisions for us. 

Before reaching for AI, we should ask not only, “Can this be automated?” but also, “What kind of human judgment does this require?”

Not everything that can be automated should be. 

Some of the most important parts of how we work and live together—the parts Momentum has spent thirty years helping people rebuild—happen in moments of friction: the honest conversation nobody wants to have, the difficult questions that reveal an assumption, or the moment someone who knows us tells us something we did not want to hear.

Those moments are uncomfortable. 

They are also where something new becomes possible.

The client email that derailed my team’s day did not require a more sophisticated predictive system. It required someone to pick up the phone and ask, “What did you mean?”AI can help us process information, recognize patterns, explore possibilities, and work more efficiently. 

What it cannot do is take responsibility for the assumptions we bring to it or the human consequences of the decisions we make with it. So perhaps, the question is not whether AI can ever understand us perfectly.

It is whether we understand ourselves well enough to recognize what we are asking it to carry and whether we are wise enough to keep the most human decisions in human hands. 

~ Christa

Want to dive deeper? Here’s some further reading.

Sources

Ready to explore how authentic appreciation fuels breakthrough leadership, strengthens organizational culture, and drives business transformation? Share your appreciation story with us.

Send Christa a message.

Fundamental of the Week #18: PRACTICE BLAMELESS PROBLEM-SOLVING  

Focus on finding a solution, not who is at fault; apply your creativity, spirit, and enthusiasm to developing solutions. Then identify lessons learned and use them to improve processes and strengthen relationships.

Momentum Consulting offers executive business coaching, top-level executive consulting, team training, and team off-sites to build and transform your business to the next level. Inquire about business consulting and leadership coaching today.

Christa Clark https://christaclark.com
Next

Committed to Being Right?