Why It Feels Like You Know: "AI Surface Expertise" and the Siren Call of AI for Strategy & Leadership

Most owners I work with are now using AI to develop thinking they would have asked me to develop in the past. I've contributed to AI-generated plans and watched them get discussed in leadership meetings, and it's gotten me wondering when AI-instead-of-expert works, and when it doesn't.

I've already written about Cognitive Runaway and AI Logic Engineering. The new term here is AI Surface Expertise: thinking that looks like expertise on the surface, but is missing critical substance that expertise itself holds.

Experts get more out of tools than novices. So as a leader, it’s important to understand when the novice’s work-with-AI is good enough for strategy and leadership work, and when it will lead down a path that looks good at the start and only shows its problems later on.

AI is enabling confidence to precede competence

My clients feel like they have a whole range of newfound competence. That feeling makes sense. In our own domains, work that is as polished as AI offers has historically meant that it is expert work. AI now delivers that finish in fields where we don’t know what good looks like…and that makes it feel like “this must be right.”

Most of my work involves applying judgment. And I can tell you, after dozens (hundreds?) of hours using AI to try to engineer “judgment systems”...AI’s performance where judgment matters is especially suspect.

So, most of what a leader does with AI is fine, when it doesn’t involve judgment, or is supporting the leader’s expert judgment. The biggest problems I see are on the decisions where the answer depends on judgment, and the judgment depends on context that requires expertise to put into the prompt.

Strategy and leadership decisions are often complex, important, expensive, uncertain, ambiguous, and novel - and those are conditions that are not well-suited to AI’s expertise, and make it challenging to put the right context into the prompt. A “narrow band of AI’s limitations” is a lot wider when you’re working on strategic questions.

AI’s good enough answers are good enough when you have the expert judgment that AI doesn’t.

Expertise is eight jobs, and AI does a partial version of most of them

When you are the expert, eight things are happening:

1. Making the work.
2. Judging the work against a standard (what good looks like).
3. Building the frame the situation needs.
4. Judging the question - whether it is well formed and whether it is the right question at this point.
5. Understanding the standard itself, including the parts nobody in the field has written down.
6. Knowing the edge of your own competence and feeling when a problem sits outside it.
7. Carrying authority, so that people reason inside the frame you endorse.
8. Paying the cost (being accountable) if the expertise is wrong.

AI does a version of most of these. Ask it to judge the work and it will. Ask it for a frame and it will help you create one. But AI’s insight is only partial, and the partial part is always the same part: what depends on knowing my particular situation rather than situations a lot like mine. And understanding what “my particular situation” means in all its fullness, complexity, ambiguity, and uncertainty.

The difficulty with AI is that its partial expertise (application to your situation) arrives with the same finish as its complete expertise (generalized knowledge about the field), so if you don’t know any better, you think it knows when it really doesn't.

Only the last job of expertise - being accountable - is out of reach entirely. AI cannot be wrong in a way that costs it anything. An expert can.

What AI Surface Expertise is

AI Surface Expertise is the artificial expertise that AI enables by doing the production side of expertise, at the finish and confidence of expert work, without the judging, framing, standard-holding, and calibrating substance that real experts have. The surface is complete. The substance underneath is incomplete, misdirected, or absent.

It is the individual version of Cognitive Runaway, and it is how a capable person, with no lapse in judgment, presents what looks like expert knowledge without expert understanding.

The risk isn't AI or the user - the risk is the pairing

AI does what it does everywhere: a partial version of a job, at full finish. We do what we do everywhere: have more expertise in some areas and less in others. Put those two dynamics together and the tool's weaknesses land where our evaluation is weakest. The two gaps line up.

That is why the same document is safe in an expert's hands and quietly unreliable in a non-expert's. It also explains why improvements aimed at one side miss: a better model produces a better partial version at the same finish, and a more careful user runs the checks they know inside the expertise they have. Neither touches the pairing, and the pairing is where the risk lives.

Fixing one side of the AI-user pair does not reach it. That's the point of Logic Engineering: it changes how the pairing actually works.

Surface Expertise looks good to the non-expert

Four of the eight jobs of expertise are where the thinness of AI’s expertise is most problematic.

The standard. AI holds the generic version of the standard in all fields; the non-expert takes AI's output as the standard. The non-expert is counting on AI to check itself, and AI is not trustworthy doing that. (If you haven’t discovered this…you will.)

The frame. AI only starts thinking when given a prompt, and inherent in every prompt is a frame. AI builds inside the frame it's given. You can overcome that limitation by asking for alternatives (and it will give alternatives) or different angles (and it will give different angles). But in judgment-heavy strategy and leadership work, those are unlikely to be the options and angles you need. (If you haven't discovered this...you will.) AI will now say when it thinks the framing is wrong…but I find that reframing is often flat, and misdirected - and it announces it just as confidently as all of its answers.

The question. Ask whether it was the right question and you get a different question and a different answer. Ask again and again and you can have five sets of questions and answers to sort through. At that point, it’s clear that real expertise lies in asking which question is the right one to ask.

The edge. An expert locates a new problem against a stock of cases and can feel when it sits outside their range. A non-expert has no cases. And I have never had AI tell me a question was outside its range. Ever.

Surface Expertise undermines strategic discussion

There are two problems that show up when leaders use an AI-generated document that is built with Surface Expertise to drive a strategic discussion.

First, unless a group consciously fights it, the team grants the document more expertise than it deserves, and the frame in the document becomes the frame the group reasons inside. As the line often attributed to Einstein goes, give me an hour and I’ll spend the first 55 minutes framing the problem. Framing matters. A lot.

In a strategic decision that is complex, important, expensive, uncertain, ambiguous, and novel, the chance that a non-expert got the frame right is a coin-toss.

Second, when someone arrives with a fully-baked answer, it gets harder to question if the premise of the decision is the right one. It’s harder to rewind to the foundation thinking when the team is presented with ten slides instead of one.

Worst of all, if you’re not looking for these problems, they’ll actually feel like a meeting going well.

What this looks like in a leadership team discussion

One owner I work with brought his leadership team an AI-generated slide deck about succession. A half-dozen times, he said, “So, this is what AI says” about different aspects of the thinking - clearly referencing AI as the source of judgment. He also undermined what AI said sometimes: “Well, you can ignore that part because it doesn’t really make sense."

Succession is quite complex. For anyone who is looking at leaving, there are 20+ roles and 20+ relationships that need to be navigated. That kind of data seems perfect for AI.

It’s also quite simple. 80% of succession readiness is actually, “Have a business that runs well, and address any gaps.” If you don’t have that, succession gets much harder and more complex.

The slide deck that drove the discussion that day was a great example of not seeing the forest for the trees. AI did a brilliant job of organizing and integrating the details.

But it did a terrible job of highlighting the critical issues that the team needed to understand and discuss that day. And it spent too much time on trees that may be important in a typical business, but weren’t important here.

The discussion we had that day was, “What do you think of a well-assembled view of all the details of succession that we will need to manage over the next 10 years?”

How I would have framed the discussion: “Here’s a broad view of what our succession needs are now, and the details for a live succession event we need to manage now.”

Those may not sound different, but they are. One gets lost in the detail and leaves the work of synthesis and application to happen after the meeting; the other aligns the team directionally and focuses them on allocating the right resources to take action.

When AI thinking drives a strategy discussion, it’s a coin-flip that it’s going to focus on the right topic. At best, that gets recovered by pushing the AI deck aside, taking a few parts that are useful. At worst, the team reaches a fast decision on the wrong basis…and we all know that errors that aren’t caught early get more expensive.

Surface Expertise feels great…but doesn’t get us as far as we think

I find it interesting that people know that AI has meaningful limitations in the domains where they’re expert, but then rely on it for sophisticated answers, strategy, and planning in areas where they have little understanding.

While our range of competence is expanding at the personal level, the environment is getting more complex, and what’s needed to solve problems and differentiate ourselves is also increasing.

So at the personal level it feels like we’re able to solve a wider range of problems…but I don’t think we are, or we won’t be able to for long. I think the tide is lifting all boats, and AI-for-all will keep raising the quality bar.

And tools amplify expertise. So I wonder if this feeling of newfound competence will only last another 6-12 months before we realize that there are limitations to DIY, and that we’re right back to where we were - picking an area to be an expert in, and using the amazing power of AI to be great in that area.

What to do with the feeling of competence AI gives us

How do we fight the siren call of Surface Expertise? Take an honest inventory of the eight jobs of expertise and ask which ones you bring. If the answer is that you did the work, plus a borrowed version of understanding the standard and carrying authority, your belief has likely outrun your competence, and that’s dangerous for strategy and leadership discussions.

Or make it concrete: could I tell a weak version of this analysis from a strong one, without AI pointing the way? If not, your feeling of expertise can’t actually tell you which parts of the work are right.

With strategic decisions that are complex, important, expensive, uncertain, ambiguous, and novel, good enough is probably not good enough, AI’s judgment is probably misguided, and the ROI of an expert is likely strong.

Frequently asked questions

What is AI Surface Expertise?

AI Surface Expertise is expert-looking work without the substance that expertise holds. It happens when a non-expert uses AI to produce work at expert finish, and the judging, framing, standard-holding, and self-calibrating parts of expertise are thin with the AI and the user. The person feels competent because the output matches what competence looks like in their own field, and the people around them read the finish the same way. The term names how capable people present, and accept, AI work they cannot evaluate. It pairs with Cognitive Runaway, which names the organizational result, and AI Logic Engineering, the organizational answer.

What are the eight jobs of an expert?

An expert does eight things at once, and AI does a partial version of seven of them.

1. Making the work.
2. Judging the work against a standard, what good looks like.
3. Building the frame the situation needs.
4. Judging the question: whether it is well formed, and whether it is the right question at this point.
5. Understanding the standard itself, including the parts nobody in the field has written down.
6. Knowing the edge of their own competence and feeling when a problem sits outside it.
7. Carrying authority, so that people reason inside the frame they endorse.
8. Paying the cost if the expertise is wrong.

AI's version of each is thin on the same part every time: what depends on this particular situation rather than situations like this one. The eighth job AI cannot do at all. A leader can use the list as an inventory: which of these do I hold in this field myself, as opposed to which ones the output did a version of for me.

How is AI Surface Expertise different from overconfidence?

Overconfidence is an estimate of your ability running ahead of your actual ability. AI Surface Expertise is that estimate being right about the wrong thing. The feeling reports, correctly, that expert-finish work was produced. It cannot report on whether the question was right, whether the frame fit, whether the standard was met, or whether the problem sat outside anyone's range, because those readings come from jobs you hold in your own field and do not hold in this one. The two travel together. Surface Expertise often carries overconfidence with it, since AI often enables belief of competence preceding actual competence.

How can a leader tell when AI output is beyond their ability to judge?

Ask whether you could tell a weak version of this analysis from a strong one without AI's help. If you can name what a weak version would get wrong, you hold enough of the standard to judge it. If you cannot, your confidence is reporting on the production, and the work needs a second reader who holds the standard in that field. The eight-jobs inventory is the longer form: which of the jobs do you hold yourself, as opposed to which ones the AI output did a version of for you. If the answer is that you used AI for production, and to know the standard and carry authority, your belief is likely running ahead of competence.

When is AI good enough in place of an expert?

Much of the time, when judgment isn’t involved. Retrieval, drafting, summarizing, and working inside a framework you already hold are places where the tool outruns a competent person and your bar is a good bar. The band where it is not: decisions whose answer depends on judgment, and whose judgment depends on context that takes expertise to put into the prompt. A fact about people. A thing nobody has admitted. A comparison to a case you have never seen. On those questions the thin output arrives confidently, so the flaw does not announce itself. When good enough is enough, and when it is not, is the subject of a later piece in this series.

How does AI Surface Expertise affect leadership, strategy discussions, and strategic planning?

Leadership and strategy are where AI Surface Expertise is most problematic. Leadership and strategic questions are complex, ambiguous, important, uncertain, expensive, and novel, and the answer depends on context about people and situation that is often difficult to put into an AI prompt. That is where AI's thin part concentrates, and where a leader's evaluation has the fewest cases to check against. In a planning discussion, the effect shows up in three ways. (1) The conversation starts at the wrong step, because the AI’s plan answered the question asked rather than the one an expert would have started with. (2) The AI’s framing becomes the team's framing, so people are forced to reason inside it. (3) The decision comes fast, because a finished-looking plan invites approval rather than argument. The danger is that the cost surfaces late, since strategic errors take quarters to show.

How do AI Surface Expertise, Cognitive Runaway, and AI Logic Engineering fit together?

AI Surface Expertise is the person-level mechanism: a leader presents and accepts AI work they cannot evaluate because it carries expert finish. Cognitive Runaway is the organizational result: mediocre output gets accepted as the standard and the standard quietly moves. AI Logic Engineering is the organizational answer: designing the reasoning so the standard is built into the work instead of checked for afterward. Because the risk lives in the pairing of AI and user, improving either side alone does not reach it. Logic Engineering is the fix that changes the pairing.

When the answer looked right and you could not say why

If a piece of AI work felt finished and you could not say what would make it wrong, that is a conversation I have with owners most weeks now. I help Second Stage companies treat AI as part of business strategy. Let's talk.