When AI Assistance Becomes Invisible: Organizational Challenges of Competence Illusion in the Age of Generative AI

Listen to a review of this article:

Abstract: The proliferation of large language models (LLMs) in knowledge work has fundamentally altered how individuals perform cognitive tasks and perceive their own capabilities. This article examines the LLM fallacy, a cognitive attribution error in which individuals systematically misinterpret AI-assisted outputs as evidence of independent competence, creating divergence between perceived and actual capability. Drawing on theories of automation bias, cognitive offloading, and distributed cognition, we analyze how LLM interaction properties—including opacity, fluency, and immediacy—obscure the boundary between human and machine contributions. Organizations face mounting challenges as traditional evaluation frameworks struggle to distinguish system-assisted performance from independently grounded expertise. We examine implications across hiring, credentialing, education, and professional development, and propose organizational responses centered on transparency architectures, process-aware evaluation, and calibrated AI literacy. This synthesis bridges individual-level attribution dynamics with institutional assessment practices, offering evidence-based guidance for organizations navigating the transformation of cognitive work in the age of generative AI.

Something unprecedented is happening in knowledge work. An analyst produces sophisticated market forecasts without understanding the underlying statistical models. A developer deploys functional code while unable to explain its architecture. A consultant generates polished strategy documents in domains where they lack subject matter expertise. Each demonstrates apparent competence through high-quality outputs, yet each relies fundamentally on large language model assistance that remains largely invisible in the final product.

This pattern represents more than productivity enhancement—it signals a structural transformation in how cognitive labor operates and how competence itself is constructed, perceived, and evaluated. The rapid integration of LLMs into everyday workflows has created what Kim et al. (2026) term the LLM fallacy: a systematic cognitive attribution error in which individuals misinterpret AI-assisted outputs as evidence of their own independent capabilities.

The stakes extend well beyond individual self-perception. Organizations increasingly face a fundamental measurement problem: when outputs can be substantially shaped by AI systems operating behind the scenes, traditional performance indicators become unreliable proxies for underlying capability. Hiring managers evaluate portfolios that may reflect AI contribution more than human expertise. Educational institutions assess work products without visibility into the generation process. Certification bodies validate skills based on demonstrations that may not transfer to unaided contexts.

Why this matters now: Three converging forces make the LLM fallacy particularly consequential. First, LLM capabilities have reached a threshold where outputs genuinely approximate or exceed typical human performance across many cognitive domains (Brynjolfsson et al., 2025). Second, integration has become seamless—these systems now function as embedded components within workflows rather than external tools requiring explicit invocation. Third, organizational evaluation systems were designed for a world where observable outputs reliably indicated underlying competence, an assumption that no longer holds in AI-mediated environments.

This article synthesizes emerging research on the LLM fallacy with established frameworks from human–AI interaction, organizational evaluation, and cognitive psychology. We examine how attribution errors emerge at the individual level, propagate through institutional systems, and reshape the relationship between performance and competence. Drawing on evidence from software development, hiring contexts, and educational settings, we identify mechanisms driving capability misattribution and propose organizational responses that preserve both AI's productivity benefits and the integrity of capability assessment.

The analysis proceeds in several stages. We first situate the phenomenon within existing literature on automation, cognitive offloading, and distributed cognition, distinguishing it from related constructs. We then examine the cognitive and system-level mechanisms that produce attributional ambiguity in LLM-mediated workflows. A domain-specific analysis reveals how the fallacy manifests across computational, linguistic, analytical, and creative work. We analyze implications for organizational evaluation systems—including hiring, education, and credentialing—before outlining evidence-based responses and future research directions.

The AI-Augmented Work Landscape

Defining Capability Misattribution in AI-Mediated Contexts

The LLM fallacy describes a specific form of metacognitive failure: the inability to accurately assess the sources and limits of one's own knowledge when working with generative AI systems (Kim et al., 2026). Unlike general overconfidence or self-enhancement biases, this phenomenon arises directly from the integration of opaque, fluent AI systems into cognitive workflows. Several conditions jointly produce the effect.

Interaction seamlessness refers to the low-friction nature of LLM engagement, where sophisticated outputs emerge from minimal prompting without exposing intermediate reasoning (Burrell, 2016). Users provide partial inputs—often underspecified instructions or fragmented ideas—and receive polished, complete outputs. This asymmetry obscures the extent of system contribution.

Output fluency describes the grammatical correctness, stylistic consistency, and contextual appropriateness of LLM-generated text, which closely resembles skilled human production (Alter & Oppenheimer, 2009). High fluency functions as a metacognitive cue, leading individuals to infer competence from surface-level coherence rather than evaluating the underlying generative process.

Attribution ambiguity emerges because human–AI outputs result from iterative co-construction rather than clearly separable contributions. Users shape outputs through prompting and selection, while systems generate substantive content. This entanglement makes it difficult to disentangle respective contributions, particularly when individuals lack visibility into what the system actually "knows" versus pattern-matches (Nisbett & Wilson, 1977).

Pipeline opacity compounds these challenges. Unlike traditional tools where users observe intermediate steps, LLMs abstract away retrieval, synthesis, and reasoning processes. Users see only inputs and outputs, limiting their ability to construct accurate mental models of how results are produced (Ananny & Crawford, 2018). Research shows that when system processes remain opaque, users form incomplete or inaccurate representations of capability boundaries, increasing attribution error susceptibility (Kocielnik et al., 2019).

The LLM fallacy differs from hallucination—a system-level failure where models generate incorrect information (Ji et al., 2023)—because it concerns user interpretation rather than output accuracy. It persists regardless of whether generated content is factually correct. Similarly, it extends beyond automation bias (over-reliance on system outputs) and cognitive offloading (delegating mental effort externally) by focusing specifically on how outputs reshape self-perception of independent capability rather than affecting task execution alone (Risko & Gilbert, 2016).

State of AI Integration in Knowledge Work

Recent evidence documents rapid LLM adoption across professional contexts. Brynjolfsson et al. (2025) found that generative AI assistance significantly improved productivity among customer service representatives, with effects concentrated among lower-skill workers. However, the study also revealed a critical pattern: performance gains occurred primarily when AI remained available, with limited evidence of skill development or knowledge transfer to unaided contexts.

This pattern appears across domains. In software development, LLMs now generate substantial code volumes, yet developers frequently lack deep understanding of implementation details, dependencies, or architectural decisions embedded in AI-generated solutions (Nam et al., 2024). Code may function correctly at surface level while containing latent issues detectable only through domain expertise—a divergence between operational success and underlying correctness that mirrors the core dynamic of the LLM fallacy.

Educational contexts show similar dynamics. When students use AI to scaffold assignments, complete problem sets, or generate explanations, they engage less deeply with underlying material, reducing opportunities for knowledge internalization (Gajos & Mamykina, 2022). While AI assistance can enhance access and support learning, it also complicates interpreting performance outcomes, as observable work products no longer reliably indicate conceptual understanding.

Across these contexts, a consistent finding emerges: AI assistance enables output production that exceeds independent capability, creating systematic gaps between demonstrated performance and transferable expertise. These gaps remain largely invisible in contexts where only outputs are evaluated, making the attribution error difficult to detect without deliberate assessment of unaided performance.

The Transformation of Evaluation Reliability

Organizations traditionally rely on observable outputs as proxies for underlying capability, an approach grounded in assumptions that outputs reflect internally generated expertise. Performance portfolios, work samples, interviews, and certifications all presume that demonstrated outputs indicate reproducible capability. The LLM fallacy fundamentally challenges this assumption.

When outputs can be substantially AI-mediated without visibility, evaluation systems face an interpretability crisis: high-quality outputs no longer reliably indicate the skills and knowledge they ostensibly demonstrate. This misalignment affects multiple organizational functions. Hiring decisions based on portfolios or technical demonstrations may conflate system-assisted performance with candidate capability. Educational assessments may reward AI-mediated completion rather than conceptual mastery. Professional certifications may validate outputs producible through AI assistance without corresponding transferable expertise.

Research on human–machine teaming reveals why this matters. Performance in hybrid human–AI systems emerges from interaction rather than isolated component capabilities (Damacharla et al., 2018). Evaluating team outputs as individual performance thus introduces systematic attribution errors. Moreover, these errors compound over time: when capability assessments inform selection, promotion, and development decisions, misattribution can produce cascading effects throughout talent systems.

The challenge extends beyond measurement precision. When individuals internalize inflated self-assessments based on AI-assisted outputs, they may avoid skill development that would otherwise occur through struggle and deliberate practice. Organizations that reward outputs without attending to their generation processes may inadvertently incentivize surface-level AI dependency over deeper capability development—creating what some researchers describe as competence erosion through outsourcing (Kirsh, 2010).

Organizational and Individual Consequences of Capability Misattribution

Organizational Performance Impacts

The LLM fallacy introduces systematic risks across multiple organizational domains, each reflecting the disconnect between demonstrated outputs and underlying capability. These risks manifest most clearly when AI-augmented performance must transfer to contexts requiring unaided execution or when collaboration depends on shared understanding rather than surface-level output production.

Hiring and selection effectiveness. Organizations face growing challenges distinguishing genuine expertise from AI-assisted performance during candidate evaluation. Portfolios, work samples, and take-home assessments—long-standing validity cornerstones—become less diagnostic when candidates can leverage LLMs to produce sophisticated outputs without corresponding independent capability. Research indicates this isn't merely theoretical: Karny et al. (2024) found that AI assistance improves observable task performance while simultaneously increasing external system reliance without corresponding gains in independent understanding.

The practical implications are substantial. A technology firm hiring developers based on coding assessments may systematically select candidates whose demonstrated performance reflects AI capability more than transferable programming skill. When these individuals join teams and face complex, ambiguous problems requiring deep architectural reasoning, performance gaps emerge—gaps invisible during evaluation. Similar dynamics appear in consulting, where client presentations and written deliverables can be substantially AI-mediated, masking deficits in domain expertise, analytical reasoning, or strategic judgment that surface only in real-time client interactions.

Team performance and knowledge transfer. AI-mediated capability gaps introduce coordination challenges within teams. When team members possess different levels of genuine versus AI-dependent capability, knowledge sharing becomes asymmetric. Individuals with deeper independent expertise can mentor, explain nuances, and transfer understanding; those with primarily AI-dependent capability cannot, because they lack internalized knowledge to transmit (Hutchins, 1995). This creates what organizational researchers call structural holes—gaps in knowledge networks that impede information flow and collaborative problem-solving.

The problem intensifies in contexts requiring real-time adaptation or creative synthesis. Meeting facilitation, client negotiation, crisis response, and strategic pivoting all demand immediate application of expertise without opportunity for AI consultation. Teams containing members whose capabilities depend heavily on AI assistance face unrecognized performance constraints in these settings.

Risk exposure and quality assurance. Perhaps most concerning are contexts where capability gaps introduce safety, quality, or compliance risks. Zhang et al. (2025) documented how LLM-generated code, while often surface-level functional, can contain latent errors undetectable without domain expertise. Similar patterns likely exist in legal document drafting, financial analysis, medical documentation, and other high-stakes domains. When individuals over-rely on AI-generated outputs they cannot fully evaluate, organizations face increased exposure to errors that manifest only under edge conditions or when assumptions embedded in AI outputs prove invalid.

This risk compounds because the LLM fallacy creates confidence-competence misalignment: individuals believe they possess capabilities they don't actually have, reducing their vigilance and error-checking. Research on metacognition shows that perceived expertise influences how carefully individuals verify outputs—greater confidence correlates with reduced verification (Koriat, 1997). When AI-assisted performance inflates self-assessed capability, it simultaneously reduces the critical evaluation that would catch AI-introduced errors.

Individual Well-being and Development Impacts

Beyond organizational effects, the LLM fallacy shapes individual professional development, career trajectories, and long-term capability building in ways that may not become apparent until AI scaffolding is removed.

Skill development and expertise acquisition. Expertise development requires sustained engagement with challenging problems, productive struggle, and iterative practice that builds internalized knowledge structures (Ericsson, 2006). AI assistance can short-circuit this process. When individuals consistently turn to LLMs for tasks that would otherwise require effortful reasoning, they miss opportunities for the cognitive engagement that builds transferable expertise.

Gajos and Mamykina (2022) demonstrated this dynamic empirically: when AI systems provided intermediate problem-solving steps, users engaged less deeply with underlying concepts, limiting incidental learning and knowledge consolidation. The result resembles what educational psychologists call illusion of understanding—individuals experience fluency with material through AI-mediated interaction without developing the independent capability to apply concepts in new contexts (Rozenblit & Keil, 2002).

This matters profoundly for career development. Early-career professionals who over-rely on AI assistance may fail to build foundational capabilities that enable later advancement. Mid-career professionals may plateau as they discover that AI-assisted outputs don't translate to leadership contexts requiring strategic judgment, mentorship, or creative problem formulation. The gap between perceived and actual capability often becomes visible only when individuals face performance contexts that don't permit AI consultation—precisely the high-stakes situations that define career advancement.

Professional identity and authenticity. The LLM fallacy introduces novel tensions around professional identity and authentic contribution. When substantial work product reflects AI generation, individuals face questions about authorship, ownership, and the relationship between their professional identity and the outputs they produce. Draxler et al. (2024) identified what they term the AI ghostwriter effect: users may not fully experience ownership of AI-generated content cognitively yet declare authorship socially, revealing dissociation between experienced and attributed contribution.

This dissociation creates psychological ambiguity. Professionals may question whether accomplishments reflect genuine capability or primarily AI assistance—a form of competence doubt that can affect motivation, satisfaction, and sense of professional efficacy. Paradoxically, while the LLM fallacy typically involves overestimating independent capability, awareness of heavy AI reliance can produce the opposite effect: individuals discount genuine contributions because they can't cleanly separate their thinking from system influence.

Economic vulnerability and career resilience. Perhaps most concerning are long-term economic implications. Professionals whose capabilities rest heavily on AI assistance face vulnerability if AI access is disrupted, if AI capabilities plateau while job demands increase, or if employers increasingly distinguish and preferentially value independent versus AI-dependent performance. The latter seems increasingly likely: as organizations recognize the LLM fallacy's effects, evaluation practices may shift toward process-based assessment, live demonstrations, and deliberate AI-free contexts—changes that would disadvantage those who've developed primarily AI-dependent capabilities.

Dillon et al. (2025) found that generative AI is already shifting work patterns, with implications for skill demand and labor market dynamics. Professionals who build genuine expertise augmented by strategic AI use are positioned differently than those whose performance depends primarily on AI scaffolding. The former possess transferable capabilities adaptable across contexts and tools; the latter face potential capability obsolescence if AI systems evolve, access patterns change, or evaluation standards adjust.

Evidence-Based Organizational Responses

Table 1: Organizational Responses to the LLM Fallacy

Intervention Category	Specific Strategy	Example Organization/Implementation	Primary Goal	Key Mechanisms
Transparency and Process Visibility	Contribution tracking and documentation	Microsoft (engineering teams)	Reduce attribution ambiguity by making AI contribution more visible.	Code annotation practices that flag AI-generated sections to enable appropriate reviewer scrutiny.
Transparency and Process Visibility	Staged evaluation with variable AI access	Goldman Sachs (analyst hiring)	Enable direct comparison of aided versus unaided performance.	Layered approach using take-home projects (AI-aided) followed by real-time case discussions (unaided) to probe understanding depth.
Transparency and Process Visibility	Cognitive process documentation	Not in source	Shift emphasis from final outputs to the generative process.	Requiring commit messages or documentation explaining "why" approaches were chosen and what alternatives were considered.
Evaluation Systems	Competency-based assessment frameworks	Stripe (engineering competency matrices)	Maintain evaluation rigor by decomposing tasks into underlying competencies.	Separating AI-augmentable dimensions (code production) from dimensions requiring independent expertise (architectural judgment, technical strategy).
Evaluation Systems	Portfolio + explanation protocols	Design thinking consultancies	Surface capability gaps and distinguish orchestration from deep understanding.	Candidates walk through project evolution and explain design decisions, trade-offs, and iterative processes in person.
Development and Calibration	Structured AI literacy and metacognitive training	Duolingo (AI tutor pedagogy)	Build accurate self-assessment and calibrated AI reliance.	Scaffolding framework where AI assistance fades over time as learner capability develops to prevent permanent dependency.
Development and Calibration	Mentorship and peer review intensification	Professional service firms (paired work models)	Help junior practitioners distinguish augmented performance from genuine skill development.	Experienced professionals review work processes and ask questions revealing understanding depth rather than just reviewing outputs.
Development and Calibration	Deliberate practice protocols	Engineering teams ("AI-free reasoning" sessions)	Counteract capability erosion and build transferable expertise.	Establishing protected, AI-free spaces like weekly problem-solving sessions or unplugged strategy sessions.
Culture and Norms	Realistic AI discourse and expectation-setting	Anthropic (transparent AI use in operations)	Create organizational norms that value both effective AI use and independent capability.	Leadership explicit discussion on when AI use is appropriate versus when human reasoning is essential.
Long-Term Adaptation	Structured reflection practices	Boston Consulting Group (AI integration reviews)	Engineer new feedback loops for learning and capability calibration.	Quarterly capability audits and after-action reviews to assess AI use patterns and potential skill gaps.

Addressing the LLM fallacy requires interventions spanning individual users, team practices, evaluation systems, and organizational culture. Effective responses distinguish between AI literacy (understanding how systems work and their appropriate uses) and metacognitive calibration (accurately assessing one's own capabilities relative to AI contribution). Both are necessary; neither alone is sufficient.

Transparency and Process Visibility Interventions

Organizations can reduce attribution ambiguity by making AI contribution more visible and creating shared understanding of where capability resides.

Contribution tracking and documentation. Research in human-centered AI emphasizes transparency as fundamental to appropriate reliance and trust calibration (Shneiderman, 2020). Organizations can adapt these principles by implementing practices that document AI involvement in work products. This might include requiring disclosure of AI tool usage in portfolios, internal documentation noting which analyses or documents involved substantial AI assistance, or even version control practices that distinguish human-authored content from AI-generated or AI-modified sections.

Microsoft's implementation approach. Microsoft's engineering teams have experimented with code annotation practices that flag AI-generated sections, enabling reviewers to apply appropriate scrutiny. This doesn't prohibit AI use but makes contribution sources transparent, allowing teams to distinguish genuinely novel problem-solving from effective AI orchestration—both valuable but different skills.

The key is balancing transparency with stigma avoidance. Documentation shouldn't imply AI assistance is problematic but rather reflect realistic understanding that different work products involve different contribution mixes. The goal is calibrated assessment, not moral judgment.

Staged evaluation with variable AI access. Organizations can design assessment processes that deliberately vary AI availability, enabling direct comparison of aided versus unaided performance. This approach parallels experimental designs in capability research but adapts them to practical contexts.

Interview protocols might include both take-home projects (where AI use is expected and appropriate) and live problem-solving sessions without AI access, creating data on both orchestration ability and independent reasoning. Educational institutions are increasingly adopting hybrid assessment models: some assignments permit full AI assistance to teach effective AI collaboration, while others prohibit it to evaluate conceptual mastery and independent application.

Goldman Sachs' analyst assessment evolution. Goldman Sachs reportedly adjusted analyst hiring to include real-time case discussions following take-home modeling exercises. Candidates discuss their analyses live, enabling interviewers to probe understanding depth, identify areas where candidates struggle to explain rationale (suggesting AI-assisted completion), and assess strategic reasoning beyond output production. This layered approach evaluates both AI orchestration skills and underlying financial acumen.

Cognitive process documentation. Rather than focusing solely on outputs, organizations can require documentation of reasoning processes, decision pathways, and alternative approaches considered. This shifts emphasis from what was produced to how it was generated, making AI contribution more visible because AI-generated insights typically arrive without the exploratory reasoning process that characterizes human problem-solving.

Engineering teams using AI code generation might require commit messages documenting why particular approaches were chosen, what alternatives were considered, and how the solution fits within broader architecture. Consulting teams might document client recommendation development processes, showing analytical steps, data interpretation, and strategic reasoning rather than only final deliverables. These practices create visibility into whether outputs reflect deep understanding or sophisticated AI orchestration.

Process-Aware Evaluation Systems

Traditional output-centric evaluation systems require rethinking to account for AI mediation. Several organizations are pioneering approaches that maintain evaluation rigor while acknowledging AI's role.

Competency-based assessment frameworks. Rather than evaluating outcomes alone, organizations can decompose complex tasks into underlying competencies and assess each dimension. For software engineering roles, this might separate architectural reasoning, debugging capability, code review quality, and novel problem formulation from code production—the dimension most easily AI-augmented. By evaluating each capability independently, often through targeted, time-limited exercises, organizations gain clearer pictures of where genuine capability resides versus where performance depends on AI scaffolding.

Stripe's engineering competency matrices. Stripe publicly shares engineering competency frameworks that distinguish multiple skill dimensions. During performance evaluation, managers assess not just code shipped but architectural decision-making, mentorship quality, technical judgment in ambiguous situations, and systems thinking. This multi-dimensional approach reduces vulnerability to the LLM fallacy because strong performance on AI-augmentable dimensions (code production) doesn't automatically indicate strength on dimensions requiring independent expertise (architectural judgment, technical strategy).

Behavioral interviewing and situation-based assessment. Organizations can emphasize assessment methods that reveal how candidates think rather than what they've produced. Behavioral interviewing—focused on past experiences, decisions made, lessons learned—provides insight into reasoning processes and conceptual frameworks that individuals have genuinely internalized. Situational exercises requiring real-time problem-solving without AI access reveal independent capability.

Educational institutions are adapting similar principles. Some business schools now use live case exercises where students analyze business problems in real-time without external resources, complementing take-home analyses where AI use is expected. This combination evaluates both AI collaboration effectiveness and independent strategic reasoning.

Portfolio + explanation protocols. Rather than prohibiting AI use in portfolios (which may be unenforceable and fails to credit genuine AI collaboration skill), organizations can require detailed explanations of portfolio work. Candidates present outputs but must thoroughly explain design decisions, trade-offs considered, how they would modify approaches for different contexts, and limitations of presented solutions.

This approach surfaces capability gaps. Individuals who've generated impressive outputs primarily through AI assistance struggle to explain nuanced reasoning, alternative approaches, or contextual adaptations because they lack the internalized expertise to access this knowledge. Those with genuine capability augmented by strategic AI use demonstrate understanding depth that transcends surface-level output quality.

Design thinking consultancies' portfolio reviews. Leading design firms have adapted portfolio reviews to probe process and reasoning. Candidates walk through project evolution, explaining ideation processes, user research interpretations, iterative design decisions, and what they'd change with hindsight. This reveals whether impressive portfolio pieces reflect deep design thinking capability or primarily skillful AI tool use—both have value, but they represent different competency profiles.

Capability Development and Calibration Programs

Organizations can actively work to build accurate self-assessment and ensure AI augmentation enhances rather than substitutes for capability development.

Structured AI literacy and metacognitive training. Liao et al. (2020) emphasize that effective human–AI collaboration requires users to understand system capabilities, limitations, and appropriate use contexts. Organizations can develop training that explicitly addresses the LLM fallacy, teaching individuals to recognize when performance depends heavily on AI versus reflects independent capability.

This training might include exercises where individuals first complete tasks with AI assistance, then attempt them independently, with facilitated reflection on performance differences. By directly experiencing capability gaps, individuals develop more calibrated self-assessment. Training can also teach strategies for using AI in ways that build rather than bypass capability—for instance, using LLMs to generate initial drafts that individuals then substantially rework and validate rather than accepting outputs wholesale.

Duolingo's AI tutor pedagogy. Duolingo's AI-powered language tutoring explicitly balances assistance with capability building. The system provides scaffolding that fades over time, gradually reducing support as learner capability develops. This approach—borrowed from educational psychology's scaffolding framework—recognizes that effective AI assistance should build toward independence rather than create permanent dependency.

Mentorship and peer review intensification. Senior practitioners can help junior colleagues distinguish AI-augmented performance from genuine capability development through active mentorship. This requires experienced professionals to review not just outputs but work processes, asking questions that reveal understanding depth and identifying over-reliance on AI.

Professional service firms are experimenting with paired work models where junior professionals work alongside experienced colleagues on client projects, enabling real-time feedback on AI use patterns, reasoning processes, and capability gaps. This contrasts with traditional models where junior staff complete assignments independently (now often AI-assisted) with only final output review—a pattern that can mask capability deficits.

Deliberate practice protocols. Organizations committed to capability building can establish regular AI-free practice contexts where professionals develop independent skills. This might include weekly problem-solving sessions, case discussions, or technical challenges completed without AI assistance—protected spaces for effortful practice that builds transferable expertise.

Engineering teams might designate certain code review sessions as "AI-free reasoning" where reviewers explain technical decisions without consulting AI documentation or assistance. Consulting teams might conduct "unplugged strategy sessions" where members develop client recommendations using only human discussion and whiteboard thinking. These practices counteract capability erosion by ensuring regular independent skill exercise.

Organizational Culture and Norms Development

Individual interventions, while necessary, are insufficient without broader cultural shifts around AI use, capability attribution, and professional development.

Realistic AI discourse and expectation-setting. Leadership can explicitly acknowledge AI's role in work while emphasizing that genuine capability development remains essential. This means rejecting both extremes—neither prohibiting AI use (which is unrealistic and fails to develop important AI collaboration skills) nor treating all AI-assisted outputs as equivalent to independently developed work.

Anthropic's transparent AI use in operations. Anthropic, the AI company, has publicly described how it uses AI extensively in internal operations while maintaining clear distinctions between AI-assisted work and human judgment-requiring decisions. Leadership explicitly discusses when AI is appropriate versus when human reasoning is essential, creating organizational norms that value both effective AI use and independent capability.

Recognition systems that value process and development. Organizations can adjust recognition and advancement criteria to emphasize capability growth, mentorship, independent problem-solving, and other dimensions that aren't easily AI-substitutable. Performance evaluations might explicitly assess how individuals balance AI leverage with capability building, rewarding those who use AI strategically to augment expertise rather than bypass learning.

Professional development could include explicit "capability milestones" separate from output-based achievements—demonstrating independent mastery of specific technical skills, completing unaided strategic analyses, or successfully mentoring others through complex problems. These milestones would signal genuine expertise rather than AI-assisted performance.

Ethics and professional standards evolution. Professional associations can develop updated ethics guidance addressing AI use, attribution, and capability representation. These standards would parallel existing norms around plagiarism and proper attribution but adapt them to AI-mediated contexts where contribution boundaries are inherently ambiguous.

Such standards might address when AI use requires disclosure, how to fairly represent AI-assisted work in portfolios or credentials, and obligations to maintain independent capability even while using AI augmentation. By establishing field-wide norms, professional associations can reduce the competitive pressure to misrepresent AI-dependent performance as independent capability.

Building Long-Term Capability in AI-Augmented Organizations

Beyond immediate responses to the LLM fallacy, organizations must develop sustainable approaches to capability development, evaluation, and management in permanently AI-mediated environments. This requires rethinking foundational assumptions about expertise, performance, and professional development.

Redefining Competence in Hybrid Human–AI Systems

Traditional competence models assume capabilities reside primarily within individuals. AI integration requires transitioning toward frameworks that recognize competence as distributed across human expertise, AI capabilities, and orchestration skill—what Clark (2010) termed the extended mind applied to organizational contexts.

Distinguishing competence dimensions. Organizations can develop explicit taxonomies that separate:

• Core human judgment: Strategic reasoning, contextual interpretation, ethical decision-making, creative problem formulation—dimensions where human cognition remains superior and AI provides limited augmentation.

• Augmentable execution: Technical implementation, document production, initial data analysis, pattern recognition in structured domains—areas where AI substantially enhances human capability when appropriately guided.

• Human–AI orchestration: The meta-skill of knowing when and how to leverage AI, how to validate AI outputs, and how to combine human and AI strengths effectively.

By explicitly mapping these dimensions, organizations can design development programs, evaluation systems, and career pathways that build appropriate capabilities in each area. This avoids both under-valuing AI orchestration skills (which are genuinely valuable) and conflating AI-augmented performance with independent expertise in core human judgment domains.

Career pathways for the AI-augmented era. Professional development frameworks may need restructuring to reflect these distinctions. Rather than single competency ladders, organizations might develop parallel advancement tracks: technical-execution specialists who excel at AI-leveraged implementation, strategic-judgment experts who focus on the uniquely human reasoning domains, and orchestration leaders who excel at coordinating human–AI collaboration.

This approach acknowledges that different professionals may develop different capability profiles in AI-augmented environments. Someone might be exceptional at leveraging AI for rapid prototyping and implementation while developing their strategic judgment more gradually. Another might focus intensively on deepening domain expertise and human judgment while using AI more selectively. Both profiles have organizational value; both represent legitimate professional development pathways.

Continuous Learning and Adaptation Systems

The LLM fallacy highlights how AI integration disrupts the feedback loops that traditionally supported learning and capability calibration. Organizations must deliberately engineer new feedback mechanisms.

Structured reflection practices. Regular structured reflection on AI use patterns can help individuals develop more accurate self-assessment. Teams might conduct quarterly "capability audits" where members assess which skills they're actively developing versus which they're increasingly delegating to AI. This creates visibility into potential capability erosion and enables proactive development planning.

These practices parallel after-action reviews used in military and healthcare contexts: systematic analysis of performance episodes to extract learning. Applied to AI-augmented work, after-action reviews might examine: What was attempted? What AI assistance was used? What worked well? Where did capability gaps appear? What independent skills should be developed based on this experience?

BCG's AI integration reviews. Boston Consulting Group has reportedly implemented regular team reviews where consultants discuss how AI tools are affecting their work patterns and capability development. These discussions surface concerns about over-reliance, identify best practices for maintaining strategic thinking skills while leveraging AI for analysis, and create peer accountability around capability development.

Capability testing and certification. Organizations might implement regular capability assessments independent of project performance—periodic exercises that evaluate independent skills without AI assistance. This parallels how pilots maintain certifications through regular simulator testing or how medical professionals complete continuing education: ongoing verification that core capabilities remain sharp despite tool evolution.

For knowledge workers, this might include quarterly problem-solving exercises, case analyses, or technical challenges completed under standardized conditions without AI access. Results would inform development planning and ensure individuals maintain transferable expertise even as they leverage AI for daily work.

Learning system design. Organizations can structure work to preserve learning opportunities rather than allowing AI to completely subsume educational aspects of projects. This might mean deliberately assigning some projects as "development focused" where junior staff work with mentorship but without AI assistance to build core skills, even if this reduces short-term efficiency.

Educational institutions are exploring similar principles through structured AI integration: certain courses or assignments permit full AI use to teach AI collaboration, while others are designated AI-free to ensure foundational capability development. Organizations can apply analogous approaches, recognizing that capability building sometimes requires deliberately constraining AI use to preserve learning opportunities.

Evaluation System Evolution and Innovation

As the LLM fallacy becomes better understood, evaluation practices will likely evolve substantially. Organizations pioneering innovative approaches may gain competitive advantages in identifying and developing genuine talent.

Multi-method assessment portfolios. Rather than relying on single evaluation approaches vulnerable to AI-mediated performance gaps, organizations can implement multi-method assessment combining:

• Work portfolio review (acknowledging potential AI contribution)

• Live problem-solving or case analysis (revealing independent reasoning)

• Detailed work process interviews (surfacing understanding depth)

• Peer/supervisor references focusing on collaboration and judgment (not just outputs)

• Longitudinal performance tracking (observing consistency across contexts)

Each method provides different information; triangulation across methods reduces individual assessment vulnerabilities. This approach parallels principles from psychological assessment where multiple measures increase validity and reduce method-specific biases.

AI-aware performance management. Performance evaluation systems can explicitly address AI use, asking managers to assess not whether employees use AI (which should be expected and encouraged for appropriate tasks) but how effectively they balance AI leverage with independent capability development and how accurately they understand their own capability boundaries.

Performance conversations might include questions like: "Where has AI assistance most enhanced your performance?" "What skills are you actively developing versus delegating to AI?" "How do you validate AI-generated work?" "Where would your performance decline substantially without AI access?" These conversations create shared understanding of capability profiles and development needs.

Predictive validity studies. Organizations can conduct internal research examining relationships between different assessment approaches and subsequent job performance. Which interview techniques best predict effectiveness in roles requiring complex judgment? How well do take-home assessments (where AI use is likely) predict on-the-job performance compared to live problem-solving exercises? Do AI-free assessment components add predictive value beyond AI-inclusive evaluations?

By systematically studying these questions, organizations can refine evaluation practices to maximize both fairness and validity—ensuring they identify candidates with genuine capability while appropriately crediting effective AI collaboration skills.

Conclusion

The LLM fallacy represents a fundamental challenge at the intersection of individual cognition and organizational evaluation. When AI assistance becomes sufficiently seamless and fluent, it creates systematic attribution ambiguity: individuals struggle to accurately assess where their capability ends and system contribution begins. This metacognitive challenge would be consequential enough as merely an individual phenomenon. Its implications magnify dramatically because organizations rely on output-based performance evaluation to make high-stakes decisions about hiring, promotion, compensation, and development.

The evidence is clear: individuals consistently overestimate independent capability after receiving AI assistance, conflating aided performance with transferable expertise. This isn't a deficit in individual character or intelligence but rather a predictable consequence of how human cognition operates under conditions of opacity, fluency, and attribution ambiguity. When AI contribution remains invisible and outputs appear indistinguishable from human-generated work, metacognitive calibration becomes systematically difficult.

For organizations, the implications are equally clear. Traditional evaluation frameworks—portfolio review, work sample assessment, demonstrated output evaluation—lose reliability when outputs increasingly reflect human–AI collaboration that remains invisible in final products. Hiring risks increase. Promotion decisions become less valid. Credential value erodes. These aren't hypothetical concerns but observable patterns emerging across software engineering, consulting, education, and professional services.

Yet the appropriate response isn't to resist AI integration or attempt to restore pre-AI evaluation practices. AI assistance delivers genuine productivity gains and enables work that wouldn't otherwise be possible. The goal isn't eliminating AI augmentation but rather developing organizational systems that maintain evaluation integrity and support capability development even as AI becomes deeply embedded in knowledge work.

This requires multi-level intervention. At the individual level, professionals need structured AI literacy and metacognitive training that builds realistic self-assessment. At the team level, mentorship practices and peer review systems must evolve to distinguish AI-augmented outputs from independent expertise. At the organizational level, evaluation frameworks must transition from output-centric models toward process-aware approaches that assess both AI orchestration skills and underlying human capabilities. At the cultural level, organizations must develop realistic norms around AI use, capability attribution, and professional development that balance AI leverage with genuine expertise building.

The most sophisticated organizations are already pioneering these approaches: implementing transparency mechanisms that document AI contribution, designing staged evaluations that vary AI access, developing competency frameworks that distinguish human judgment from augmentable execution, and creating deliberate practice contexts that preserve capability development opportunities. These innovations don't represent AI resistance but rather mature integration—recognizing that sustainable AI augmentation requires maintaining the human expertise that makes augmentation valuable.

Looking forward, the LLM fallacy challenges organizations to fundamentally rethink what competence means in hybrid human–AI systems, how expertise develops when cognitive work is distributed across humans and machines, and how evaluation systems can maintain validity when performance no longer straightforwardly indicates capability. Organizations that address these challenges effectively—balancing AI productivity gains with capability development, augmentation with assessment integrity, efficiency with learning—will build more sustainable competitive advantages than those that either resist AI integration or embrace it without attending to its implications for human expertise.

The deeper question the LLM fallacy raises is this: In a world where impressive outputs can be generated through AI collaboration without corresponding independent expertise, how do we preserve the capability development processes that created human expertise in the first place? If we allow AI to fully substitute for the effortful practice and productive struggle through which expertise develops, we risk creating a self-undermining system—one where increasing AI capability paradoxically reduces the human expertise needed to effectively direct, validate, and extend AI systems themselves.

Addressing the LLM fallacy thus becomes not merely a measurement or evaluation challenge but an existential question about how organizations preserve human capability development in an age of increasingly powerful AI augmentation. The organizations that answer this question most effectively—that find ways to leverage AI's benefits while maintaining robust human expertise development—will not only avoid the LLM fallacy's pitfalls but position themselves to thrive in the permanently AI-mediated future of knowledge work.

Research Infographic

The LLM Fallacy Slide Deck

References

1. Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235.

2. Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989.

3. Brynjolfsson, E., Li, D., & Raymond, L. R. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889–942.

4. Burrell, J. (2016). How the machine "thinks": Understanding opacity in machine learning algorithms. Big Data & Society, 3(1), 2053951715622512.

5. Clark, A. (2010). Supersizing the Mind: Embodiment, Action, and Cognitive Extension. Oxford University Press.

6. Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K. (2018). Common metrics to benchmark human-machine teams (HMT): A review. IEEE Access, 6, 38637–38655.

7. Dillon, E. W., Jaffe, S., Immorlica, N., & Stanton, C. T. (2025). Shifting work patterns with generative AI (Working Paper No. 33795). National Bureau of Economic Research.

8. Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024). The AI ghostwriter effect. ACM Transactions on Computer-Human Interaction, 31(2), 1–40.

9. Ericsson, K. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 683–703). Cambridge University Press.

10. Gajos, K. Z., & Mamykina, L. (2022). Do people engage cognitively with AI? Impact of AI assistance on incidental learning. In Proceedings of the 27th International Conference on Intelligent User Interfaces (pp. 794–806). ACM.

11. Hutchins, E. (1995). Cognition in the Wild. MIT Press.

12. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38.

13. Karny, S., Mayer, L. W., Ayoub, J., Song, M., Su, H., Tian, D., Karny, M., & Steyvers, M. (2024). Learning with AI assistance. In Proceedings of the ACM Collective Intelligence Conference. ACM.

14. Kim, H., Yu, H., & Yi, H. (2026). The LLM fallacy: Misattribution in AI-assisted cognitive workflows. arXiv preprint arXiv:2604.14807.

15. Kirsh, D. (2010). Thinking with external representations. AI & Society, 25(4), 441–454.

16. Kocielnik, R., Amershi, S., & Bennett, P. N. (2019). Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–14). ACM.

17. Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349–370.

18. Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–15). ACM.

19. Nam, D., Macvean, A., Hellendoorn, V. J., Vasilescu, B., & Myers, B. A. (2024). Using an LLM to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (pp. 1–13). IEEE.

20. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259.

21. Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688.

22. Rozenblit, L., & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive Science, 26(5), 521–562.

23. Shneiderman, B. (2020). Human-centered artificial intelligence: Three fresh ideas. AIS Transactions on Human-Computer Interaction, 12(3), 109–124.

24. Zhang, Z., Wang, C., Wang, Y., Shi, E., Ma, Y., Zhong, W., Chen, J., Mao, M., & Zheng, Z. (2025). LLM hallucinations in practical code generation: Phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering, 2(ISSTA), 481–503.

---

Jonathan H. Westover, PhD is Chief Research Officer (Nexus Institute for Work and AI); Associate Dean and Director of HR Academic Programs (WGU); Professor, Organizational Leadership (UVU); OD/HR/Leadership Consultant (Human Capital Innovations). Read Jonathan Westover's executive profile here.

---

Suggested Citation: Westover, J. H. (2026). When AI Assistance Becomes Invisible: Organizational Challenges of Competence Illusion in the Age of Generative AI. Human Capital Leadership Review, 36(2). doi.org/10.70175/hclreview.2020.36.2.4