Mind the Gap: Why Understanding AI Error Boundaries Is the Key to Unlocking Human-AI Team Performance
Download the article here:
Listen to a Review of this Article:
Abstract: Organizations increasingly deploy artificial intelligence to augment human decision-making in high-stakes domains, yet mounting evidence reveals that AI accuracy alone does not reliably translate into superior human-AI team outcomes. This article examines the critical but underexplored role of human mental models—specifically, users' understanding of when and where an AI system errs—in shaping the effectiveness of AI-advised decision-making. Drawing on foundational experimental research by Bansal, Nushi, Kamar, Lasecki, et al. (2019), the article unpacks three properties of AI systems and tasks—error boundary parsimony, stochasticity, and task dimensionality—that determine how readily humans learn to complement an AI teammate. Evidence-based organizational responses are presented, spanning system design, explainability strategy, update governance, and workforce development. The article concludes with forward-looking pillars for building durable human-AI collaboration capability, arguing that practitioners must optimize not only for what the AI gets right, but for how predictably humans can learn what it gets wrong.
There is a seductive simplicity to the way most organizations evaluate their AI investments: accuracy goes up, therefore value goes up. A diagnostic model improves from 90% to 94%, and leadership assumes clinical outcomes will improve proportionally. A fraud-detection engine tightens its precision score, and the risk committee declares victory. But anyone who has watched a skilled radiologist ignore a correct AI flag—or, worse, accept an incorrect one—knows that the relationship between model performance and team performance is far messier than a leaderboard metric suggests.
The gap between AI accuracy and real-world team outcomes is not merely a deployment nuisance; it is a fundamental design challenge. Bansal, Nushi, Kamar, Lasecki, et al. (2019) articulated this challenge with unusual clarity in their landmark paper at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP). Their central argument was that the success of AI-advised human decision-making depends critically on the human's mental model of the AI's capabilities—specifically, the human's ability to learn the AI's error boundary, the pattern that distinguishes where the system succeeds from where it fails. Two AI systems of identical accuracy can produce dramatically different team outcomes depending on whether the human partner can discern and internalize that error pattern.
The practical stakes are enormous. In healthcare, a misaligned mental model means a physician trusts a readmission classifier precisely when it misfires, sending a high-risk patient home (Caruana et al., 2015). In criminal justice, it means a judge defers to a recidivism predictor in the very population subgroup where the tool performs worst (Angwin et al., 2016). In finance, it means a loan officer rubber-stamps an AI recommendation when the model has drifted outside its training distribution. In each case, the human-AI team performs worse than either party would have alone—not because the AI is inaccurate on average, but because the human cannot tell when it is inaccurate.
This article translates the research program initiated by Bansal, Nushi, Kamar, Lasecki, et al. (2019)—and extended by a growing body of complementary work—into a practitioner-oriented guide. It unpacks the science behind mental models of AI error, identifies design and organizational interventions supported by evidence, and outlines a forward-looking agenda for building lasting human-AI collaboration capability.
The Human-AI Collaboration Landscape
Defining Mental Models in the Context of AI-Advised Decision-Making
A mental model, in the cognitive science tradition, is an internal representation that a person uses to reason about and predict the behavior of a system they interact with (Norman, 1988). When humans collaborate with AI systems, they inevitably construct such models—sometimes consciously, often implicitly—about what the AI can and cannot do (Kulesza et al., 2012).
In the specific setting of AI-advised human decision-making, where an AI system provides a recommendation and a human makes the final call, the most consequential mental model concerns the AI's error boundary. Bansal, Nushi, Kamar, Lasecki, et al. (2019) formally defined the error boundary as "a function f that describes for each input x whether model output h(x) is the correct action for that input" (p. 3). In practical terms, a doctor working with a patient readmission classifier does not need to understand every weight in a neural network; she needs to understand under what patient profiles the classifier tends to get things wrong—and under what profiles she can trust it.
This framing shifts the emphasis away from global trust or distrust in AI toward situational discernment: the ability to appropriately accept AI recommendations when they are correct and override them when they are not. Lee and See (2004), in their foundational review of trust in automation, called this "appropriate reliance"—the calibration of trust to the actual capabilities of the automated system. Without an accurate mental model of the error boundary, appropriate reliance becomes a matter of luck rather than skill.
Prevalence, Drivers, and the State of Practice
The structure Bansal, Nushi, Kamar, Lasecki, et al. (2019) described—human receives AI recommendation, decides to accept or override, receives feedback—is now the dominant paradigm for deploying AI in high-stakes settings. It appears in clinical decision support systems where physicians use algorithmic risk scores (Caruana et al., 2015), in surgical environments where anesthesiologists receive real-time hypoxaemia predictions (Lundberg et al., 2018), in courtrooms where judges consult recidivism risk tools (Angwin et al., 2016), and in lending where credit analysts review automated scoring outputs.
Yet despite the ubiquity of this pattern, the overwhelming majority of AI development still optimizes exclusively for model accuracy (Bansal, Nushi, Kamar, Lasecki, et al., 2019). Model selection processes compare candidates on area under the curve, F1 scores, or mean absolute error, with little attention to whether one model's error pattern is more learnable by the human who will partner with it. As Grosz (1996) argued decades ago, successful collaborative systems demand capabilities beyond raw task performance—they require the ability to support mutual understanding between team members.
The consequences of ignoring this mismatch have been documented empirically. Yin et al. (2019) found that stated accuracy alone has an uneven and sometimes counterproductive effect on human trust. Lai and Tan (2019) showed that AI predictions with explanations did not uniformly improve human decision-making. And Bansal, Nushi, Kamar, Weld, et al. (2019) demonstrated, in a companion study, that updating an AI to a more accurate model can actually degrade team performance when the update breaks the human's existing mental model—a phenomenon they termed the performance/compatibility tradeoff.
Later work sharpened the point. Bansal et al. (2021) studied what they called complementary performance—the goal that a human-AI team should outperform both the human alone and the AI alone. Across several decision tasks, they found that AI explanations tended to increase how often people accepted the AI's recommendations whether or not those recommendations were correct, and that explanations did not reliably improve team accuracy beyond what was achieved by simply showing the AI's confidence. In other words, the most common industry response to the trust problem—add an explanation—does not automatically produce the discernment that teams need.
These findings collectively paint a picture of an industry that has mastered the engineering of individual AI performance but has barely begun the engineering of team performance.
Organizational and Individual Consequences of Misaligned Mental Models
Organizational Performance Impacts
When a human decision-maker holds an inaccurate mental model of an AI's error boundary, two types of costly mistakes become likely. First, the human may accept the AI's recommendation when it is wrong—what Bansal, Nushi, Kamar, Lasecki, et al. (2019) termed a "false accept." Second, the human may override the AI when it is actually correct—a "false override." Both degrade team performance, but asymmetrically: in high-stakes settings, the cost of a false accept typically far exceeds the opportunity cost of a false override.
These two error types map onto a distinction human factors researchers drew long before machine learning entered the workplace. Parasuraman and Riley (1997) described misuse of automation as overreliance—operators trusting a system in conditions it cannot handle—and disuse as the neglect or underuse of automation that would have helped. They also named a third failure, abuse, which occurs when designers automate functions without adequately considering the consequences for the people who must work alongside the system. Each maps cleanly onto AI-advised decision-making. False accepts are misuse. False overrides, multiplied across a workforce, become disuse. And deploying a model selected purely for accuracy, without asking whether its errors can be learned by the people who must catch them, is a modern form of what Parasuraman and Riley called abuse.
In the controlled experiments reported by Bansal, Nushi, Kamar, Lasecki, et al. (2019), this asymmetry was deliberately built into the payoff structure: participants earned $0.04 for correctly accepting the AI but lost $0.16 for accepting a wrong recommendation—a 4:1 penalty ratio that mirrors the cost asymmetries common in healthcare, fraud detection, and safety-critical operations. The research found that participants with incorrect mental models earned substantially lower cumulative rewards, particularly in the early rounds of interaction when their models were still forming. Crucially, participants' performance improved over time as they refined their mental models, converging toward an optimal policy—but the rate and ceiling of that convergence depended heavily on properties of the AI's error boundary.
At the organizational level, these individual-level dynamics compound. An enterprise deploying an AI model whose error boundary is difficult for frontline workers to learn will experience chronically suboptimal team performance, even if the model's standalone accuracy is excellent. Meanwhile, a competitor deploying a slightly less accurate model with a more learnable error boundary may see superior operational outcomes because its human operators can reliably compensate for the AI's weaknesses.
Individual Wellbeing and Decision-Maker Impacts
Beyond performance metrics, misaligned mental models impose a cognitive and emotional toll on the humans in the loop. When an AI system's errors appear unpredictable—when it seems to fail randomly rather than in a discernible pattern—the experience can be deeply disorienting. The concept of stochasticity in error boundaries, as articulated by Bansal, Nushi, Kamar, Lasecki, et al. (2019), directly maps to this experience: a stochastic error boundary means the AI sometimes fails even in the regions where it usually succeeds, making it difficult for the human to develop a reliable internal rule for when to trust it.
The experimental data vividly illustrate this effect. In Bansal, Nushi, Kamar, Lasecki, et al.'s (2019) study of one-sided versus two-sided stochastic error boundaries, participants facing two-sided stochasticity (where errors could occur on both sides of the boundary, albeit with different probabilities) showed near-zero improvement over 100 rounds of interaction. Their behavior remained close to 50% accuracy—effectively coin-flipping—because the inconsistency of the AI's errors prevented them from forming any stable mental model. In contrast, participants facing non-stochastic boundaries achieved near-optimal performance over the same period.
For professionals in high-stakes domains, this experience of persistent uncertainty about an AI tool's reliability can erode confidence, increase decision fatigue, and contribute to automation distrust—a withdrawal from AI tools altogether that Lee and See (2004) identified as one of the core failure modes of human-automation collaboration.
The behavioral research on how people respond to algorithmic advice helps explain why this withdrawal can be so stubborn. Dietvorst et al. (2015) found that people who watched an algorithm make mistakes became less likely to choose it over a human forecaster, even when the algorithm clearly outperformed the human—a pattern they called algorithm aversion. Seeing a machine err seems to damage confidence in it more than seeing a person make the same mistake. Yet the picture is not one-directional. Logg et al. (2019) found that in many judgment tasks, lay participants actually gave more weight to advice when it was attributed to an algorithm than to a person, although experienced forecasters were more inclined to discount it.
Taken together, these findings suggest that people do not hold a single, stable attitude toward AI. They swing—sometimes deferring too readily, sometimes abandoning a useful tool after a few visible failures. An accurate mental model of the error boundary is what keeps that swing from happening. When a professional knows where the AI tends to fail, an observed error confirms her model rather than shattering her trust; she can keep relying on the system in its zone of competence while scrutinizing it outside that zone. Without that model, each visible error invites a global verdict—"this tool can't be trusted"—that throws away the value the AI provides in the cases it handles well.
Evidence-Based Organizational Responses
Table 1: Human-AI Collaboration Systems and Case Studies
Organization / System | Application Domain | AI Tool or Platform | Mental Model / Error Challenge | Design or Organizational Intervention | Key Outcomes and Findings |
Microsoft Research / CAJA Platform | Research Testbed (Assembly-line scenario) | Open-source, game-like testbed for human-AI interaction experiments | Understanding AI error boundary properties, including parsimony, stochasticity, and task dimensionality | Configurable testbed platform making error-boundary properties experimentally manipulable to pre-test mental model formation | Enabled systematic isolation and study of parsimony, stochasticity, and dimensionality; showed that simpler/non-stochastic boundaries substantially improve team accuracy and learning |
Seattle Children's Hospital | Surgery / Healthcare | Explainable machine-learning system for predicting hypoxaemia events during surgery | Managing error boundary stochasticity and establishing appropriate clinician reliance/calibration per patient instance | Real-time feature-importance explanations using SHAP values provided to anesthesiologists at the individual-patient level | Provided clinicians with a basis to judge whether model reasoning was sound for a given instance, facilitating calibrated trust and decision support |
Northpointe (now Equivant) / COMPAS | Criminal Justice | COMPAS recidivism prediction tool | High-dimensional, opaque error boundaries leading to misaligned mental models and systematic racial disparities in error rates | Algorithmic risk scoring presented to judicial decision-makers (evaluated critically by independent investigation) | Judges were unable to discern error boundaries across demographic subgroups, preventing appropriate reliance and leading to inequitable human-AI team decision-making |
Healthcare Prediction Systems (Caruana et al., 2015) | Healthcare (Pneumonia Risk & Hospital Readmission) | Generalized additive models vs. Neural Networks | Opaque neural networks learning dangerously misleading associations (e.g., asthma linked to lower pneumonia mortality risk due to treatment artifacts) that humans cannot anticipate | Transition to intelligible, interpretable models (generalized additive models) to make error boundaries explicitly visible | Enabled clinicians to recognize and correct for erroneous model assumptions, improving patient safety and model trustworthiness |
Designing AI Systems With Parsimonious Error Boundaries
The most distinctive recommendation to emerge from Bansal, Nushi, Kamar, Lasecki, et al.'s (2019) research is also the most counterintuitive from a traditional machine learning perspective: when selecting among models for deployment in human-AI teams, practitioners should prefer models whose error boundaries are parsimonious—simple enough for a human partner to learn—even if this means accepting a slight reduction in raw accuracy.
Parsimony, in this context, refers to the representational complexity of the error boundary. An error boundary that can be described with a small number of features and conjunctive rules (e.g., "the model errs on elderly patients with high blood pressure") is more parsimonious than one requiring multiple disjunctive conditions across many features. In their experiments, Bansal, Nushi, Kamar, Lasecki, et al. (2019) found that team performance decreased significantly as the number of conjunctions in the error boundary increased, even when model accuracy was held constant. A single-conjunction error boundary (e.g., "red and square") produced markedly higher team scores than a two-conjunction boundary (e.g., "red and square or small and circle").
Effective approaches include:
Evaluate candidate models on error-boundary complexity, not just accuracy: During hyperparameter tuning and model selection, teams should characterize the error boundary of each candidate—using tools such as decision trees fit to the model's error pattern—and favor models whose error profiles can be stated in fewer rules.
Use regularization techniques that promote simpler error regions: Techniques like tree-based regularization (Wu et al., 2018) or constraints that penalize error-boundary complexity can steer training toward models that are easier for humans to partner with.
Communicate error boundaries explicitly during onboarding: Rather than telling users only the model's overall accuracy, provide concise descriptions of the conditions under which it tends to fail.
Microsoft Research developed the CAJA platform—the open-source, game-like testbed used in the Bansal, Nushi, Kamar, Lasecki, et al. (2019) experiments—specifically to enable systematic study of how humans learn AI error boundaries under controlled conditions. By abstracting AI-advised decision-making into a configurable assembly-line scenario, CAJA allowed researchers to isolate the effects of parsimony, stochasticity, and dimensionality on mental model formation. The platform's design philosophy—making error-boundary properties experimentally manipulable—offers a template for organizations seeking to pre-test how learnable a model's failure patterns will be before committing to deployment.
Minimizing Stochasticity in System Errors
If parsimony determines how simple the error boundary is to describe, stochasticity determines how reliably that description holds. A non-stochastic error boundary means the AI always errs in the specified region and always succeeds outside it. A stochastic boundary introduces probabilistic noise: the AI sometimes gets things right even in its weak zone, or sometimes gets things wrong in its strong zone. This noise is deeply corrosive to mental model formation.
Bansal, Nushi, Kamar, Lasecki, et al. (2019) demonstrated this experimentally by varying the conditional probability of error inside and outside the error boundary formula. When errors were deterministic (the AI always failed on items matching the boundary condition), participants steadily improved their override accuracy, with near-100% of workers choosing the optimal action by the end of the session. When errors were stochastic—particularly when stochasticity was two-sided—learning stalled, and team performance remained poor throughout.
Effective approaches include:
Prefer models with deterministic or near-deterministic failure patterns: When two models have comparable accuracy but different stochasticity profiles, the one with more predictable failures will yield better team performance.
Identify and manage sources of stochasticity: Bansal, Nushi, Kamar, Lasecki, et al. (2019) identified three common sources: generalization trade-offs (where reducing model complexity introduces probabilistic errors near decision boundaries), representation mismatch (where the human and AI see different feature spaces), and inherent outcome stochasticity. Each source calls for different mitigation strategies.
Communicate uncertainty at the instance level: Well-calibrated confidence scores, when they faithfully reflect the probability of error, can partially compensate for stochasticity by giving the human a decision-relevant signal (Guo et al., 2017).
University of Washington researchers and anesthesiologists, in work reported by Lundberg et al. (2018), developed and tested an explainable machine-learning system, called Prescience, to predict hypoxaemia (low blood oxygen) during surgery using operating-room records from University of Washington hospitals. A key design choice was to give anesthesiologists feature-level explanations (based on SHAP values) showing which patient and procedure factors were driving each risk prediction. In the study's evaluation, anesthesiologists anticipated hypoxaemia events more accurately when they had the system's predictions and explanations than when they relied on their own judgment alone. The approach speaks directly to the stochasticity challenge: by showing why the model predicted high or low risk for each case, the system gave clinicians a basis for judging whether the model's reasoning was sound in that particular instance—helping them calibrate when to trust and when to question.
Managing Task Dimensionality and Feature Presentation
Even an AI system with a parsimonious, non-stochastic error boundary can be difficult to learn if the task itself presents an overwhelming number of features to the human decision-maker. Bansal, Nushi, Kamar, Lasecki, et al. (2019) found that team performance generally decreased as the number of human-visible features increased, consistent with prior findings on human reasoning under information overload (Poursabzi-Sangdeh et al., 2021).
This result has a sharp practical implication: adding features to a model that improve its standalone accuracy may actually degrade team performance if those additional features make it harder for the human partner to identify the error pattern.
Effective approaches include:
Analyze the marginal value of each feature for both machine and human reasoning: Bansal, Nushi, Kamar, Lasecki, et al. (2019) recommended a trade-off analysis: weigh the marginal gain in model performance from adding a feature against the marginal loss in the human's ability to form an accurate mental model.
Curate the information presented to human decision-makers: The features the AI uses internally need not all be shown to the user. Interface design should prioritize the features most relevant to understanding the error boundary.
Use progressive disclosure to manage cognitive load: Start with a concise set of high-salience features and allow the user to drill down into additional detail as needed.
Northpointe (now Equivant) developed the COMPAS recidivism prediction tool used in criminal sentencing decisions across the United States. The tool generates risk scores based on a large feature set, and its deployment became the subject of a landmark investigation by ProPublica (Angwin et al., 2016) that revealed systematic racial disparities in error rates. From the mental-models perspective, COMPAS illustrates the danger of high-dimensional, opaque error boundaries: judges using the tool had limited ability to discern where the model's error boundary fell along racial, geographic, and socioeconomic lines, making appropriate reliance effectively impossible. The case underscored what Bansal, Nushi, Kamar, Lasecki, et al. (2019) formalized—that the human's ability to learn an error boundary is a prerequisite for equitable and effective AI-advised decision-making.
Governing Model Updates for Backward Compatibility
One of the most practically significant extensions of the mental-models framework came from the companion study by Bansal, Nushi, Kamar, Weld, et al. (2019), which introduced the concept of backward compatibility in AI model updates. The core finding was that deploying a more accurate model—when the new model introduces errors on instances where the previous version was correct—can break the human's existing mental model and degrade team performance, even as standalone model accuracy improves.
This means the common practice of "hot-swapping" a model with its improved successor—without attending to how the error boundary has shifted—carries real risk in human-AI team settings.
Effective approaches include:
Measure backward compatibility during model updates: Quantify the proportion of inputs where the old model was correct that remain correct in the new model. Introduce new errors sparingly, especially in regions where users have already learned to trust the system.
Use compatibility-aware loss functions during training: Bansal, Nushi, Kamar, Weld, et al. (2019) demonstrated that adding a backward-compatibility term to the training objective can regularize model updates to preserve existing trust relationships.
Stage rollouts and provide transition support: When error boundary shifts are unavoidable, provide users with explicit guidance about what has changed—where the new model's strengths and weaknesses differ from the old one.
Healthcare prediction systems offer a cautionary illustration. Caruana et al. (2015) documented the development of intelligible models for predicting pneumonia risk and hospital readmission, noting that a neural network model—while highly accurate on average—had learned a dangerously misleading association between asthma and reduced pneumonia mortality risk (reflecting a data artifact: asthma patients received more aggressive treatment). An interpretable model (a generalized additive model) made this error boundary visible, enabling clinicians to recognize and correct for it. The episode highlights both the value of transparent error boundaries and the risk that model updates or opaque architectures can introduce errors that no mental model can anticipate.
Investing in Explainability That Reveals Error Boundaries
The explainable AI (XAI) movement has produced a rich toolkit—LIME, SHAP, attention maps, prototype-based reasoning—for illuminating how a model reaches a prediction. But Bansal, Nushi, Kamar, Lasecki, et al. (2019) drew an important distinction that practitioners often overlook: understanding how a system works is not the same as understanding when it will fail. Explanations of individual predictions can help with the former but may not efficiently build the latter.
Effective approaches include:
Develop and deploy error-boundary explanations: As Nushi et al. (2018) proposed, visualization tools that explicitly depict the conditions under which a model errs—rather than just the features driving its predictions—can accelerate mental model formation.
Combine instance-level and population-level explanations: Instance-level explanations (e.g., SHAP values for a specific patient) help with immediate decisions; population-level error summaries help build durable mental models over time.
Evaluate explanations by their effect on team performance, not just user satisfaction: An explanation users like is not necessarily one that helps them override correctly. Amershi et al. (2019) emphasized in their guidelines for human-AI interaction that setting appropriate expectations about system capabilities is more important than maximizing initial user enthusiasm.
Build deliberate friction into the highest-stakes decisions: Buçinca et al. (2021) tested cognitive forcing functions—design features such as asking people to record their own judgment before seeing the AI's recommendation—and found that these designs reduced overreliance on incorrect AI advice more effectively than simple explanations did. The trade-off was that participants liked the forcing designs less, so organizations should reserve them for decisions where a false accept is costly.
Kocielnik et al. (2019) found that how developers frame AI system expectations significantly influences user behavior—with designs that set realistic expectations outperforming those that emphasize high accuracy. Their work reinforces Bansal, Nushi, Kamar, Lasecki, et al.'s (2019) broader point: the interface layer between human and AI must be engineered for calibrated trust, not maximum trust.
The tension Buçinca et al. (2021) uncovered deserves attention from anyone procuring or designing AI tools. The interfaces that users rate most favorably are often the ones that make acceptance easiest, and easy acceptance is exactly what produces false accepts. Bansal et al. (2021) observed a related dynamic: explanations increased acceptance across the board, which helps when the AI is right and hurts when it is wrong. A practical implication is that pilot evaluations of AI tools should measure override accuracy—how often users correctly reject wrong recommendations—rather than relying on satisfaction surveys or adoption rates as the primary signal of success.
Building Long-Term Human-AI Collaboration Capability
Recalibrating the Psychological Contract Between Humans and AI
The deployment of AI advisory systems fundamentally alters the psychological contract of professional work. A physician who has spent decades honing diagnostic intuition is now asked to weigh that intuition against algorithmic recommendations—and to do so thoughtfully, case by case, under time pressure. Mohammed et al. (2010), in their review of team mental model research, showed that teams perform best when members hold compatible models of both the task and each other's roles. In human-AI teams, this means the human must internalize not just the task domain but also the AI's role—including its specific failure modes.
Building this internalization requires ongoing organizational investment. Initial training sessions are insufficient. Rouse et al. (1992) demonstrated that mental models in complex systems are refined through experience and feedback over time—and Bansal, Nushi, Kamar, Lasecki, et al.'s (2019) experimental data confirmed this, showing that participants' performance converged toward optimal only after dozens of interactions with feedback. Organizations that treat AI onboarding as a one-time event, rather than a continuous calibration process, will see chronic under-performance in their human-AI workflows.
Recalibration also involves preserving a meaningful sense of professional agency. Dietvorst et al. (2018) found that people were considerably more willing to use an imperfect algorithm when they were allowed to adjust its forecasts, even when the permitted adjustments were small. That finding carries a useful lesson for workflow design. Professionals are more likely to engage thoughtfully with AI recommendations when the system positions them as active judges rather than as rubber stamps. Workflows that frame the human role as confirming the AI's output tend to invite either passive acceptance or quiet resistance; workflows that invite the human to weigh, adjust, and record reasoning tend to support the kind of active error-boundary learning that Bansal, Nushi, Kamar, Lasecki, et al. (2019) showed is necessary for strong team performance.
Practical steps include incorporating error-boundary review into regular professional development, creating "red team" exercises where practitioners deliberately practice identifying AI failures, and establishing feedback loops that surface real-world mismatches between user mental models and actual model performance.
Institutionalizing Human-Aware Model Selection
Bansal, Nushi, Kamar, Lasecki, et al. (2019) proposed that parsimony, non-stochasticity, and dimensionality should become standard criteria in model selection—alongside accuracy, fairness, and latency. Operationalizing this vision requires new institutional processes.
First, organizations need human-aware evaluation metrics that capture error-boundary properties. This might involve fitting an interpretable surrogate (e.g., a shallow decision tree) to a model's error pattern and reporting the tree's complexity as a proxy for parsimony. It might involve computing the entropy of errors conditioned on feature regions as a measure of stochasticity.
Second, model selection committees should include members who represent the end-user perspective—clinicians in healthcare, analysts in finance, case workers in social services—and who can assess, at least qualitatively, whether an error boundary is likely to be learnable by frontline decision-makers.
Third, the trade-off analysis Bansal, Nushi, Kamar, Lasecki, et al. (2019) described—weighing the marginal gain in model accuracy against the marginal loss in mental-model accuracy—should become an explicit step in deployment review. Kamar (2016) argued that the direction of hybrid intelligence requires considering how human and machine capabilities complement each other; human-aware model selection is a concrete mechanism for realizing that vision.
Fourth, procurement deserves the same scrutiny as internal development. Many organizations will never train their own models; they will buy them. Vendor evaluations, however, still tend to center on headline accuracy figures and feature checklists. Buyers can ask sharper questions: Under what conditions does the system perform worst? Can the vendor describe those conditions in a few plain-language rules? How consistent are the system's failures within those conditions? What changes in the error pattern should users expect when the vendor ships an update? A vendor that cannot answer these questions is, in effect, asking frontline staff to discover the error boundary on the job, one costly mistake at a time.
None of this requires abandoning accuracy as a goal. The point is that accuracy is one input into a broader judgment about which system will make the team most effective. Amershi et al. (2019) framed many of their guidelines around making clear what a system can do and how well it can do it; human-aware model selection extends that principle upstream, into the decisions about which system should reach users in the first place.
Designing Continuous Learning Systems for Human-AI Teams
The most resilient human-AI teams will not be those that achieve the perfect mental model once and freeze it, but those that build organizational capacity for continuous mutual adaptation. This means investing in systems and processes that detect when mental models have drifted out of calibration—due to model updates, distribution shifts, or personnel changes—and trigger corrective interventions.
Andrews et al. (2023), in their theoretical review of shared mental models in human-AI teams, highlighted that the measurement and maintenance of shared understanding between human and AI teammates remains an open research challenge, but one with clear practical urgency. Their framework suggests that organizations should monitor not only model performance metrics but also behavioral indicators of mental-model quality—such as override rates, time-to-decision, and the correlation between human confidence and decision correctness.
Technology-enabled support tools can play a role here. Bansal, Nushi, Kamar, Lasecki, et al. (2019) speculated that tools such as real-time scratchpads that record and summarize observed error patterns could help humans manage more complex error boundaries than they could hold in memory alone. As AI systems grow in capability and are deployed in higher-dimensional task spaces, such cognitive scaffolding will become indispensable.
Finally, organizations should cultivate a culture that treats human-AI collaboration as a skill to be developed, not a static configuration to be installed. The evidence is clear that mental models improve with interaction and feedback (Bansal, Nushi, Kamar, Lasecki, et al., 2019). The organizational question is whether the structures exist to ensure that improvement is supported, accelerated, and sustained.
Conclusion
The research program initiated by Bansal, Nushi, Kamar, Lasecki, et al. (2019) delivers a message that is both simple and profound: accuracy is necessary but insufficient for effective human-AI collaboration. What matters for team performance is not just how often the AI is right, but how predictably the human can learn when it is wrong. Three properties—the parsimony and non-stochasticity of the AI's error boundary, and the dimensionality of the task—determine the learnability of that error pattern and, consequently, the ceiling on team performance.
For practitioners, the actionable takeaways are direct. First, incorporate error-boundary properties into model selection: when two models are comparably accurate, choose the one whose failure pattern is simpler and more deterministic. Second, manage model updates with care, recognizing that a more accurate model can harm team performance if it disrupts the mental models users have already built. Third, invest in explainability tools that illuminate not just how the AI reasons, but where it fails. Fourth, design interfaces that reduce task dimensionality for the human partner, curating the information presented to support error-boundary learning. Fifth, treat human-AI calibration as an ongoing organizational process, not a one-time deployment event.
The broader implication is that the unit of analysis for AI deployment must shift from the model to the team. Optimizing for model performance in isolation is like optimizing one player on a doubles tennis team—it helps, but only if the partners can coordinate. Building that coordination requires a fundamental expansion of what we measure, what we optimize, and what we value when we design, select, and deploy AI systems for human-centered work.
Research Infographic
References
Amershi, S., Weld, D. S., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S. T., Bennett, P. N., Quinn, K. I., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13.
Andrews, R. W., Lilly, J. M., Srivastava, D., & Feigh, K. M. (2023). The role of shared mental models in human-AI teams: A theoretical review. Theoretical Issues in Ergonomics Science, 24(2), 129–175.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica.
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019). Beyond accuracy: The role of mental models in human-AI team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 2–11.
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019). Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 33, 2429–2437.
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–16.
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21.
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–1730.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155–1170.
Grosz, B. J. (1996). Collaborative systems (AAAI-94 presidential address). AI Magazine, 17(2), 67–85.
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 1321–1330.
Kamar, E. (2016). Directions in hybrid intelligence: Complementing AI systems with human intelligence. Proceedings of the 25th International Joint Conference on Artificial Intelligence, 4070–4073.
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019). Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14.
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012). Tell me more? The effects of mental model soundness on personalizing an intelligent agent. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1–10.
Lai, V., & Tan, C. (2019). On human predictions with explanations and predictions of machine learning models: A case study on deception detection. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 29–38.
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103.
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., & others. (2018). Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering, 2(10), 749–760.
Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 36(4), 876–910.
Norman, D. (1988). The psychology of everyday things. Basic Books.
Nushi, B., Kamar, E., & Horvitz, E. (2018). Towards accountable AI: Hybrid human-machine analyses for characterizing system failure. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6(1), 126–135.
Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2021). Manipulating and measuring model interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–52.
Rouse, W. B., Cannon-Bowers, J. A., & Salas, E. (1992). The role of mental models in team performance in complex systems. IEEE Transactions on Systems, Man, and Cybernetics, 22(6), 1296–1308.
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018). Beyond sparsity: Tree regularization of deep models for interpretability. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 1670–1678.
Yin, M., Vaughan, J. W., & Wallach, H. (2019). Understanding the effect of accuracy on trust in machine learning models. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–12.
Jonathan H. Westover, PhD, Chief Research Officer (Nexus Institute for Work and AI); Co-Founder & Chief Workforce and Learning Officer (Future State University); Founder & CEO (Human Capital Innovations); Professor of Organizational Leadership & Change (UVU). Read Jonathan Westover's executive profile here.
Suggested Citation: Westover, J. H. (2026). The Influence of Organizational Climate on Organizational Citizenship Behavior among Millennial Employees with Work-Life Balance as Moderator.. Human Capital Leadership Review, 39(3). doi.org/10.70175/hclreview.2020.39.3.4
