Designing for Predictability: How Mental Models of AI Error Boundaries Shape Human-AI Team Performance in High-Stakes Decision Making
Download the article here:
Listen to a review of this article:
Abstract: As artificial intelligence systems are increasingly deployed to advise human decision-makers in high-stakes domains—including healthcare, criminal justice, and financial services—organizations face a critical but often overlooked challenge: the accuracy of an AI system alone does not determine the performance of the human-AI team. This article examines the role of human mental models of AI capabilities, specifically mental models of AI error boundaries, in shaping collaborative decision-making outcomes. Drawing on foundational experimental research demonstrating that properties such as the parsimony and stochasticity of an AI's error boundary significantly influence whether humans can learn when to trust or override AI recommendations, this article translates those findings into actionable organizational strategies. Evidence-based interventions are presented across interface design, model selection, training protocols, and governance structures, illustrated with examples from healthcare, surgery, criminal justice, and AI research and development. Forward-looking pillars for building long-term human-AI collaboration capability—including psychological contract recalibration, continuous learning systems, and distributed oversight—are proposed for practitioners seeking to maximize the return on AI-augmented decision systems.
There is a quiet assumption running through most AI deployment strategies: if we build a more accurate model, decision outcomes will improve. It sounds reasonable. And it is incomplete in ways that can be dangerous.
Imagine a hospital deploying an AI system to predict which patients are at high risk of 30-day readmission. The model performs impressively in laboratory testing. Clinicians are told to consult its recommendations when deciding which patients to enroll in a costly but effective outpatient support program. Months later, leaders discover that the decisions made by clinicians using the tool are barely better than before, and in some cases worse. How could a highly accurate model fail to improve decisions?
The answer, increasingly supported by research, is that accuracy is a necessary but insufficient condition for effective human-AI teaming. What matters just as much—sometimes more—is whether the human decision-maker can build an accurate mental model of the AI: a working understanding of when the system gets things right and, critically, when it gets things wrong (Bansal, Nushi, Kamar, Lasecki, et al., 2019). Without this understanding, the human cannot perform the one job that justifies keeping them in the loop: knowing when to trust the AI and when to override it.
This matters now more than ever. Organizations across healthcare, financial services, criminal justice, and talent management are deploying AI-advised decision-making systems at scale. Regulatory frameworks increasingly require meaningful human oversight of high-risk AI systems; the European Union's AI Act, for example, requires that such systems be designed so that the people overseeing them can understand their capacities and limitations, remain aware of the tendency to over-rely on their output, and decide when to disregard or override it (Regulation [EU] 2024/1689, 2024). Yet "meaningful oversight" is hollow if the humans providing it cannot distinguish the AI's zones of competence from its zones of failure.
This article draws on the foundational experimental work of Bansal, Nushi, Kamar, Lasecki, et al. (2019) and related research to offer practitioners a rigorous yet accessible guide to designing AI systems—and the organizational ecosystems around them—that support effective mental model formation. The practical stakes are high: get this right, and human-AI teams have the best chance of outperforming either humans or AI working alone; get it wrong, and organizations may be worse off than if they had never deployed the AI at all.
The Human-AI Collaboration Landscape
Defining Mental Models and Error Boundaries in AI-Advised Decision Making
The concept of a mental model has deep roots in cognitive science and human-computer interaction. Norman (1988) established that people naturally construct internal representations of the systems they interact with, using these models to predict system behavior and guide their own actions. In the context of human-AI collaboration, Kulesza et al. (2012) demonstrated that the soundness of a user's mental model of an intelligent agent affects how effectively they can work with that agent.
Bansal, Nushi, Kamar, Lasecki, et al. (2019) brought this concept into sharp focus for AI-advised decision making by introducing the notion of an AI's error boundary. In their formulation, the error boundary is a function that describes, for each input, whether the AI model's output is correct. This is distinct from the model's decision boundary, which describes what the model predicts. The error boundary answers a more operationally useful question: "For what kinds of inputs does this AI system make mistakes?"
A human's mental model of the error boundary, then, is their internalized understanding of which situations the AI handles well and which it does not. In AI-advised decision making—where the AI provides a recommendation and the human makes the final call—the quality of this mental model is the linchpin of team performance. A doctor who knows that a readmission predictor struggles with a particular kind of patient can compensate accordingly. One who lacks this insight may blindly trust a flawed recommendation or, equally problematically, override a correct one out of unwarranted suspicion.
Prevalence, Drivers, and the State of Practice
AI-advised decision making has become one of the most common forms of human-AI collaboration in high-stakes domains. As Kamar (2016) articulated in her vision for hybrid intelligence, many real-world problems are best addressed not by full automation but by complementary systems in which humans and AI each contribute their distinctive strengths. Healthcare diagnostics, credit assessments, hiring recommendations, and judicial risk assessments all follow this pattern: an AI system generates a prediction or recommendation, and a human expert renders the final judgment.
Yet current practice often optimizes for the wrong metric. Bansal, Nushi, Kamar, Lasecki, et al. (2019) observed that even in applications where people make the final decisions, AI development typically optimizes for the accuracy of the model's inferences while ignoring the performance of the human-AI team. This accuracy-centric paradigm persists despite accumulating evidence that AI accuracy does not reliably translate into team performance. Stated model accuracy affects people's trust in uneven ways (Yin et al., 2019), providing predictions and explanations does not uniformly improve human decisions (Lai & Tan, 2019), and making a model's internals more transparent can even impair people's ability to notice and correct its mistakes, apparently because of information overload (Poursabzi-Sangdeh et al., 2021).
The drivers of this gap are both technical and organizational. On the technical side, standard machine learning pipelines lack mechanisms for evaluating or optimizing properties relevant to human comprehension, such as the simplicity of error patterns. On the organizational side, AI development teams and end-user communities often operate in silos, with data scientists optimizing models in isolation from the practitioners who must ultimately work alongside them. This disconnect means that the properties of AI systems that matter most for team performance—interpretability of error patterns, consistency of failure modes, manageable task complexity—are rarely surfaced, let alone prioritized.
Organizational and Individual Consequences of Misaligned Mental Models
Organizational Performance Impacts
When humans cannot form accurate mental models of AI capabilities, two categories of error emerge. First, misplaced trust: the human accepts the AI's recommendation when it is wrong, inheriting the AI's error. Second, misplaced distrust: the human overrides the AI when it is actually correct, forfeiting the benefit of accurate advice. Both failure modes degrade team performance, but their costs are often asymmetric, particularly in high-stakes settings where the penalty for an incorrect action far exceeds the reward for a correct one.
The experimental evidence is instructive. Using CAJA, an open-source, configurable, game-like platform for studying AI-advised decision making, Bansal, Nushi, Kamar, Lasecki, et al. (2019) demonstrated that two AI systems with identical accuracy can produce markedly different team performance depending on the properties of their error boundaries. An AI whose errors followed a simple, learnable pattern (a parsimonious error boundary) yielded significantly higher team performance than one whose errors followed a more complex pattern, even though both were equally accurate. Similarly, AI systems whose errors were non-stochastic—consistently wrong for a well-defined set of inputs—enabled better mental model formation than systems with probabilistic, fuzzy error patterns.
The practical implication is sobering: an organization might invest heavily in developing or procuring a highly accurate AI system, only to find that its real-world value is undermined because practitioners cannot learn when to rely on it. Grosz (1996) recognized decades ago that collaborative systems demand capabilities beyond task-level accuracy, but this insight has been slow to penetrate mainstream AI development practice.
Individual Wellbeing and Stakeholder Impacts
The consequences of misaligned mental models extend beyond organizational metrics to the individuals on both sides of the decision. For the human decision-maker—the clinician, judge, loan officer, or hiring manager—working with an unpredictable AI system creates what Lee and See (2004) described as a trust calibration challenge. When people cannot calibrate their trust to an automated system's actual capabilities, they risk both overreliance and underuse, along with the uncertainty and erosion of professional confidence that come with repeated surprises.
This is particularly acute where mistakes carry severe consequences. Bansal, Nushi, Kamar, Lasecki, et al. (2019) designed their experimental payoff structure to mirror high-stakes conditions: participants earned a modest reward for correctly accepting the AI's recommendation but lost four times as much for accepting it when it was wrong. Under these conditions, participants who could not learn the error boundary accumulated persistently worse outcomes—a dynamic that, in real-world settings, translates into clinical errors, unjust decisions, or wrongful loan denials.
For the people affected by these decisions—patients, defendants, job applicants—the stakes are higher still. When a decision-maker blindly trusts an AI system that systematically fails for a particular group, the resulting errors are not randomly distributed; they may disproportionately harm already vulnerable populations. This connects mental model failure directly to concerns about algorithmic fairness and accountability that have galvanized public attention in recent years (Angwin et al., 2016).
Evidence-Based Organizational Responses
Table 1: Error-Boundary Properties, Organizational Responses, and Illustrative Examples
Property or Challenge | What It Means | Why It Matters for Teams | Organizational Response | Illustrative Example |
Parsimony | How simply the conditions under which the AI errs can be described | Simpler error patterns are easier for people to learn, improving team performance at equal accuracy | Evaluate models on error-boundary complexity as well as accuracy; prefer intelligible models | Intelligible models for pneumonia risk and readmission (Caruana et al., 2015) |
Stochasticity | How consistently errors fall inside versus outside the error region | Noisy, inconsistent errors make it hard to form stable trust rules | Calibrate confidence; communicate uncertainty and known failure modes | Explainable hypoxaemia prediction during surgery (Lundberg et al., 2018) |
Task dimensionality | How many features the human must consider | More visible features make error patterns harder to discover | Curate features; use progressive disclosure; align human and machine features | COMPAS risk scores and simpler alternatives (Angwin et al., 2016; Dressel & Farid, 2018) |
Model updates | Whether a new model breaks patterns users have learned | A more accurate update can lower team performance if it introduces errors where users trusted the old model | Measure backward compatibility; use compatibility-aware training; stage rollouts | Backward-compatible updates (Bansal, Nushi, Kamar, Weld, et al., 2019) |
Human-aware governance | Whether selection and oversight consider team performance | Without it, accuracy-only choices persist | Add human-centered criteria to model selection; pilot with end users; cross-functional governance | Guidelines for human-AI interaction (Amershi et al., 2019) |
Designing for Parsimonious Error Boundaries
The most direct intervention suggested by the research is to design or select AI systems whose error patterns are simple enough for humans to learn. Bansal, Nushi, Kamar, Lasecki, et al. (2019) formalized this as parsimony—the degree to which an error boundary can be described with a small number of conditions. Their experiments showed that team performance decreased as the complexity of the error boundary increased, as measured by the number of conjunctions defining the error region. This finding aligns with broader arguments that, in high-stakes settings, inherently interpretable models are preferable to post hoc explanations of black boxes (Rudin, 2019).
Effective approaches for practitioners:
Evaluate candidate models on error boundary complexity, not just accuracy: During model selection, compare finalists not only on accuracy metrics but on the complexity of their failure patterns. If two models have similar accuracy, prefer the one whose errors cluster around fewer, simpler conditions.
Use error analysis tools to characterize and simplify error boundaries: Nushi et al. (2018) proposed hybrid human-machine analyses, including decision trees fit to a system's failures, to characterize when and why AI systems fail.
Consider regularization techniques that promote interpretability: Wu et al. (2018) demonstrated tree regularization methods that encourage deep models to behave in ways that can be approximated by simple decision trees.
Accept modest accuracy trade-offs when they substantially improve learnability: Bansal, Nushi, Kamar, Lasecki, et al. (2019) argued that a small sacrifice in model accuracy can be outweighed by a larger gain in overall team performance when it makes errors easier to learn.
Caruana et al. (2015) demonstrated the value of intelligibility in healthcare by developing generalized additive models with pairwise interactions to predict pneumonia risk and 30-day hospital readmission. These models achieved accuracy competitive with more complex methods while making the learned relationships visible to clinicians. In earlier work on the pneumonia data, a rule-based model had learned that patients with asthma had lower risk of dying from pneumonia—an artifact of the more aggressive care asthma patients historically received. The authors noted that a black-box model such as a neural network might learn the same dangerous pattern without anyone being able to see it, whereas an intelligible model exposes such patterns so that clinicians can recognize and correct them. The example shows how transparent models make the AI's likely failure modes legible to the people who must catch them.
Reducing Stochasticity in AI System Errors
Bansal, Nushi, Kamar, Lasecki, et al. (2019) identified a second critical property: the stochasticity of the error boundary. A non-stochastic error boundary cleanly separates the AI's successes from its failures—the system always fails for a specific, identifiable set of inputs and succeeds otherwise. A stochastic error boundary is fuzzy: the AI sometimes fails and sometimes succeeds for the same type of input, making it much harder for humans to build a reliable mental model.
The experimental results were striking. When errors were deterministic, the share of participants choosing the optimal action rose steadily over the session, approaching the optimal policy. When stochasticity was two-sided—errors could occur on both sides of the boundary—participants showed little improvement over many rounds and remained close to chance. Even one-sided stochasticity made correct trust patterns harder to learn.
Effective approaches for practitioners:
Prioritize confidence calibration: Well-calibrated confidence scores help humans develop probabilistic mental models of AI reliability. Guo et al. (2017) showed that modern neural networks are often poorly calibrated, producing overconfident predictions that can mislead human partners.
Communicate uncertainty transparently: Rather than presenting recommendations as bare yes/no outputs, provide calibrated probability estimates or uncertainty indicators that signal when the AI is operating outside its zone of competence.
Set accurate expectations about AI performance upfront: Kocielnik et al. (2019) found that designs that set appropriate expectations about an imperfect AI system's capabilities improved users' perceptions and acceptance of it, and Yin et al. (2019) showed that stated accuracy influences how much people trust a model.
Identify and communicate known failure modes explicitly: Rather than relying on users to discover error patterns through trial and error, proactively share known limitations in plain language.
Lundberg et al. (2018) developed and tested Prescience, an explainable machine learning system that predicts hypoxaemia (low blood oxygen) during surgery, using operating-room records from University of Washington hospitals. The system provided anesthesiologists with feature-level explanations showing which patient and procedure factors drove each prediction. In the study's evaluation, anesthesiologists anticipated hypoxaemia events more accurately when they had the system's predictions and explanations than when they relied on their own judgment alone. By showing why the model predicted high or low risk in each case, the system gave clinicians a basis for judging whether its reasoning was sound in that particular instance—helping them calibrate when to rely on it and when to question it.
Managing Task Dimensionality
The third factor identified by Bansal, Nushi, Kamar, Lasecki, et al. (2019) is task dimensionality—the number of features defining each instance that the human must consider. As the number of features grows, the space of possible error boundaries grows rapidly, making it harder for people to discover the relevant patterns. Their results showed that team performance generally decreased as the number of human-visible features increased, consistent with evidence that showing people more model detail can overwhelm rather than help them (Poursabzi-Sangdeh et al., 2021).
This creates a genuine tension in AI system design: adding features may improve the AI's standalone accuracy while degrading the human's ability to form an accurate mental model, potentially reducing overall team performance.
Effective approaches for practitioners:
Conduct feature relevance analyses from a human-centered perspective: Evaluate not only which features improve model accuracy but which are meaningful and interpretable to the human decision-maker.
Analyze the marginal trade-off explicitly: Bansal, Nushi, Kamar, Lasecki, et al. (2019) recommended weighing the marginal gain in model performance from each added feature against the marginal loss in the accuracy of human mental models.
Layer information presentation: Design interfaces that present the most critical information first and allow progressive disclosure of additional details on demand.
Align human-visible and machine-visible features where possible: When the AI reasons about features the human cannot observe, or vice versa, mental model formation becomes harder. Where feasible, ensure that the features most important to the AI's predictions are visible and meaningful to the human partner.
The COMPAS recidivism prediction tool illustrates the stakes in criminal justice. An investigation by ProPublica found that the tool's errors differed systematically by race: Black defendants who did not reoffend were more likely than White defendants to be labeled higher risk, while White defendants who did reoffend were more likely to be labeled lower risk (Angwin et al., 2016). Judges and other users received risk scores generated from a large set of inputs, with little information about where the tool's errors concentrated. Later research found that a simple model using just two features—age and number of prior convictions—predicted recidivism about as accurately as COMPAS (Dressel & Farid, 2018). The case suggests that complexity added little predictive value while making it harder for the people relying on the scores to understand when they might be wrong.
Building Effective Onboarding and Training Protocols
Bansal, Nushi, Kamar, Lasecki, et al. (2019) observed that mental models evolve through interaction: participants gradually refined their understanding of the AI's error boundary over successive rounds of feedback, with performance improving as their models became more accurate. That learning, however, was slow and depended heavily on the properties of the error boundary, and in real settings feedback is often delayed, partial, or absent.
These findings suggest that organizations cannot simply deploy an AI system and expect users to figure it out through trial and error. Structured onboarding and ongoing training are essential.
Effective approaches for practitioners:
Design guided onboarding experiences: Expose users to representative examples of both AI successes and failures, drawn from the actual distribution of cases the AI encounters.
Provide explicit failure case libraries: Curate documented examples of cases where the AI is known to err, organized by the features that characterize these failures.
Use progressive complexity in training: Start with cases where the error boundary is easy to see, then introduce more complex scenarios.
Monitor for mental model drift: As AI systems are updated and their error boundaries shift, previously accurate mental models may become outdated. Bansal, Nushi, Kamar, Weld, et al. (2019) demonstrated that updates that are not backward compatible—that introduce new errors on inputs where the previous model was correct—can break users' mental models and degrade team performance, even when the updated model is more accurate overall.
Microsoft Research addressed this challenge directly in its work on backward-compatible AI updates (Bansal, Nushi, Kamar, Weld, et al., 2019). Recognizing that people build mental models through accumulated experience with a specific system, the researchers showed that deploying a more accurate model could hurt team performance if its errors appeared where users had learned to trust the old model. They proposed training methods that add a compatibility term to the learning objective, reducing new errors on previously correct cases. The work exemplifies a shift from purely accuracy-driven model development to human-centered model lifecycle management.
Establishing Human-Aware Model Selection and Governance
The findings of Bansal, Nushi, Kamar, Lasecki, et al. (2019) ultimately point to a governance-level recommendation: organizations should institutionalize human-aware model selection. This means evaluating candidate models not only on traditional metrics such as accuracy, precision, and recall but also on properties that affect team performance—parsimony of the error boundary, stochasticity of errors, feature dimensionality, and backward compatibility with existing mental models.
Effective approaches for practitioners:
Incorporate human-centered criteria into model evaluation rubrics: When comparing candidate models during development or procurement, assess error boundary complexity and consistency alongside standard metrics.
Conduct human-in-the-loop pilot testing before deployment: Test candidate models with representative end users to assess whether practitioners can learn the error boundary within a reasonable number of interactions.
Establish update protocols that assess backward compatibility: Before deploying an updated model, evaluate what share of previously correct predictions remain correct, and plan communication and retraining for significant shifts.
Create cross-functional AI governance teams: Include both data scientists and domain practitioners so that human factors considerations shape model selection and deployment decisions.
Researchers at Microsoft codified many of these principles in their guidelines for human-AI interaction, developed and validated through a synthesis of more than two decades of design recommendations and testing with practitioners (Amershi et al., 2019). The guidelines include recommendations that directly support appropriate mental models, such as making clear what a system can do and how well it can do it. Translating such high-level guidelines into specific implementation decisions remains an ongoing challenge, which is why governance structures that bring designers, data scientists, and practitioners together matter.
Building Long-Term Human-AI Collaboration Capability
Psychological Contract Recalibration
The concept of psychological contracts—implicit expectations between parties in a working relationship—applies usefully to human-AI teaming. When an organization deploys an AI system, practitioners develop implicit expectations about what the system will do, how reliably it will perform, and how much of their professional judgment it replaces. Lee and See (2004) argued that trust in automation should be designed for appropriate reliance—neither too much nor too little. This requires ongoing recalibration as both the AI system and people's understanding of it evolve.
Building long-term collaboration capability requires organizations to treat these expectations as a managed asset. This means regularly communicating changes in AI capabilities, providing forums for practitioners to share their experiences with AI performance (including surprise failures), and explicitly acknowledging the complementary nature of human and AI contributions. Mohammed et al. (2010), in their review of team mental model research, found that shared understanding among team members is associated with better team performance. Extending this principle to human-AI teams suggests that organizations should build shared understanding across the workforce about an AI system's strengths, limitations, and appropriate uses.
Continuous Learning and Feedback Systems
Mental models are not static; they evolve through experience and feedback. Bansal, Nushi, Kamar, Lasecki, et al. (2019) showed that participants gradually improved their understanding of the AI's error boundary through repeated interaction with outcome feedback. Rouse et al. (1992) similarly emphasized that mental models of complex systems are built and refined through experience with the system's behavior.
Organizations should design continuous learning systems that support this process. These include structured feedback mechanisms that help practitioners understand why the AI was right or wrong in specific cases, not just that it was; regular recalibration sessions where teams review AI performance data and update their collective understanding; and interfaces that evolve with users' growing expertise. Doshi-Velez and Kim (2017) called for a rigorous science of interpretable machine learning grounded in evaluation with real users; that vision extends naturally to ongoing interpretability, in which the tools available to human collaborators evolve alongside the AI systems they work with.
A promising direction is what Bansal, Nushi, Kamar, Lasecki, et al. (2019) described as scratchpad-style support—tools that record and summarize observed error patterns, helping users maintain and refine their mental models across many interactions. Such tools acknowledge that human memory is limited and that mental models of complex systems benefit from external support.
Distributed Oversight and Collaborative Governance
As human-AI collaboration scales, maintaining accurate mental models becomes a collective governance problem, not merely an individual cognitive one. No single practitioner can observe all the ways an AI system might fail across diverse patient populations, market conditions, or case types. What is needed is a distributed oversight architecture that aggregates insights about AI performance across many users and contexts and feeds that intelligence back into the system and its users.
Grosz (1996) anticipated this need in her vision for collaborative systems, arguing that effective collaboration requires shared plans and mutual awareness of capabilities. Kamar (2016) developed this vision further, proposing hybrid intelligence in which human and machine capabilities are combined based on their complementary strengths. At the organizational level, this translates into governance structures that treat error boundary monitoring as a shared responsibility—using incident reporting, collaborative performance dashboards, and regular cross-functional reviews to refine collective understanding of when and where AI systems can be trusted.
The goal is not perfection but resilience: an organizational capacity to detect emerging failure modes quickly, communicate them effectively, and adapt both the AI system and human practices in response. This kind of adaptive governance is especially important given that model updates can disrupt established mental models (Bansal, Nushi, Kamar, Weld, et al., 2019)—a dynamic that, at organizational scale, requires coordinated change management rather than ad hoc individual adjustment.
Conclusion
The research initiated by Bansal, Nushi, Kamar, Lasecki, et al. (2019) delivers a message that is both simple and profound: the same AI system, with the same accuracy, can produce very different team outcomes depending on whether humans can learn when it fails. This insight has far-reaching implications for how organizations design, deploy, and govern AI-advised decision-making systems.
The actionable takeaways for practitioners can be summarized in five principles:
Optimize for team performance, not just AI accuracy: Treat the human-AI team as the unit of analysis, recognizing that the parsimony, non-stochasticity, and dimensionality of the error boundary directly affect outcomes.
Design AI systems with learnable error patterns: Prefer models whose failures cluster around simple, consistent conditions that human collaborators can discover and internalize.
Invest in structured onboarding and continuous learning: Provide guided training, explicit failure documentation, and ongoing feedback rather than assuming accurate mental models will develop on their own.
Manage model updates as trust events: Evaluate and communicate the impact of updates on established mental models, prioritizing backward compatibility and planning retraining when it cannot be maintained.
Build distributed governance systems: Treat error boundary monitoring as a shared organizational capability, feeding practitioner insights back into both system improvement and user support.
The era of AI-advised decision making is here. The question is no longer whether humans and AI systems will collaborate but whether that collaboration will be designed to succeed. Success depends not just on building smarter AI, but on building AI that people can understand well enough to use wisely.
Research Infographic
Designing Predictable AI Slide Deck
References
Amershi, S., Weld, D. S., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S. T., Bennett, P. N., Quinn, K. I., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Association for Computing Machinery.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica.
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019). Beyond accuracy: The role of mental models in human-AI team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 2–11.
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019). Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), 2429–2437.
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1721–1730). Association for Computing Machinery.
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning (arXiv:1702.08608). arXiv.
Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), Article eaao5580.
Grosz, B. J. (1996). Collaborative systems (AAAI-94 presidential address). AI Magazine, 17(2), 67–85.
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321–1330). PMLR.
Kamar, E. (2016). Directions in hybrid intelligence: Complementing AI systems with human intelligence. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (pp. 4070–4073).
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019). Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–14). Association for Computing Machinery.
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012). Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1–10). Association for Computing Machinery.
Lai, V., & Tan, C. (2019). On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 29–38). Association for Computing Machinery.
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., & Lee, S.-I. (2018). Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering, 2(10), 749–760.
Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 36(4), 876–910.
Norman, D. A. (1988). The psychology of everyday things. Basic Books.
Nushi, B., Kamar, E., & Horvitz, E. (2018). Towards accountable AI: Hybrid human-machine analyses for characterizing system failure. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6(1), 126–135.
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2021). Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–52). Association for Computing Machinery.
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union.
Rouse, W. B., Cannon-Bowers, J. A., & Salas, E. (1992). The role of mental models in team performance in complex systems. IEEE Transactions on Systems, Man, and Cybernetics, 22(6), 1296–1308.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018). Beyond sparsity: Tree regularization of deep models for interpretability. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 1670–1678.
Yin, M., Vaughan, J. W., & Wallach, H. (2019). Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–12). Association for Computing Machinery.
Jonathan H. Westover, PhD, Chief Research Officer (Nexus Institute for Work and AI); Co-Founder & Chief Workforce and Learning Officer (Future State University); Founder & CEO (Human Capital Innovations); Professor of Organizational Leadership & Change (UVU). Read Jonathan Westover's executive profile here.
Suggested Citation: Westover, J. H. (2026). Designing for Predictability: How Mental Models of AI Error Boundaries Shape Human-AI Team Performance in High-Stakes Decision Making. Human Capital Leadership Review, 39(4). doi.org/10.70175/hclreview.2020.39.4.2
