<img src="https://spectrum.ieee.org/media-library/pixel-art-figure-in-a-colorful-digital-cube-with-shadow-and-connected-emoji-faces.png?id=66966345&width=1245&height=700&coordinates=0%2C237%2C0%2C238"/><br/><br/><p><strong>Imagine sitting down at </strong>your desk and logging in for a performance review, with an AI system analyzing the conversation. You’ve been working long hours, balancing deadlines, and your manager asks how you’re doing. You say you’re fine, and maybe even smile, but there’s a hint of hesitation and your voice wavers. As you shift your posture, your shoulders slump.</p><div class="rm-embed embed-media"><iframe height="110px" id="noa-web-audio-player" src="https://embed-player.newsoveraudio.com/v4?key=q5m19e&id=https://spectrum.ieee.org/emotion-ai-context?draft=1&bgColor=F5F5F5&color=1b1b1c&playColor=1b1b1c&progressBgColor=F5F5F5&progressBorderColor=bdbbbb&titleColor=1b1b1c&timeColor=1b1b1c&speedColor=1b1b1c&noaLinkColor=556B7D&noaLinkHighlightColor=FF4B00&feedbackButton=true" style="border: none" width="100%"></iframe></div><p><span>These are subtle cues that to the human eye might hint at underlying stress. But to an AI model that’s been trained only to categorize emotions as “happy” or “sad,” such nuances are likely lost. It logs the words and a smile and moves on—and unless your human manager intervenes, the fact that you’re tired, unfocused, and maybe a couple of days from burnout never enters the equation.</span></p><p>“<a href="https://spectrum.ieee.org/building-an-ai-that-feels" target="_blank">Emotion AI</a>,” which estimates how people feel based on facial expressions, voice tone, and behavior, seems to be suddenly everywhere; it’s being used in employee well-being and recruitment interviews, education platforms, and driver-monitoring systems. Technology call-center platforms such as <a href="https://www.nice.com/" target="_blank">NiCE</a> and <a href="https://www.genesys.com/" rel="noopener noreferrer" target="_blank">Genesys</a> use AI to detect when a customer sounds frustrated and prompt agents in real time to slow down or respond with more empathy. Giant companies like <a href="https://raveintelligence.com/meta-voice-ai-surge-emotional-intelligence/" rel="noopener noreferrer" target="_blank">Meta</a> and startups such as <a href="https://www.hume.ai/" rel="noopener noreferrer" target="_blank">Hume AI</a> are developing more-expressive voice AI systems that can detect emotional cues in the person they’re “talking” to and adjust how they communicate.</p><p>What’s more, hundreds of companies already offer virtual AI companionship apps, a fast-growing market that may be worth an <a href="https://www.sphericalinsights.com/reports/ai-companion-market#:~:text=Table_content:%20header:%20%7C%20Base%20Year:%20%7C%202024,CAGR:%20%7C%202024:%20CAGR%20of%2031.05%25%20%7C" rel="noopener noreferrer" target="_blank">estimated US $555 billion</a> by 2035—and robot buddies have also entered the picture. Intuition Robotics’s <a href="https://elliq.com/?srsltid=AfmBOoqjBb7RoBuC0piFi5F-u5d64LbS_BVhLwG79xwEbTnrZwBx86fR" rel="noopener noreferrer" target="_blank">ElliQ</a>, for example, is a small device vaguely resembling a white desk lamp that’s now being used to engage older adults in conversation in hopes of reducing loneliness.</p><p>But while the field of emotion AI is advancing at a rapid clip, most existing systems are focused on detecting a limited number of signals to label one specific emotion at a time—which is insufficient if you’re trying to understand the human condition. In the real world, human signals and emotions are contextual, overlapping, and constantly changing. A laugh can signal joy, nervousness, or both; a raised voice might signal enthusiasm just as easily as frustration. To make the job of emotion detection even more difficult, reactions differ greatly from one individual to the next, depending on demographics, cultural background, and countless other variables.</p><p>In other words, there’s a gap between what we’re expecting AI to pick up on and what AI can actually deliver. That’s the gap a new field of research—what we call human-context AI—is working to close. Instead of looking at just one input and labeling it, human-context AI increasingly has the capacity to take stock of an individual’s personality and character, and to track emotions in real time while combining <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC12292624/" rel="noopener noreferrer" target="_blank">multiple inputs</a>, including facial dynamics, voice, tone, language, and behavior. Crucially, responses are also evaluated in the context of a specific environment, such as a performance review or professional coaching session. The result? Computers are learning to read the scene, rather than just the screen.</p><h2>The Origins of Emotion AI</h2><p>The story of emotion-sensing AI began almost three decades ago in the MIT Media Lab, where the American electrical engineer and computer scientist <a href="https://spectrum.ieee.org/how-and-why-companies-will-engineer-your-emotions" rel="noopener noreferrer" target="_blank">Rosalind Picard</a> coined the term “affective computing.” Her work introduced the radical idea that computers could be taught to recognize and respond to human emotions.</p><p>Picard’s <a href="https://cs.uwaterloo.ca/~jhoey/teaching/cs886-affect/papers/Picard-AffectiveComputing/9780262281584_chap6.pdf" rel="noopener noreferrer" target="_blank">early experiments</a> focused on single modalities: facial expressions, tone of voice, and physiological signals, such as skin conductance or heart rate. The goal was to give machines a window into human feeling, helping them become more empathetic. It was an exciting vision, but back then the science and hardware weren’t ready. Computing power was limited, sensors were crude, and datasets were narrow and biased.</p><p class="shortcode-media shortcode-media-rebelmouse-image"> <img alt="Pixel art of three party-hatted figures in a box, each losing a slice of cake." class="rm-shortcode" data-rm-shortcode-id="915714dd60f44acd05b8adbdd1ed711f" data-rm-shortcode-name="rebelmouse-image" id="af91e" loading="lazy" src="https://spectrum.ieee.org/media-library/pixel-art-of-three-party-hatted-figures-in-a-box-each-losing-a-slice-of-cake.png?id=66966369&width=980"/> <small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Josie Norton</small></p><p>Over the next decades, researchers and companies got better at measuring the many ways in which humans express themselves. In the 2010s, <a href="https://en.wikipedia.org/wiki/Sentiment_analysis" target="_blank">sentiment analysis</a>—the processing of large volumes of text to suss out emotional undertones—began to reach the mainstream. At the same time, marketing firms, including my company, <a href="https://www.neurologyca.com/" target="_blank">Neurologyca</a>, began using video and webcams to measure and catalogue customer reactions. Biometric devices and activity trackers, such as Fitbits and Apple watches, also became ubiquitous, generating new streams of data about people’s sleep, step counts, stress levels, and more.</p><p>Unsurprisingly, scientists soon confirmed that larger volumes of personalized data led to greater accuracy in reading human emotions. In 2019, researchers at Cornell demonstrated that <a href="https://arxiv.org/abs/1905.07039" target="_blank">combining multiple types of signals</a> improves emotion sensing. Their system joined physiological data, such as brain activity measured by electroencephalography (EEG) and heart rate, with visual cues like facial expression, outperforming systems that relied on just one input. Around the same time, Picard and her team at MIT found that humanoid robots <a href="https://news.mit.edu/2018/personalized-deep-learning-equips-robots-autism-therapy-0627" rel="noopener noreferrer" target="_blank">trained on data unique to a specific person</a> were substantially better at reading that person’s reactions and feelings than robots acting without personalized data.</p><p><span>More recent studies align with these findings. In 2024, <a href="https://www.sciencedirect.com/science/article/abs/pii/S095741742400589X" target="_blank">scientists in South Korea</a> showed that fusing physiological, environmental, and personal data to recognize emotion resulted in a 32 percent error reduction. <a href="https://dl.acm.org/doi/10.1145/3746270.3760232" target="_blank">Another paper, published in 2025</a>, demonstrated that user-specific information significantly enhances emotion recognition performance.</span></p><p>Today, our devices know who we are; our habits and tendencies, likes and dislikes. They’ve also gotten smaller and more efficient. Tiny, low-power cameras and microphones embedded in phones, laptops, and virtual-reality and augmented-reality devices can detect dozens of human signals simultaneously, from eye movements and micro-expressions to breathing rhythms, voice modulation, and posture. Advances in computing have also made it possible to integrate audio, video, biometric, and text data, often without even transmitting raw data to the cloud. And researchers at <a href="https://vhil.stanford.edu/publications/predictive-analytics/cognitive-load-inference-using-physiological-markers-virtual" rel="noopener noreferrer" target="_blank">Stanford</a>, <a href="https://www.cl.cam.ac.uk/~pr10/publications/ptb09.pdf" rel="noopener noreferrer" target="_blank">Cambridge and MIT</a>, and <a href="https://sap.ist.i.kyoto-u.ac.jp/lab/bib/intl/LAL-AAAI-sympo17.pdf" rel="noopener noreferrer" target="_blank">Kyoto University</a>, in Japan, as well as <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC12292624/" rel="noopener noreferrer" target="_blank">the Software College of Northeastern University </a>in Shenyang, China, are exploring how fusing such inputs can refine the sensitivity and accuracy of human-machine interactions.</p><p>And yet, despite so many breakthroughs, machines still can’t reliably interpret emotion or even physical stress. Just last year, a survey published in the<a href="https://psycnet.apa.org/doiLanding?doi=10.1037%2Fabn0001013" rel="noopener noreferrer" target="_blank"> <em><em>Journal of Psychopathology and Clinical Science</em></em></a> revealed that stress scores on smartwatches rarely, if ever, matched the level of stress that users were experiencing. In fact, a quarter of those surveyed reported feeling the direct opposite of what their smartwatches were reporting.</p><p>Why the disconnect? We’ve gotten very good at capturing signals, but not at interpreting them. A fitness tracker might infer from your heart rate that you’re stressed and recommend easing off training, but it doesn’t know if your increased heart rate is due to excitement, tiredness, or an extra cup of coffee. Gauging emotions in real-world settings is even more difficult. To solve this complex problem, machines need context.</p><h2>From Neuromarketing to Emotion-Sensing AI</h2><p>My company, Neurologyca, was founded in Spain in 2015, and started out in neuromarketing. Working with major European brands and conglomerates, our cofounder, Juan Graña, had realized that companies lacked solid data on consumers. At the time, most customer feedback came through surveys, which posed questions such as, “On a scale of 1 to 10, how joyful does this car advertisement make you feel?” or “Which emoji best describes your mood?” Naturally, these overly simplistic tools led to high levels of self-reporting bias, as people often misjudge or misstate their own reactions.</p><p>To get around this problem, Neurologyca set up labs, using neuroscience and cognitive science to more accurately capture human responses to products, logos, advertisements, and experiences. In addition to using biometric tools such as heart monitors, eye trackers, and EEG, we recorded millions of video frames of human reactions, logging each specific context and the resulting facial and bodily movements. To do this, we mapped over 790 points of reference, including corners of the mouth, size of the eyes and pupils, blink rate, and angling of the head. All of this data was collected and stored anonymously under strict European privacy standards.</p><p>Next, we paired this information with findings from decades of neuroscience and behavioral science studies on how biometrics, speech patterns, and human movement are related to emotion—research we continue to gather from academic institutions across Europe. We also created a database of situational contexts—for example, “watching a dog food commercial” or “hearing a new song”—and the human feelings they engendered.</p><p>In our work with companies, not only did this approach allow us to recognize nuanced emotions, it also let us identify which reactions indicated positive or negative outcomes. Take, for example, the context of horror-film trailers: Our research helped us figure out that the most successful elicit a very specific mix of emotions, namely a little bit of fear, a little bit of anxiety, but also some joy. With this knowledge, we could quickly rate viewer reactions to help a film company figure out how to tweak its trailer for the desired impact.</p><p class="shortcode-media shortcode-media-rebelmouse-image"> <img alt="Colorful 3D blocks explain Neurologyca\u2019s behavioral, situational, and personal context layers" class="rm-shortcode" data-rm-shortcode-id="ceca390665355bd35746d0a57c65863f" data-rm-shortcode-name="rebelmouse-image" id="8096e" loading="lazy" src="https://spectrum.ieee.org/media-library/colorful-3d-blocks-explain-neurologyca-u2019s-behavioral-situational-and-personal-context-layers.png?id=66966347&width=980"/> <small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Neurologyca</small></p><p>Within a few years, we discovered that a model trained on our database could accurately evaluate emotion using just a webcam. We stopped needing to host focus groups in rooms full of equipment. Instead, we were able to do such things as sending out a new perfume sample to paid participants around the world along with a link. When people opened the link, it turned on their cameras, allowing us to record their faces as they sniffed the perfume for the first time. Suddenly, we had expanded our reach: Rather than using small focus groups in one or two countries, we could quickly assess 1,000 people across the planet, comparing how someone in Japan, India, or Germany might feel about a certain product.</p><p>About four years ago, as AI was becoming pervasive, we realized that our models had applications well beyond neuromarketing. Importantly, these models are grounded in directly observed human behavior rather than inferred patterns or loosely labeled open datasets. Looking beyond brands and companies, we established that our model could be integrated into AI systems to help them understand human emotion at a much more granular level. In other words, we could provide a layer of context.</p><h2>For Empathetic AI, Context Is Key</h2><p>When we talk about “a layer of context,” we mean three different types of context. The first is situational or environmental context; for example, a performance review, a telemedicine session, or a horror-film viewing. The second is personal context, which includes an individual’s specific history, goals, and baseline state. The third is behavioral context, which covers the individual’s reaction over the course of the event or interaction by evaluating real-time changes in attention, confidence, engagement, and cognitive load.</p><p>Most systems today focus on only situational context, although some are starting to include personal context. Very few include behavioral context or combine all three in a meaningful way. What we’ve built at Neurologyca is a logic layer that fuses the three and translates them into structured, machine-readable information that allows AI systems and agents to respond more effectively. Our technology is being used to enhance systems in development, as well as some that have already been deployed, including driver-safety apps like <a href="https://www.netradyne.com/" target="_blank">Netradyne</a>, home assistants like <a href="https://alexa.amazon.com/about" target="_blank">Amazon Alexa</a>, and health-care AI platforms like <a href="https://www.sully.ai/" target="_blank">Sully.ai</a>.</p><p>It works as follows: Situational context is determined by the platform or application, be it a professional coaching session, a meditation app, or a driver’s safety monitor. Personal context already lives within each respective platform—or if not, it can be created through sharing of personal data or monitoring via camera. (Most wellness and professional-development apps, for example, contain each user’s profile, history, and prior sessions.) Last but not least, behavioral context is collected and analyzed in real time using our models. In the end, our logic layer fuses these three streams of information.</p><p>Our system doesn’t assign fixed weights to the three contexts. Instead, it provides a continuous calibration, with the balance shifting depending on the specific situation. For example, a pause in speech might signal uncertainty in a performance review, but something entirely different in a relaxation setting. If signals are ambiguous or overlapping, our system reflects that uncertainty through lower confidence scores rather than forcing a definitive interpretation.</p><p>What’s more, our system can work without ever sending raw data to the cloud, thereby easing privacy concerns. In many cases, video, audio, and biometric signals never leave the device. Instead, our lightweight models extract information locally and share only what’s necessary. Cloud systems, meanwhile, are used for training, pattern analysis, and model improvement. The result is a hybrid architecture: edge-based processing for speed and privacy combined with cloud-based learning for continuous improvement.</p><p>The result? By incorporating context, AI systems are beginning to interpret aspects of the human state as interactions unfold, dynamically adapting to emotions rather than reacting after the fact. The range of potential applications is broad and still evolving. Picture a professional-development platform that uses a human avatar to perform a mock interview and then provide feedback and tips on how to appear more confident, likeable, and well-informed. Or a meditation app that knows exactly how well you slept and how anxious you’re feeling, and can recommend an appropriate breathing meditation. Or a humanoid robot teacher that can tell when a student is confused or bored and step in to get them back on track.</p><h2>Avoiding Potential Dangers on the Road Ahead</h2><p>There have long been debates about the ethics of emotion-sensing AI. Some critics question whether systems should attempt to infer human feelings from external signals at all. They argue that reducing people to measurable outputs risks oversimplifying human experience while opening the door to manipulation, surveillance, and unfair judgments in workplaces, schools, and public spaces.</p><p>We take those risks extremely seriously. In fact, our technology aims to reduce the dangers of oversimplifying human emotion. Human-context AI is not based on the assumption that a machine can definitively know what someone is feeling. Rather, it is an attempt to move beyond simplistic labels by incorporating situational, personal, and behavioral context, while explicitly representing uncertainty when signals are ambiguous or incomplete.</p><p>That said, ethical concerns regarding implementation are real and have shaped the kinds of projects we pursue. We would never, for example, accept military engagements to help with interrogations. Not only for ethical reasons: E<span>motion AI cannot reliably detect deception, and claiming otherwise would be overstating what the technology can actually do.</span> And while our technology can be used to gauge crowd behavior and predict things like when a football stadium is at risk of becoming destructively rowdy, we don’t want our technology deployed for surveillance. In short, we believe that using our logic layer on anyone who hasn’t opted in would be intrusive and ethically problematic.</p><p><span>In Europe, our systems are designed to comply with the EU AI Act’s restrictions on emotion recognition in workplaces and schools; as we expand into the United States, we apply jurisdiction-specific guidelines while maintaining the same core ethical commitments.</span></p><p>We also don’t advise companies to become overly reliant on our technology. Hiring and firing decisions should not be based on our outputs alone. Instead, our logic layer is designed to support human understanding and surface emotions that might otherwise go unnoticed.</p><p>Let’s return to the scenario of the performance review. Never mind basic AI—all humans, and even great managers, miss things during conversations. There’s a lot happening at once, as people process what’s being said, how to respond, and the greater context of the situation. These days, many exchanges also occur virtually or via video, adding more distractions while shared context is stripped away.</p><p>While we would never claim that our models understand humans better than their fellow humans, we believe we can offer an added layer to help managers capture and interpret behavioral signals that might otherwise get lost, providing greater visibility into how a conversation is unfolding.</p><p>Our model can track patterns moment to moment, picking up, for example, a shift in engagement, an instance when something didn’t land, or a change in how someone is behaving. The model won’t tell the manager what these moments mean or what to do about them; it simply makes them easier to see and follow up.</p><p>Human-context AI is at an early stage. The use cases, the adoption patterns, and the actual impact are all still evolving. At the same time, emotion-sensing systems are quickly being incorporated into real products and platforms. And without context—without knowing <em><em>why</em></em> people feel the way they do—AI risks misunderstanding us in critical moments. <span class="ieee-end-mark"></span></p>
Reference: https://ift.tt/8OEZbge
