The Identity AI Invents From Your Data

Personalization used to answer a relatively simple question: What might this person want next?

AI is beginning to answer a more complicated one: Who do we think this person is?

Systems can create patterns across someone's data to infer preferences, needs, and intentions. Those conclusions can determine what content someone sees, what a service recommends, or what an AI-powered experience does next. The problem is that an inference can be convincing without being complete.

In this episode of the #shifthappens podcast, Privacy, Data, and AI Governance Leader Jo Ann Davaris explores what organizations need to change as personalization becomes inference. Her argument puts the person back at the center: Organizations need to understand what their AI is inferring, give people meaningful ways to correct those conclusions, and make sure accountability remains clear when AI acts on them.

When Personalization Becomes Inference

Personalization has always involved assumptions. For instance, a retailer recommends another product based on what someone bought, or a streaming service suggests a show based on what they watched. Those experiences generally start from an observable behavior and predict what the person might want next.

AI can go further. Given enough signals, a system can infer what someone prefers, what they may need, or what they intend to do. That inferred picture can then influence recommendations and other decisions.

The distinction matters because an inference is not a fact about a person. A shared account, an unusual purchase, missing context, or a temporary behavior can all change the story the data appears to tell. Yet, the affected person may have little visibility into the conclusion the system reached.

As Jo Ann puts it, “Just because an algorithm reached a conclusion about me, doesn't mean it actually knows me.”

When organizations introduce more sophisticated personalization, the objective cannot just be accuracy. They also need to consider whether people can understand how AI reached a conclusion about them and what happens when that conclusion is wrong.

Why AI Changes what Transparency Requires

Traditional transparency often focuses on disclosure: what information an organization collects, how it uses that information, and which systems are involved.

AI introduces another layer, and people may also need to understand what the system has concluded from their information. Providing that visibility does not mean exposing every data point behind a model. A person should not have to sort through hundreds of attributes to understand why their experience changed.

Meaningful transparency helps answer more practical questions: What did the system infer about me? Why did that affect my experience? What can I do if it is wrong?

That last question becomes especially important as personalization becomes deeply embedded in products.

Jo Ann argues that people should have an opportunity to shape or challenge what AI infers about them. Depending on the experience, that could include correcting an assumption, adjusting how personalization works, or choosing not to use certain personalized features.

“Think about the places where people may want a little bit more agency, may want a little bit more of an ability to shape, or challenge what you're inferring about me,” she mentions.

The goal is not to remove personalization. It is to make sure increasingly intelligent personalization does not remove the person from the process.

Why AI Governance Cannot Stay in Organizational Silos

The same challenge appears inside organizations.

Privacy and security traditionally examine different dimensions of data. Privacy considers why information is collected and used and how that affects the individual. Security considers how information, systems, and access are protected.

AI increasingly brings those responsibilities together. A system might be technically secure while still using data in ways that create an unexpected inference about someone. Conversely, an appropriate use of data can still become risky if access, permissions, or downstream systems are poorly protected. Bearing this in mind, organizations cannot wait until the end of a project to bring these perspectives together.

Privacy, security, product, legal, compliance, risk, and technical teams need sufficient shared visibility to understand whether AI works and what information it relies on, what it infers, and what decisions follow.

For multinational organizations, governance also must accommodate differences between regions and business units. A common set of minimum expectations can establish what every part of the organization must protect. Local teams can then apply those expectations to their own regulatory environments, products, and risk profiles.

What Leaders Can Do Before AI Makes the Next Decision

Policies and assessments matter, but Jo Ann's advice ultimately brings leaders back to the experience itself.

Her advice gives leaders four places to start: preserve people's agency; bring privacy and security together earlier; establish a governance baseline that works across the organization; and clarify accountability before teams are under pressure.

Keep the Person in Personalization

Give people a way to see, question, and correct what AI infers about them.

Start with the experiences where an inference can meaningfully affect what someone sees, receives, or is able to do. Make it clear why the experience changed and give people a practical way to correct an assumption when AI gets the context wrong.

An algorithmic conclusion can inform personalization without becoming an unquestioned version of the person behind the data.

Treat Privacy and Security as One

Bring privacy and security teams into AI assessments and decisions from the start.

Privacy considers how and why information is used and what that means for the individual. Security considers how information, systems, and access are protected. AI increasingly brings those responsibilities together, so both perspectives need to inform decisions before a launch or problem forces them into the same room.

Standardize the Center, Trust the Edges

Set minimum governance expectations centrally, then give regions and business units room to apply them to their own environment.

Define the principles, controls, and risk boundaries that should be shared across the organization. From there, give local teams the autonomy to respond to different regulations, products, and risk profiles while remaining accountable to that shared baseline.

Consistency does not require every team to make the same decision. It requires every decision to meet the same minimum standard.

Name the Decision Maker First

Decide who owns the final call before a high-stakes situation puts accountability to the test.

Run realistic AI tabletop exercises around prominent products and the systems they depend on. Use those scenarios to identify missing stakeholders, clarify who provides input, and establish who ultimately makes the decision.

Several teams may need a voice. Accountability still needs one clear owner.

As Jo Ann puts it, “There can only be one decision maker.”

These actions support the broader framework Jo Ann shares: See it, experience it, and stress test it. Leaders need visibility into where AI is shaping experiences, firsthand understanding of what those experiences mean for customers, and confidence that their governance will hold up when assumptions or systems fail.

Keep the Person Bigger Than the Profile

The identity AI constructs from data can make an experience more relevant, anticipate what someone needs, and reduce the effort required to find the right information or service. However, usefulness should not turn inference into certainty.

Organizations need to know where AI is building conclusions about people, understand how those conclusions affect experiences, and preserve meaningful ways for people to correct the record. They also need governance that connects privacy, security, risk, product, and technical decisions without losing clear accountability.

Jo's framework gives leaders a practical starting point: see where AI is influencing decisions, experience what those decisions mean for customers, and stress-test whether the organization is ready when the system gets something wrong.

As AI learns more from our data, strong governance should make sure the person behind that data is not forgotten.

Soundtrack of Shift

Jo Ann's Soundtrack of Shift is “Running Up That Hill” by Kate Bush.

For her, the song connects to the human side of AI and the importance of understanding another person's experience. That same idea runs through her approach to personalization: data can help a system construct a picture of someone, but it cannot capture the whole human being behind it.

Explore more soundtracks shaping how leaders approach change and transformation today.

Episode Resources

#shifthappens Research: 2026 State of AI

#shifthappens Insights:

#shifthappens Podcasts:

Dux Raymond Sy on LinkedIn

Dana Simberkoff on LinkedIn

Jo Ann Davaris on LinkedIn

#shifthappens podcast

Relevant Episodes

Episode 141: The Identity AI Invents From Your Data