AI-Generated User Research: The Use of Virtual Synthetic Users

Can AI replace users in user interviews?

One-line summary: Not feasible.

While AI-generated "virtual, synthetic users" and user research reports can be useful in certain situations, they cannot completely replace real user research, and over-reliance may pose risks.

Recently, Nielsen Norman Group published an interesting article comparing the effectiveness of specialized synthetic user platforms, ChatGPT, and real user research, revealing the pros and cons of virtual user research. Let's discuss why virtual user research may be risky and when synthetic users can be used for research.

Why Virtual User Research May Be Risky?

1. Overly Optimistic Responses: Virtual users tend to be more optimistic than real people, which can lead to inaccurate business decisions. AI sometimes tries to "please" humans, using "flattering" language that may mislead our judgment of user needs.

2. Unclear Data Sources: What data are these virtual users generated from? This is often not transparent enough, and we need to pay extra attention to avoid making decisions based on unreliable information.

3. Difficulty in Deep Exploration: Virtual research struggles to deeply understand user behavior and attitude details, which offers limited help to designers who want to truly improve user experience. Deep user insights usually require face-to-face communication and observation.

4. Impact on Organizational Culture: In companies where user experience culture is not mature enough, management might mistakenly think user research is simple, thus overlooking the importance of real user research. This affects not only current project quality but may also change company work methods and culture.

When Can Synthetic Users Be Used?

As a complement to desk research:

1. Quick Industry User Understanding: If teams need to quickly grasp basic information and types of industry users, virtual users can provide initial understanding, helping everyone get started quickly.

2. Hypothesis Foundation: Before conducting real user research, virtual users can help teams form preliminary hypotheses, providing direction for subsequent research.

Even when real user research is not possible, using virtual research may still be misleading. So, if it must be used, it's essential to be aware of potential risks and be prepared.

Conclusion

There are always different views on this topic. We can decide how to proceed based on specific project circumstances, but decisions must be well-reasoned. Using new tools is fine, but understanding why we use them is more important. In portfolios, the key isn't blindly following trends in using new tools, but rather considering whether they suit the current project.