AI tutors: what works, and what can go wrong

AI chat tools can explain almost anything, at any hour, in plain language. For students that is a big change. But a tool that can answer every question is not automatically a good tutor. Two recent studies show both sides clearly, and point to what makes the difference.

When an AI tutor helped students learn more

In 2024 and 2025, physics teachers at Harvard University ran a randomised experiment in their own introductory course. Students learned some topics in a lively, research-based class session and other topics with an AI tutor that the teachers had designed. The tutor did not simply hand out answers: it was built on good teaching practice, guiding students through problems step by step, encouraging them to think for themselves and keeping the workload manageable.

Students learned significantly more with the AI tutor than in the class session, in less time, and reported feeling more engaged and motivated. The authors, Gregory Kestin and colleagues, are careful to say that this does not mean AI should replace teachers. Their tutor was carefully designed, it covered introductory material, and the students were at a highly selective university. But it shows what a well-designed AI tutor can do.

When an AI tool made learning worse

Another study, by Hamsa Bastani and colleagues, worked with around a thousand high school students in Turkey learning mathematics. Some students practised with a standard AI chat assistant, some with a version designed as a tutor (it gave hints and guidance, and was told not to simply give the answer), and some practised without AI.

During practice, both AI groups solved far more problems correctly than students without AI. But on a later exam taken without AI, the students who had used the standard chat assistant did worse than students who had never used AI. Many had used it to get answers rather than to understand. The tutor version largely avoided this harm: its students did about as well on the exam as students who practised without AI.

The lesson is uncomfortable but useful: getting the right answer with AI is not the same as learning. If the tool does the thinking, you get the marks today and miss the learning you need for the exam.

Why AI tutors can go wrong

There are two main risks.

1. Doing the work for you. Learning happens when you retrieve, reason and make mistakes you then correct. An assistant that produces finished answers on request removes exactly that effort. This is what the Bastani study found.

2. Confident mistakes. Large language models generate text that sounds right. Sometimes it is not: they can invent facts, misquote sources or answer a slightly different question. This is often called "hallucination". It is especially risky for students, who may not yet know enough to notice.

What a good AI tutor does

Putting the research on intelligent tutoring systems together with these studies, a helpful AI tutor:

• Works from trusted material. Answers are based on sources you can check, such as your lecture notes or textbook, rather than on whatever the model happens to generate. Researchers call this approach retrieval-augmented generation.
• Shows its sources. Every point should lead back to where it came from, so you can read the original and judge it yourself.
• Says when it does not know. If your materials do not cover a question, a good tutor tells you so instead of filling the gap with guesses.
• Keeps you thinking. It explains, asks you to try, quizzes you and gives feedback, rather than writing your work for you.
• Protects your privacy. Your notes and questions are used to help you, not shared or used to train models.

How to use an AI tutor well

Whatever tool you use, a few habits make the difference between learning and copying:

1. Try first. Attempt the problem or write your own answer before asking. Then ask the tutor to check it or explain where you went wrong.
2. Ask "why", not just "what". Ask for the reasoning, an example or a comparison, and then explain it back in your own words.
3. Check the source. Open the passage the answer is based on. If an answer has no source, treat it with caution.
4. Quiz yourself. Turn what you have learned into flashcards or practice questions, and come back to them over the following days.
5. Follow your institution's rules. Many universities have rules about using AI in assessed work. Check them, and say when you used AI if your institution requires it.

Where Studentlytica AI stands

We designed Studentlytica AI around these lessons. The AI Tutor answers from the course materials you upload, shows where each point comes from, and tells you when your materials do not cover a question. Study tools turn your notes into flashcards, quizzes and practice questions with feedback, so you practise recalling rather than re-reading. You can read more in How the Studentlytican AI Tutor answers from your own notes, and about the research behind tutoring in What is an intelligent tutoring system?

No tool can learn for you. A good AI tutor can make your own effort go further.

Sources

• Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, 17458. doi.org/10.1038/s41598-025-97652-6
• Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. doi.org/10.1073/pnas.2422633122
• Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33. arxiv.org/abs/2005.11401