Part 1: You Think You’re Asking. You’re Also Teaching.

What Are We Teaching AI? — Part 1 of 2

You ask an AI assistant to write a function. It returns something that looks reasonable, until you notice a problem.

“This breaks when two requests arrive at the same time.”

The assistant tries again. You explain why the second version still fails, describe the constraint it missed, and eventually accept a solution.

You came for help. Along the way, you supplied a task, evaluated several answers, identified a failure, and explained what a better answer should do.

That is useful work. Some of it also resembles the work involved in creating training and evaluation data for AI.

The important question is what happens to those judgments after the conversation ends.

When people hear “data labeling,” they often imagine someone drawing boxes around objects in a photograph. But labels can also express a preference: this answer is more useful, this explanation is misleading, this solution satisfies the requirements.

The InstructGPT research demonstrated one way to use such judgments: human-written demonstrations and rankings of model responses helped train models to follow instructions more effectively.[1] That establishes the value of human feedback. It does not establish that every ordinary conversation enters the same process.

Still, familiar interactions can contain similar information. Choosing one answer over another expresses a preference. Correcting a factual error supplies a proposed correction. Explaining why a solution fails provides a failure case and a criterion for improvement.

The difference is that, as a user, you may experience all of this simply as finishing your task.

Consider how much expertise can fit inside a short correction:

“Don’t retry this operation automatically. It isn’t idempotent.”

The value is in the judgment behind the sentence. You recognized a risk, connected it to a property of the system, and rejected a superficially convenient solution. If that exchange were appropriately collected, checked, and selected, it could become useful material for evaluating or improving a model.

But several different things are easily confused here.

An assistant can use your correction as context for its next answer without changing the underlying model. A product may retain preferences through a memory feature. A provider may also select eligible interactions for later evaluation or training. These are different mechanisms; one is not proof of another.

Whether a particular exchange is retained or used for training depends on the service, its policies, the applicable agreement, and your settings. An assistant saying “I understand” does not tell you which of those mechanisms is involved.

Feedback also needs interpretation. A thumbs-down could mean the answer was incorrect, too long, badly formatted, or simply unwelcome. Asking for another response does not explain what was wrong with the first one.

Even explicit preferences have limits. We can prefer confidence over caution, or agreement over an accurate correction. Research on sycophancy has found that human preference judgments can favor answers that align with a user’s views, and that optimizing for such preferences can sometimes come at the expense of truthfulness.[2]

So the question extends beyond how much feedback AI systems can collect. It includes what those signals actually reward, and how reliably they represent quality.

For me, the uncomfortable part is how invisible the contribution can feel. An interface presents a question box and a helpful answer. The effort spent reviewing, correcting, and explaining is easy to treat as a minor inconvenience, even when it is the most valuable human contribution in the exchange.

There can be a fair exchange here. Users receive assistance, and feedback can help improve a product. But fairness is easier to assess when people understand what they are contributing and have meaningful choices about its use.

I want to distinguish the value of my feedback from assumptions about where it goes. I can recognize that my corrections contain expertise without claiming that every message automatically trains a future model.

The next time an assistant misses something and you explain the mistake, notice what you are adding. You are supplying a standard of judgment: what counts as correct, useful, safe, or appropriate in a particular context.

Whether that judgment becomes training data is a separate question. Its value already exists.

In Part 2: When Everyday Work Becomes AI Data, we will look beyond individual corrections at the knowledge, workflows, and professional standards people bring into AI-assisted work.

References

1. Ouyang et al. (2022). Training language models to follow instructions with human feedback.
2. Anthropic (2023). Towards understanding sycophancy in language models.