CLOUDY podcast | #33 Why the Chatbot Doesn't Understand You: From Prompts to Context Engineering

How has chatbot communication evolved in recent months?

We are moving from classic prompt engineering toward context engineering. This is an expansion into further methods and techniques used to improve or customize a chatbot so it provides better answers. In the past, prompt engineering was so vital that it even existed as a standalone job position.

Today, for simpler tasks, chatbots generally understand what a user expects and can tune their responses accordingly. However, when dealing with something more complex and structured—where we require a specific procedure or a special format—it pays to be much more specific in the instructions.

What is the difference between a chatbot and an AI agent?

A chatbot is designed as a tool for chatting; we give it a question, and it generates a verbal response. An agent has the ability to take actions. For example, when it can search the web—which most modern chatbots can now do—it can already be considered an agent.

These functions are gradually advancing and being integrated into "raw" chatbots. Today, agents can perform tasks such as sending emails, creating charts, or using Deep Research functions, where they search the web, verify information, and create an accurate, structured overview.

What happens in the background when a chatbot uses "Thinking"?

Reasoning arrived with so-called reasoning models. It is based on the "Chain of Thought" technique. With complex tasks, we don't risk the chatbot getting lost; instead, we give it the task step-by-step.

Reasoning models are fine-tuned to have this process integrated natively. When using the "thinking" function, the chatbot breaks the question down into several smaller sub-tasks, completes each one, and then returns the final answer along with a description of the individual steps. It is an excellent technique for achieving a better result.

What is RAG technology and why is it important for companies?

Retrieval-Augmented Generation (RAG) is a way to teach a chatbot new information simply and cheaply. The alternative is fine-tuning, which is a mathematical-computational process that changes the model's internal parameters; it takes time and costs money.

With RAG, we connect an external database (knowledge base) containing documents and images to the chatbot. The model "drills" through them to find the specific information we need. For companies, this is safer and more transparent. In fine-tuning, we cannot say exactly what is inside the model, whereas with RAG, we see exactly which documents the model is working with.

Can we overwhelm a chatbot with information? What is "context drift"?

Today's models have context windows, which represent the maximum allowed range of information (tokens) the chatbot can work with at any given moment. It operates on a "sliding window" principle—as new information is added, the oldest information is deleted.

If you keep a single communication thread going for too long, the chatbot may lose track of earlier information. It can also happen that the model looks at the beginning and the end of the input, while what is in the middle gets lost from its "sight." In such cases, it’s worth starting a new thread or using a memory function, where previous information is compressed to consume fewer resources.

Should people fear for their jobs because of AI agents?

Manufacturers are striving for maximum suitability for everyday tasks, such as creating presentations, documents, or handling messages through app integrations. Nevertheless, there is no risk of AI replacing large numbers of people in the near future. AI can save a significant amount of time—for instance, when creating presentations—but we still cannot rely solely on an agent because errors still occur and outputs must be checked.

What is the best way to verify information from a chatbot?

Do not rely on outputs 100%. They must be verified manually, for example, via a Google search or by consulting an expert in the field. A good technique is to ask the chatbot to explain things more simply or in greater detail.

If it’s a topic the chatbot itself doesn't fully understand, we may notice that certain things don't make logical sense. We can also tell it which terms we already know and ask it to explain new information within that context. This allows it to adapt to our specific knowledge level.

Recommendations from the Aliter Technologies Expert:

Don't be afraid to experiment: The key is to open the interface and try things out. Don't limit yourself to text input; actively test all available functions.

Use advanced tools: Beyond basic chat, try features like Deep Research for in-depth topic exploration, working with attachments, or generating charts and presentations.

Decompose complex tasks: For complicated assignments, it is still effective to break the work into smaller steps to help the chatbot maintain logical structure and accuracy.

Tailor explanations to your knowledge: If you don't understand something, ask the chatbot to explain the topic using terms you already know.

Trust but verify: AI outputs are improving in quality, but you still shouldn't rely on them 100%. Always verify important facts through other sources.

Monitor the context window: If you plan a long discussion, remember that the chatbot may start forgetting older information over time. In those cases, it's better to start a new thread.

You can listen to the full podcast on 👉 Spotify, 👉 Apple podcastoch or watch it on 👉 YouTube.