How to Design LLM Chatbot Conversation Flows: States, Fallbacks, and Handoffs

Overview

An LLM chatbot conversation flow is the combination of user intents, conversation states, context requirements, actions, and exit conditions that guide an interaction. The language model may generate natural responses, but the surrounding application should still provide structure. Treat the model as one component in a controlled workflow rather than as the entire workflow.

A practical design usually separates five concerns:

• Intent: What is the user trying to accomplish, such as checking an order, changing an appointment, or understanding a policy?
• State: Where is the user in the process, such as identifying an account, collecting a date, confirming a change, or waiting for a tool result?
• Context: Which facts are known, which are missing, and which must not be assumed?
• Action: Should the assistant answer, ask a question, call a tool, refuse, or escalate?
• Exit condition: How will the system know the task is complete, abandoned, unsafe to continue, or ready for a human handoff?

This approach works for customer support bots, internal IT assistants, onboarding flows, and other LLM application development projects. It also makes prompt engineering more useful: prompts can describe the current state and permitted action instead of asking a model to manage an undefined conversation.

A simple state model

A basic chatbot state machine might include START, IDENTIFY_INTENT, COLLECT_DETAILS, CONFIRM_ACTION, EXECUTE_ACTION, ANSWER, RECOVER, HANDOFF, and END. Not every bot needs all of these states, but naming them forces the team to decide what should happen at each stage.

For every state, document the expected input, required context, allowed tools, response style, transition rules, and fallback path. Keep sensitive operations behind explicit confirmation and server-side validation. A model-generated statement should not be treated as proof that an action succeeded; the application should verify the tool result before reporting completion.

Checklist by scenario

1. Starting a new conversation

• Define the bot's primary jobs in one or two sentences.
• Give the assistant a short greeting that sets expectations without listing every possible capability.
• Identify whether the user's first message already contains an intent and usable details.
• Route vague requests to a focused clarification question rather than a broad “How can I help?” loop.
• Provide a visible path to a human or alternate support channel when the bot cannot handle the request.

A useful opening prompt can instruct the assistant to classify the request internally, but the application should retain control over important routing decisions. For example: “Determine whether the request concerns billing, account access, order status, technical support, or an unsupported topic. If the category is uncertain, ask one concise question that distinguishes the two most likely categories.”

2. Collecting information

Use an intent-and-slot plan before writing prompts. A slot is a piece of information needed to complete an intent.

Ask for only the next missing slot when possible. Avoid requesting information already present in the conversation, and do not ask the user to repeat details after a harmless misunderstanding. If multiple values are missing, collect them in a logical order and explain why a sensitive value is needed before requesting it.

3. Answering a question from approved knowledge

• Identify whether the question can be answered from the available knowledge source.
• Pass relevant retrieved context to the model with clear boundaries between instructions and source material.
• Tell the assistant to acknowledge uncertainty when the source does not support an answer.
• Offer a useful next step, such as narrowing the question or connecting with support.
• Record unanswered questions for later review without exposing internal logs to the user.

For knowledge-heavy assistants, a retrieval-augmented generation workflow can help separate search from response generation. The [production-ready RAG guide](https://trainmyai.net/build-production-ready-rag-application-step-by-step) and [RAG application tutorial](https://trainmyai.net/rag-application-tutorial-retrieval-augmented-generation-workflow) provide useful adjacent implementation guidance.

4. Handling ambiguity or a misunderstood request

Design a recovery path rather than relying on a generic fallback. A good recovery response briefly states the interpretation, identifies the missing distinction, and asks one question. For example: “Do you want to update the delivery address for an existing order, or check whether an address can be changed before dispatch?”

After one or two unsuccessful attempts, offer buttons, categories, a short menu, or human assistance. Preserve the conversation summary so the user does not have to restart. If the user changes topics, explicitly close or suspend the previous state and start the new intent.

5. Refusing or limiting a request

Define refusal conditions for unsupported, unsafe, unauthorized, or privacy-sensitive requests. The assistant should avoid inventing a reason, reveal only the explanation that is appropriate for the product, and suggest a legitimate alternative. A refusal is more useful when it includes the next available path: an approved process, general information, or a human review.

6. Handoff to a human agent

Use an AI chatbot handoff workflow when the user asks for a person, the issue requires account authority, repeated recovery has failed, the emotional context is escalating, or the system cannot safely verify the required information. Pass a compact handoff summary containing:

• Conversation identifier and user-provided contact details permitted by your data policy.
• Detected intent and current state.
• Verified facts and unresolved slots.
• Actions already attempted and their results.
• Relevant transcript excerpts, with unnecessary sensitive data removed.
• The reason for escalation and the user's preferred outcome.

Tell the user what will happen next without promising a response time unless your team has defined one. After handoff, prevent the bot from repeatedly answering in parallel unless the human workflow explicitly supports that behavior.

What to double-check

State and context

• Can the system distinguish a new intent from a follow-up to the current intent?
• Are state transitions explicit, or can the model silently skip a required confirmation?
• Is every stored field labeled as user-provided, retrieved, inferred, or unverified?
• Does the context window contain only the information needed for the current decision?

Tools and actions

• Are tool names, inputs, permissions, and failure responses documented?
• Does the application validate tool arguments independently of the model?
• Can an action be retried safely, or could a retry create a duplicate?
• Does the assistant distinguish a successful tool response from a timeout or partial failure?

Privacy and security

• Are sensitive fields minimized, masked, and retained only as long as necessary?
• Does the handoff summary exclude secrets and irrelevant personal information?
• Are authentication and authorization handled by application services rather than by conversational wording?
• Have prompt-injection and untrusted-document scenarios been considered if the bot uses retrieval or tools?

Testing

Build a reusable test script with normal, ambiguous, adversarial, and interrupted conversations. Include cases such as:

1. User states a complete request in the first message.
2. User provides one required slot at a time.
3. User gives conflicting values.
4. User changes topic midway through a task.
5. A tool returns an empty result, error, timeout, or unauthorized response.
6. The user repeats the same request after two failed answers.
7. The user asks for a human immediately.
8. The conversation contains irrelevant or sensitive text.

Score each run for intent accuracy, correct state transition, slot handling, factual grounding, tool-use correctness, recovery quality, and appropriate handoff. The [LLM prompt testing framework](https://trainmyai.net/llm-prompt-testing-evaluation-framework) can help teams turn these scenarios into a repeatable evaluation process.

Common mistakes

• Using one giant system prompt: A long prompt cannot replace explicit state, permissions, validation, and error handling.
• Designing only the happy path: Real users omit details, change their minds, repeat themselves, and use unexpected language.
• Asking several questions at once: A large form-like reply increases the chance that the user answers incompletely.
• Letting the model claim success: Completion should be based on a verified application result, not persuasive wording.
• Making fallback a dead end: “I do not understand” without a category, example, or handoff leaves the user responsible for debugging the bot.
• Hiding the human path: Escalation should be a designed state with a clear summary, not an improvised apology.
• Skipping regression tests: A model, tool, knowledge source, or prompt change can alter behavior in unrelated intents.

For prompt-specific improvements, maintain versioned prompt templates and record which state, model configuration, tools, and knowledge sources produced each result. The guide to [building a team prompt library](https://trainmyai.net/build-maintain-team-prompt-library) offers a practical way to organize this material.

When to revisit

Review the conversation flow before seasonal planning cycles, major onboarding periods, and planned feature launches. Revisit it whenever workflows, tools, model capabilities, knowledge sources, authentication rules, or support policies change. A new intent taxonomy, repeated failed conversation, unexpected handoff pattern, or unresolved evaluation case is also a reason to update the design.

At each review, compare real or safely anonymized conversations with the documented state machine. Add missed intents, tighten ambiguous transitions, remove unnecessary questions, and update the handoff summary. Run the reusable test script after prompt, tool, retrieval, or routing changes, then record which cases improved and which remain unresolved.

Use this final pre-release checklist:

1. List every supported intent and its required slots.
2. Map each intent to states, tools, confirmations, fallbacks, and exit conditions.
3. Define the clarification question for each common ambiguity.
4. Define refusal and human-handoff criteria.
5. Validate tool permissions, inputs, outputs, and retry behavior.
6. Test normal, incomplete, conflicting, interrupted, and failed conversations.
7. Review privacy exposure in stored context and handoff summaries.
8. Version prompts and flow definitions so regressions can be traced.
9. Schedule the next review after the next workflow or tool change.

A reliable LLM chatbot conversation flow is not a single prompt. It is a maintained product workflow that combines conversation design, application controls, prompt engineering, evaluation, and human support. Start with a small set of high-value intents, make every transition explicit, and expand only when the recovery and handoff paths are as clear as the happy path.