Multimodal prompt engineering helps AI systems work with more than plain text, but reliable results depend on how you prepare the input and define the task. This guide provides a reusable workflow for prompting AI with images, screenshots, and PDF documents, including practical templates, scenario checklists, verification steps, and troubleshooting advice.
Overview
Vision-capable language models can interpret visual information, extract text, compare layouts, summarize documents, and answer questions about images. They are useful for tasks such as reviewing screenshots, classifying visual content, analyzing forms, locating values in reports, and turning scanned documents into structured notes.
However, an image or PDF does not automatically communicate your intended objective. A model may identify the visible content correctly but answer the wrong question, overlook a small field, confuse document order, or treat an uncertain reading as fact. Good multimodal prompt engineering reduces these problems by making four things explicit:
- The input: what the attached image, page, or document represents.
- The task: what the model should inspect, extract, compare, or explain.
- The output: the required format, level of detail, and ordering.
- The uncertainty rule: what to do when content is unclear, missing, cropped, or unreadable.
Capabilities, supported file types, image limits, document handling, and SDK interfaces vary by model and provider. Treat those details as implementation settings to verify before deployment rather than assumptions to embed permanently in a prompt.
Checklist by scenario
1. Before sending an image
- Confirm that the relevant content is visible and not covered by menus, overlays, or compression artifacts.
- Crop unrelated areas when they could distract the model or expose unnecessary sensitive information.
- Keep important text large enough to inspect. If a label is difficult for a person to read, it may also be difficult for the model.
- Use the clearest available image and preserve the original orientation.
- Tell the model whether it should analyze the whole image or a particular region.
- Specify whether you need transcription, classification, explanation, comparison, or defect detection.
2. Prompting with screenshots
Screenshot analysis works best when the prompt supplies context that is not visible in the pixels. A screenshot of an error message, dashboard, or user interface may contain several plausible points of focus.
Use a prompt such as:
Role: You are assisting with technical screenshot analysis.
Context: This screenshot shows a web application after a user submitted a form.
Task:
1. Identify any visible error messages.
2. Quote the exact error text when readable.
3. Describe the interface area where each message appears.
4. Separate observations from possible explanations.
Output:
- Observed messages
- Location in the interface
- Likely causes, clearly labeled as hypotheses
- Recommended next checks
If text is partially obscured or unreadable, say so instead of guessing.
For a UI review, ask the model to describe evidence before suggesting a fix. This makes it easier to distinguish what the screenshot shows from what the model infers.
3. AI prompts for PDF analysis
PDFs can contain selectable text, scanned pages, tables, diagrams, headers, footnotes, and multiple columns. Before writing the prompt, establish whether the task concerns the full document, selected pages, or a specific section. If page numbers are available, use them in the request.
A reusable document-analysis prompt is:
Analyze the attached document for the purpose of [purpose].
Scope: Review pages [page range] and focus on [sections or fields].
Tasks:
1. Summarize the main points in no more than [length].
2. Extract the following fields: [field list].
3. Preserve units, dates, names, and document terminology exactly where readable.
4. Identify contradictions, missing values, and ambiguous text.
Output the result as:
- Summary
- Extracted fields
- Page references
- Uncertainty or verification notes
Do not infer values that are not present in the document.
For long documents, process logical sections instead of asking one prompt to perform every task at once. A staged workflow might extract text and page references first, summarize selected sections second, and run a consistency check last. This approach also creates intermediate outputs that can be reviewed or tested.
4. Comparing two or more images
When comparing images, define the comparison criteria and require a structured result. Otherwise, the model may focus on visually prominent but irrelevant differences.
Compare Image A and Image B for these criteria: [criteria].
For each criterion, report:
- Image A observation
- Image B observation
- Difference: none, minor, or substantial
- Evidence or location
- Confidence: high, medium, or low
Do not report differences caused only by cropping, scale, or image quality unless they affect the requested criteria.
5. Common visual tasks
| Task | Prompt emphasis | Output to request |
|---|---|---|
| OCR or transcription | Exact visible text, reading order, unclear characters | Verbatim text with uncertainty markers |
| Screenshot diagnosis | Visible evidence before possible causes | Observations, hypotheses, next checks |
| Form extraction | Field names, page locations, missing values | Structured fields with page references |
| Table analysis | Headers, rows, units, merged cells | Normalized table plus caveats |
| Image classification | Defined labels and borderline cases | Label, evidence, and confidence |
| Document summarization | Scope, audience, length, key sections | Summary organized by topic or page |
What to double-check
Input quality and layout
Inspect the original input before changing the prompt. Check resolution, rotation, contrast, cropping, page order, and whether small text or table columns remain legible. For PDFs, determine whether pages are text-based, scanned, or mixed. A poor extraction result may be an input problem rather than a prompt problem.
Output constraints
State the required schema, ordering, units, and treatment of missing values. If the result feeds another application, request structured output and validate it in code rather than trusting visual formatting alone. Related guidance on schemas and error handling is available in Structured Output Prompts: JSON Schema Examples and Error-Handling Patterns for LLMs.
Evidence and uncertainty
Ask for page numbers, regions, quoted text, or other evidence whenever the result will support a decision. Use explicit instructions such as “say unknown,” “mark as unreadable,” or “separate observation from inference.” Confidence labels can help triage review, but they are not a substitute for checking the original input.
Privacy and access
Before sending a document or screenshot to an AI service, review the data-handling requirements that apply to your workflow. Remove unnecessary personal, confidential, or credential-related information where possible. Avoid placing secrets in screenshots, and confirm that your selected tool, account configuration, and retention settings are appropriate for the material.
Evaluation
Test prompts against representative easy, difficult, and ambiguous examples. Include rotated pages, low-quality scans, dense tables, missing fields, multiple languages, and misleading visual context when those cases occur in production. Track extraction accuracy, unsupported claims, formatting failures, and refusal to guess. A repeatable test set is more useful than judging one impressive response; see the LLM Evaluation Test Suite for a broader evaluation workflow.
Common mistakes
- Asking “What is in this image?” for a precise task: Replace the broad question with a defined objective, scope, and output format.
- Using one prompt for unrelated operations: Separate transcription, interpretation, and recommendations when each requires different verification.
- Ignoring reading order: Tell the model how to handle columns, tables, captions, footnotes, and page sequence.
- Requesting exact values without an uncertainty rule: Instruct the model to mark unclear characters and preserve missing values rather than complete them.
- Treating inferred content as observed fact: Require separate sections for visible evidence and possible explanations.
- Assuming all PDFs behave the same way: Check whether the file contains selectable text, scanned images, embedded tables, or unusual layouts.
- Failing to protect the prompt and workflow: Images and documents can contain instructions intended to manipulate an AI system. Treat embedded text as data to analyze, not as authority to follow, and apply a documented input-safety process. The prompt injection prevention checklist provides related safeguards.
- Changing models without regression testing: Re-run the same visual test cases whenever the model, file-processing path, prompt, or output schema changes.
When to revisit
Keep this checklist with the workflow rather than treating it as a one-time setup document. Revisit it before a new project, before seasonal planning or reporting cycles, and whenever the source documents or screenshots change materially.
Update the implementation notes when a provider changes supported file types, document limits, image handling, vision capabilities, SDK examples, or privacy controls. Those details can change independently of the prompt itself. Recheck them during quarterly maintenance, after a model migration, and after any change to preprocessing or file conversion.
Use this practical review sequence:
- Collect a small set of current representative images and documents.
- Confirm that the input format, resolution, page order, and preprocessing still work.
- Run the existing prompt and record extraction, interpretation, and formatting errors.
- Revise one variable at a time: input preparation, instructions, schema, or model configuration.
- Compare the revised output with a human-checked reference set.
- Document known failure cases and add them to regression testing.
- Review privacy, access, and retention settings before moving the workflow into regular use.
Reliable multimodal AI development is less about finding a magical phrase and more about building a controlled loop: prepare the input, define the task, constrain the output, expose uncertainty, and test against real examples. That process makes image prompts, PDF analysis, and screenshot workflows easier to maintain as tools and inputs evolve.