Kroger Grocery Order From List Photo
Upload a photo of your handwritten or printed grocery list, and the agent will extract the items using OCR, search Kroger for each item to find the best-priced match, add them to your Kroger cart, then send you a notification that your order is ready for checkout.
Remixed from Kroger Grocery Auto-Order from Google Doc (original by firef1ie)
1 remix
Step 1 of 8
Tool1
Upload Grocery List Photo to File Management
Upload the user's grocery list photo with File Management using action 'upload_standard'. The user will provide an image file of a handwritten or printed grocery list. Save the returned file_id and any available URL so it can be passed to the OCR step.
Tool2
Extract Text via OCR
Send the uploaded grocery list image to the OCR agent using the File Management file_id or URL from the previous step. Extract all text from the image, including handwritten and printed items. Return the full extracted text content preserving line breaks and list structure as much as possible.
Prompt3
Parse Grocery Items
Goal: Parse the raw OCR-extracted text into a clean structured list of grocery items | Inputs: Raw text extracted from the grocery list photo via OCR. May contain handwriting artifacts, partial words, or inconsistent formatting. | Outputs: A JSON array of grocery item objects, each with: name (string), quantity (number), unit (string or null), notes (any brand/preference hints from the list) | Constraints: Correct obvious OCR misreadings of common grocery items (e.g. 'mi1k' -> 'milk', 'eg9s' -> 'eggs'). Merge duplicate items by summing quantities. Preserve brand hints or qualifiers (e.g. organic, large, low-fat). Default quantity to 1 if not specified or not legible. Skip any crossed-out or scribbled-over items if detectable. | Success criteria: Every legible item from the OCR text is represented in the output array with at minimum a name and quantity. OCR artifacts are cleaned up into recognizable grocery item names.
Workflow preview
What the agent will follow (tools, prompts, and workflow steps).
1. Call tool: File Management (Upload Grocery List Photo to File Management).
Instructions:
Upload the user's grocery list photo with File Management using action 'upload_standard'. The user will provide an image file of a handwritten or printed grocery list. Save the returned file_id and any available URL so it can be passed to the OCR step.
Parameters:
{"action":"upload_standard"}
2. Call tool: Document OCR Agent (Extract Text via OCR).
Instructions:
Send the uploaded grocery list image to the OCR agent using the File Management file_id or URL from the previous step. Extract all text from the image, including handwritten and printed items. Return the full extracted text content preserving line breaks and list structure as much as possible.
3. Apply the following prompt: Goal:
Parse the raw OCR-extracted text into a clean structured list of grocery items
Inputs:
Raw text extracted from the grocery list photo via OCR. May contain handwriting artifacts, partial words, or inconsistent formatting.
Outputs:
A JSON array of grocery item objects, each with: name (string), quantity (number), unit (string or null), notes (any brand/preference hints from the list)
Constraints:
Correct obvious OCR misreadings of common grocery items (e.g. 'mi1k' -> 'milk', 'eg9s' -> 'eggs'). Merge duplicate items by summing quantities. Preserve brand hints or qualifiers (e.g. organic, large, low-fat). Default quantity to 1 if not specified or not legible. Skip any crossed-out or scribbled-over items if detectable.
Success criteria:
Every legible item from the OCR text is represented in the output array with at minimum a name and quantity. OCR artifacts are cleaned up into recognizable grocery item names.
4. For each grocery_item, complete the steps in this section once. Iterate over each grocery item parsed from the list. For each item, search Kroger, select the best match, and add it to the cart. Track whether each item was successfully added or not found.
5. Call tool: Grocery Shopping - Kroger (Search Kroger for Item).
Instructions:
Search Kroger for the current grocery_item. Use the item name plus any brand/qualifier hints as the search query. Retrieve at least 5 product results. Include pricing and availability data for each result. The goal is to find in-stock options so we can compare prices.
6. Apply the following prompt: Goal:
Select the best product from Kroger search results for the current grocery item
Inputs:
The current grocery_item (name, quantity, unit, notes) and the list of Kroger search results with prices and availability
Outputs:
A single selected product object with its Kroger product ID, name, price, and quantity to add. Also output a boolean 'found' field (true if a good match exists, false if nothing suitable was found). If not found, output a 'not_found_reason' string.
Constraints:
Prioritize: 1) Correct item type - must actually be the right product, 2) In-stock availability, 3) Lowest price among suitable matches. Respect brand preferences if specified in the item notes. Do not select a wrong item just because it is cheap.
Success criteria:
The selected product is genuinely the right item, is in stock, and is the best available price for that item
7. Call tool: Grocery Shopping - Kroger (Add Item to Kroger Cart).
Instructions:
If the previous step found a suitable product (found=true), add it to the user's Kroger cart using the selected product ID and the quantity from the grocery list. If found=false, skip this step and log the item as not found. Track all cart additions and all skipped items for the final summary.
8. Call tool: AgentPMT Send Human Request (Send Completion Notification).
Instructions:
Send the user a notification that their Kroger grocery order is ready in their cart. Include in the message: the total number of items added to cart, the estimated total cost, a list of any items that could not be found on Kroger so the user can add them manually, and a note about any OCR items that were unclear or may need verification. The message should be friendly and actionable.