Kroger Grocery Order From List Photo

Workflow preview

What the agent will follow (tools, prompts, and workflow steps).

1. Call tool: File Storage - 10MB or less (Upload Grocery List Photo).

Instructions:
Upload the user's grocery list photo to file storage. The user will provide an image file (photo of a handwritten or printed grocery list). Store the file and return the file storage ID and URL so it can be passed to the OCR step.

2. Call tool: Document OCR Agent (Extract Text via OCR).

Instructions:
Send the uploaded grocery list image to the OCR agent using the file storage ID or URL from the previous step. Extract all text from the image, including handwritten and printed items. Return the full extracted text content preserving line breaks and list structure as much as possible.

3. Apply the following prompt: Goal:
Parse the raw OCR-extracted text into a clean structured list of grocery items

Inputs:
Raw text extracted from the grocery list photo via OCR. May contain handwriting artifacts, partial words, or inconsistent formatting.

Outputs:
A JSON array of grocery item objects, each with: name (string), quantity (number), unit (string or null), notes (any brand/preference hints from the list)

Constraints:
Correct obvious OCR misreadings of common grocery items (e.g. 'mi1k' -> 'milk', 'eg9s' -> 'eggs'). Merge duplicate items by summing quantities. Preserve brand hints or qualifiers (e.g. organic, large, low-fat). Default quantity to 1 if not specified or not legible. Skip any crossed-out or scribbled-over items if detectable.

Success criteria:
Every legible item from the OCR text is represented in the output array with at minimum a name and quantity. OCR artifacts are cleaned up into recognizable grocery item names.

4. For each grocery_item, complete the steps in this section once. Iterate over each grocery item parsed from the list. For each item, search Kroger, select the best match, and add it to the cart. Track whether each item was successfully added or not found.

5. Call tool: Grocery Shopping - Kroger (Search Kroger for Item).

Instructions:
Search Kroger for the current grocery_item. Use the item name plus any brand/qualifier hints as the search query. Retrieve at least 5 product results. Include pricing and availability data for each result. The goal is to find in-stock options so we can compare prices.

6. Apply the following prompt: Goal:
Select the best product from Kroger search results for the current grocery item

Inputs:
The current grocery_item (name, quantity, unit, notes) and the list of Kroger search results with prices and availability

Outputs:
A single selected product object with its Kroger product ID, name, price, and quantity to add. Also output a boolean 'found' field (true if a good match exists, false if nothing suitable was found). If not found, output a 'not_found_reason' string.

Constraints:
Prioritize: 1) Correct item type - must actually be the right product, 2) In-stock availability, 3) Lowest price among suitable matches. Respect brand preferences if specified in the item notes. Do not select a wrong item just because it is cheap.

Success criteria:
The selected product is genuinely the right item, is in stock, and is the best available price for that item

7. Call tool: Grocery Shopping - Kroger (Add Item to Kroger Cart).

Instructions:
If the previous step found a suitable product (found=true), add it to the user's Kroger cart using the selected product ID and the quantity from the grocery list. If found=false, skip this step and log the item as not found. Track all cart additions and all skipped items for the final summary.

8. Call tool: AgentPMT Send Human Request (Send Completion Notification).

Instructions:
Send the user a notification that their Kroger grocery order is ready in their cart. Include in the message: the total number of items added to cart, the estimated total cost, a list of any items that could not be found on Kroger so the user can add them manually, and a note about any OCR items that were unclear or may need verification. The message should be friendly and actionable.