1 line
12 KiB
JSON
1 line
12 KiB
JSON
{"messages": [{"role": "system", "content": "\nImagine you are an agent browsing the web, just like humans. Now you need to complete a task. In each iteration, you will receive an observation that includes the accessibility tree of the webpage and a screenshot of the current viewpoint. The accessbility tree contains information about the web elements and their properties. The screenshot will feature numerical labels placed in the TOP LEFT corner of web elements in th current viewpoint. Carefully analyze the webpage information to identify the numerical label corresponding to the web element that requires interaction, then follow the guidelines and choose one of the following actions:\n1. Click a web element.\n2. Delete existing content in a textbox and then type content.\n3. Scroll up or down the whole window.\n4. Go back, returning to the previous webpage.\n5. Answer. This action should only be chosen when all questions in the task have been solved.\n\nCorrespondingly, action should STRICTLY follow the format specified by one of the following lines:\nYour action should be readable, simple, and only **ONE-LINE-OF-CODE** at a time, avoid using loop statement and only use if-else control if necessary. Predefined action functions are as follow:\n\n```\ndef do(action, argument, element):\n \"\"\"A single browsing operation on the webpage.\n Args:\n :param action: one of the actions from [\"Click\", \"Right Click\", \"Type\", \"Search\", \"Hover\", \"Scroll Up\", \"Scroll Down\", \"Press Enter\", \"Switch Tab\", \"Select Dropdown Option\", \"Wait\"].\n :param argument: optional. Only for \"Type\", \"Search\", \"Switch Page\", and \"Select Dropdown Option\", indicating the content to type in, page number(start from 0) to switch, or key to press.\n \"Search\" action is equivalent to \"Type\" action plus \"Enter\" key press.\n :param element: optional. Only for \"Click\", \"Right Click\", \"Type\", \"Search\", \"Select Dropdown Option\", and \"Hover\". Should be specific element id in the html.\n Returns:\n None. The webpage will be updated after executing the action.\n \"\"\"\n\ndef exit(message):\n \"\"\"Ending the browsing process if the assistant think it has fulfilled the goal.\n Args:\n :param message: optional. If user's instruction is a question, return assistant's answer in the message based on the browsing content.\n Returns:\n None.\n \"\"\"\n\ndef go_backward():\n \"\"\"Go back to the previous page.\n \"\"\"\n\ndef go_forward():\n \"\"\"Go forward to the next page.\n \"\"\"\n```\n\nHere are some examples:\n- # Element: the 'REPORTS' section on the left sidebar\ndo(action=\"Click\", element=\"7\")\n- # Element: the 'Period' dropdown, middle center\ndo(action=\"Select Dropdown Option\", argument=\"Month\", element=\"20\")\n- # Element: the 'From' date picker input field, middle center\ndo(action=\"Type\", argument=\"01/01/2023\", element=\"22\")\n- do(action=\"Scroll Down\")\n- exit(message=\"The top-3 best-selling products in January 2023 are: 1\")\n- # Element: The search bar\ndo(action=\"Search\", argument=\"international airport near Carnegie Mellon University within a driving distance of 50 km\", element=\"13\")\n- # Note: Pittsburgh International Airport, Southern Beltway, Findlay Township, Allegheny County, 15231, United States\n# Element: The field labeled 'Pittsburgh International Airport' in the top left corner\ndo(action=\"Type\", argument=\"Cleveland Hopkins International Airport\", element=\"14\")\n\n\nKey guidelines you MUST follow:\n* Action guidelines \n- Use either screenshot or accessibility tree to obtain the numerical_label. Sometimes the accessibility tree captures more elements than the screenshot. It's safe to select these elements without scrolling\n- For text input, use Type action directly (no need to click first). All existing texts in the textbox will be deleted automatically before typing\n- You must not repeat the same actions over and over again. If the same action doesn't work, try alternative approaches\n- Use ANSWER only after completing ALL task requirements\n- When answer not found after thorough search, use N/A\n- Wrap content for Type and ANSWER with square brackets []\n- Preserve text inside quotation marks exactly as provided\n\n\nYour reply should strictly follow the format:\nThought: Your reasoning trace. A good practice is to summarize information on the current web page that are relevant to the task goal, then generate a high-level plan that contains the sequence of actions you probably need to take\nAction: Based on this reasoning, identify the single most optimal action. You should output it in the format specified above (under \"STRICTLY follow the format\")\n\nAfter each action, you'll receive a new observation. Proceed until task completion. Website Note \"- Always save progress through appropriate buttons (Save, Submit, Post, etc.)\n- Always remember to interact with dropdown options after expanding\n- Clear filters before setting new ones\n- Use date format: month/day/year (e.g., 1/1/16, 12/31/24)\n- When searching phone numbers, remove the country code\n- When searching product name, use single but not plural form\n- When the web page contains a table, aggregate the rows with the same item\n- Use the \"ID\" field in filters to search order ID \n- To change the page title, click on the title gridcell \n- To modify product attributes like size and color, go to \"Configuration\" after landing on the product page \n- Type in the \"Product\" field to search reviews for a product \n- When the task asks for most frequently appeared items, return a list of all items shown up during search \n- When there is no item after filtering, conclude that the task is complete and return N/A \n- Do not go to Advanced Reporting\n\nIMPORTANT NOTICES:\n- do not give up in early round, try to explore as much as you can\n- The answer to the task may not in the current html page, you should first naviate to the most relavant page first!\n\nNow solve the following task.\nfind the items on discount. "}, {"role": "user", "content": "<html data-bbox=\"0,0,1280,2453\"> <body data-bbox=\"0,0,1280,2453\"> <div data-bbox=\"0,0,1280,2453\"> <header data-bbox=\"0,0,1280,128\"> <div data-bbox=\"0,0,1280,40\"> <ul data-bbox=\"682,10,578,20\"> <a id=\"0\" data-bbox=\"697,10,78,19\"> My Account </a> <a id=\"1\" data-bbox=\"790,10,84,19\"> My Wish List </a> <li data-label=\"or\" data-bbox=\"889,10,46,20\"> <a id=\"2\" data-bbox=\"889,10,46,19\"> Sign In </a> </li> <span data-bbox=\"965,10,157,19\"> Welcome, Emma Lopez! </span> <a id=\"3\" data-bbox=\"1137,10,123,19\"> Create an Account </a> </ul> <span data-bbox=\"19,9,28,19\"> Skip to Content </span> </div> <div data-bbox=\"0,41,1280,87\"> <a title=\"one_stop_market_logo\" data-bbox=\"20,71,200,27\"> <img id=\"4\" title=\"one_stop_market_logo\" data-bbox=\"20,71,200,27\"> </img> </a> <div data-bbox=\"1197,71,63,33\"> <a data-bbox=\"1197,71,63,33\"> <span data-bbox=\"1235,70,1,1\"> My Cart </span> <span data-bbox=\"1236,74,24,24\"> <span id=\"5\" data-bbox=\"1244,76,8,19\"> 3 </span> <span data-bbox=\"1251,73,1,1\"> 3 <span data-bbox=\"1251,99,37,19\"> items </span> </span> </span> </a> </div> <div data-bbox=\"919,71,265,57\"> <form data-bbox=\"934,71,250,57\"> <div data-bbox=\"934,71,250,57\"> <span data-bbox=\"932,102,1,1\"> Search </span> <div data-bbox=\"934,71,250,57\"> <input id=\"6\" type=\"text\" placeholder=\"Search entire store here...\" data-bbox=\"934,71,250,32\"> </input> <a id=\"7\" data-bbox=\"934,108,116,19\"> Advanced Search </a> </div> </div> <button title=\"Search\" type=\"submit\" data-bbox=\"1152,71,22,32\"> <span data-bbox=\"1173,75,1,1\"> Search </span> </button> </form> </div> </div> </header> <div data-bbox=\"0,128,1280,106\"> <ul data-bbox=\"0,128,1280,106\"> <li data-bbox=\"8,128,194,53\"> <span id=\"8\" data-bbox=\"190,128,12,53\"> </span> <span id=\"9\" data-bbox=\"20,145,162,19\"> Beauty & Personal Care </span> </li> <li data-bbox=\"212,128,162,53\"> <span id=\"10\" data-bbox=\"362,128,12,53\"> </span> <span id=\"11\" data-bbox=\"224,145,130,19\"> Sports & Outdoors </span> </li> <li data-bbox=\"384,128,209,53\"> <span id=\"12\" data-bbox=\"581,128,12,53\"> </span> <span id=\"13\" data-bbox=\"396,145,177,19\"> Clothing, Shoes & Jewelry </span> </li> <li data-bbox=\"603,128,145,53\"> <span id=\"14\" data-bbox=\"736,128,12,53\"> </span> <span id=\"15\" data-bbox=\"615,145,113,19\"> Home & Kitchen </span> </li> <li data-bbox=\"758,128,139,53\"> <span id=\"16\" data-bbox=\"885,128,12,53\"> </span> <span id=\"17\" data-bbox=\"770,145,107,19\"> Office Products </span> </li> <li data-bbox=\"907,128,229,53\"> <span id=\"18\" data-bbox=\"1124,128,12,53\"> </span> <span id=\"19\" data-bbox=\"919,145,197,19\"> Tools & Home Improvement </span> </li> <li data-bbox=\"8,181,172,53\"> <span id=\"20\" data-bbox=\"168,181,12,53\"> </span> <span id=\"21\" data-bbox=\"20,198,140,19\"> Health & Household </span> </li> <li data-bbox=\"190,181,182,53\"> <span id=\"22\" data-bbox=\"360,181,12,53\"> </span> <span id=\"23\" data-bbox=\"202,198,150,19\"> Patio, Lawn & Garden </span> </li> <li data-bbox=\"382,181,107,53\"> <span id=\"24\" data-bbox=\"477,181,12,53\"> </span> <span id=\"25\" data-bbox=\"394,198,75,19\"> Electronics </span> </li> <li data-bbox=\"499,181,211,53\"> <span id=\"26\" data-bbox=\"698,181,12,53\"> </span> <span id=\"27\" data-bbox=\"511,198,179,19\"> Cell Phones & Accessories </span> </li> <li data-bbox=\"720,181,122,53\"> <span id=\"28\" data-bbox=\"830,181,12,53\"> </span> <span id=\"29\" data-bbox=\"732,198,90,19\"> Video Games </span> </li> <li data-bbox=\"852,181,206,53\"> <span id=\"30\" data-bbox=\"1046,181,12,53\"> </span> <span id=\"31\" data-bbox=\"864,198,174,19\"> Grocery & Gourmet Food </span> </li> </ul> </div> <main data-bbox=\"0,259,1280,1884\"> <span data-bbox=\"20,254,304,54\"> One Stop Market </span> <div data-bbox=\"20,343,1240,1800\"> <div data-bbox=\"20,343,1240,1760\"> <strong data-bbox=\"20,340,161,24\"> Product Showcases </strong> <div data-bbox=\"20,379,1240,1724\"> <ol data-bbox=\"20,379,1240,1656\"> <img id=\"32\" data-bbox=\"20,379,228,285\"> </img> <div data-bbox=\"20,674,228,190\"> <a id=\"33\" title=\"Pre-baked Gingerbread House Kit Value Pack, 17 oz., Pack of 2, Total 34 oz.\" data-bbox=\"20,674,225,19\"> Pre-baked Gingerbread House Kit Value Pack, 17 oz., Pack of 2, Total 34 oz. </a> </div> <img id=\"34\" data-bbox=\"273,379,228,285\"> </img> <div data-bbox=\"273,674,228,230\"> <a id=\"35\" title=\"V8 +Energy, Healthy Energy Drink, Steady Energy from Black and Green Tea, Pomegranate Blueberry, 8 Ounce Can ,Pack of 24\" data-bbox=\"273,674,224,19\"> V8 +Energy, Healthy Energy Drink, Steady Energy from Black and Green Tea, Pomegranate Blueberry, 8 Ounce Can ,Pack of 24 </a> </div> <img id=\"36\" data-bbox=\"526,379,228,285\"> </img> <div data-bbox=\"526,674,228,190\"> <a id=\"37\" title=\"Elmwood Inn Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion, 16-Ounce Pouch\" data-bbox=\"526,674,210,19\"> Elmwood Inn Fine Teas, Orange Vanilla Caffeine-free Fruit Infusion, 16-Ounce Pouch </a> </div> <img id=\"38\" data-bbox=\"779,379,228,285\"> </img> <div data-bbox=\"779,674,228,270\"> <a id=\"39\" title=\"Belle Of The Ball Princess Sprinkle Mix| Wedding Colorful Sprinkles| Cake Cupcake Cookie Sprinkles| Ice cream Candy Sprinkles| Yellow Gold Red Royal Red Rose Icing Flowers Decorating Sprinkles, 8OZ\" data-bbox=\"779,674,171,19\"> Belle Of The Ball Princess Sprinkle Mix| Wedding Colorful Sprinkles| Cake Cupcake Cookie Sprinkles| Ice cream Candy Sprinkles| Yellow Gold Red Royal Red Rose Icing Flowers Decorating Sprinkles, 8OZ </a> </div> <img id=\"40\" data-bbox=\"1032,379,228,285\"> </img> <div data-bbox=\"1032,674,228,190\"> <a id=\"41\" title=\"So Delicious Dairy Free CocoWhip Light, Vegan, Non-GMO Project Verified, 9 oz. Tub\" data-bbox=\"1032,674,224,19\"> So Delicious Dairy Free CocoWhip Light, Vegan, Non-GMO Project Verified, 9 oz. Tub </a> </div> </ol> </div> </div> </div> </main> </div> </body> </html>"}, {"role": "assistant", "content" : "# Note: There are no function about it.\nexit(message=\"N/A\")"}]} |