267 lines
13 KiB
Plaintext
267 lines
13 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Buy the highest rated product from the meat substitute category within a budget between 100 and 200.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the homepage of "One Stop Market" visible. The page displays various product categories and a search bar at the top.
|
||
- **Page Changes:** No changes occur as this is the starting point.
|
||
- **Possible Purpose:** The initial state sets the context for the subsequent actions.
|
||
|
||
#### 2. **Hovering Over "Grocery & Gourmet Food" Menu**
|
||
- **Action:** I move the cursor over the "Grocery & Gourmet Food" menu item in the top navigation bar.
|
||
- **Page Changes:** A dropdown menu appears, listing subcategories under "Grocery & Gourmet Food."
|
||
- **Possible Purpose:** The intent is likely to explore specific subcategories within "Grocery & Gourmet Food" to find a particular type of product.
|
||
|
||
#### 3. **Clicking on "Meat Substitutes" Subcategory**
|
||
- **Action:** I click on the "Meat Substitutes" option from the dropdown menu.
|
||
- **Page Changes:** The webpage transitions to a new section titled "Meat Substitutes," displaying products related to meat substitutes.
|
||
- **Possible Purpose:** The goal is to view and possibly purchase products that are alternatives to traditional meat.
|
||
|
||
#### 4. **Observing the "Meat Substitutes" Product Page**
|
||
- **Action:** I scroll slightly to view the products listed under "Meat Substitutes."
|
||
- **Page Changes:** The page reveals a grid of products with images, names, prices, and customer ratings.
|
||
- **Possible Purpose:** The intent is to browse through the available meat substitute products to evaluate options.
|
||
|
||
#### 5. **Sorting the Products**
|
||
- **Action:** I click on the "Sort By" dropdown menu located at the top right of the product grid.
|
||
- **Page Changes:** A list of sorting options appears, such as "Position," "Name," "Price," etc.
|
||
- **Possible Purpose:** The goal is to organize the product display to facilitate easier browsing or decision-making.
|
||
|
||
#### 6. **Selecting "Position" from the Sort Options**
|
||
- **Action:** I select "Position" from the sorting dropdown menu.
|
||
- **Page Changes:** The products are rearranged based on their default position setting.
|
||
- **Possible Purpose:** The intent is to view the products in their default order, which might highlight featured or popular items.
|
||
|
||
### Summary
|
||
In this video segment, I navigate from the homepage to the "Meat Substitutes" section by interacting with the "Grocery & Gourmet Food" menu. I then browse the products and adjust the sorting to "Position" to better organize the displayed items. Each action is focused on exploring and potentially selecting meat substitute products.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Adding Item to Cart**
|
||
- **Action:** I click on the "Add to Cart" button for the "Beyond Meat Beefy Crumble, 5 Pound -- 2 per case" product.
|
||
- **Page Changes:** A notification appears at the top of the page confirming that the item has been added to the cart. The cart icon updates to show 1 item with a total value of $108.27.
|
||
- **Possible Purpose:** The purpose is to add the selected product to the shopping cart for potential purchase.
|
||
|
||
#### 2. **Navigating to the Cart**
|
||
- **Action:** I click on the cart icon located in the top right corner of the page.
|
||
- **Page Changes:** The webpage transitions to display the cart contents, showing the recently added item along with its price and quantity.
|
||
- **Possible Purpose:** The intent is to review the items currently in the cart before proceeding to checkout.
|
||
|
||
#### 3. **Proceeding to Checkout**
|
||
- **Action:** I click on the "Proceed to Checkout" button within the cart summary.
|
||
- **Page Changes:** The page navigates to the checkout section, starting with the "Shipping" step. Here, the shipping address is pre-filled with the details: "Emma Lopez, 111 Magnolia Path, Atlanta, Georgia 30303, United States."
|
||
- **Possible Purpose:** The goal is to move forward in the purchasing process by providing necessary shipping information.
|
||
|
||
#### 4. **Confirming Shipping Information**
|
||
- **Action:** I review the pre-filled shipping address and confirm it is correct. No changes are made to the address.
|
||
- **Page Changes:** The page remains on the "Shipping" step, with the address details visible and a "Next" button available to proceed.
|
||
- **Possible Purpose:** The purpose is to ensure the shipping address is accurate before moving to the next step in the checkout process.
|
||
|
||
#### 5. **Moving to Review & Payments**
|
||
- **Action:** I click the "Next" button to advance from the "Shipping" step to the "Review & Payments" step.
|
||
- **Page Changes:** The page transitions to the "Review & Payments" section. Here, the order summary is displayed on the right side, showing the cart subtotal ($108.27), shipping cost ($5.00), and order total ($113.27). The payment method section is shown with "Check / Money order" selected as the default option.
|
||
- **Possible Purpose:** The intent is to review the order details and select a payment method before finalizing the purchase.
|
||
|
||
#### 6. **Placing the Order**
|
||
- **Action:** I click the "Place Order" button at the bottom of the "Review & Payments" section.
|
||
- **Page Changes:** After clicking, a loading spinner appears, indicating the order is being processed. The button text changes to "Please wait..." to prevent multiple submissions.
|
||
- **Possible Purpose:** The final action is to submit the order for processing, completing the purchase.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to add a product to the cart, navigate to the cart, proceed to checkout, confirm shipping details, review the order, and finally place the order. Each step is methodically executed to ensure the purchase process is completed accurately.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "menuitem_Meat Substitutes",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Meat Substitutes\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_$100.00 - $199.99( 4 item )",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"$100.00 - $199.99( 4 item )\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/grocery-gourmet-food/meat-substitutes.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/grocery-gourmet-food/meat-substitutes.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Beyond Meat Beef Beefy Crumble, 5 Pound -- 2 per case.",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Beyond Meat Beef Beefy Crumble, 5 Pound -- 2 per case.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Add to Cart",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Add to Cart\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue611 My Cart 1 1 items",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue611 My Cart 1 1 items\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Proceed to Checkout",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Proceed to Checkout\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Next",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Next\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#shipping"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#shipping"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Item in Cart",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_text(\"Item in Cart\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#shipping"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#payment"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Place Order",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Place Order\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#payment"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/checkout/#payment"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |