239 lines
12 KiB
Plaintext
239 lines
12 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
List products from living room furtniture category by descending price
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the webpage of "One Stop Market" fully loaded, specifically on the "Health & Household" section under "Product Showcases."
|
||
- **Page Changes:** No action has been taken yet, so the page remains static.
|
||
- **Possible Purpose:** The initial state sets the context for the subsequent actions.
|
||
|
||
#### 2. **Hovering Over "One Stop Market" Text**
|
||
- **Action:** I move the cursor over the text "One Stop Market" located at the top of the page.
|
||
- **Page Changes:** There is no visible change to the webpage; it remains the same.
|
||
- **Possible Purpose:** Hovering might be a preparatory action, possibly to ensure the correct element is targeted before interacting with it.
|
||
|
||
#### 3. **Clicking on "One Stop Market" Text**
|
||
- **Action:** I click on the text "One Stop Market" at the top of the page.
|
||
- **Page Changes:** The page reloads or redirects to the main "One Stop Market" homepage or a general product listing page.
|
||
- **Possible Purpose:** The intent is likely to navigate back to the main page of the website or to refresh the current view to a more general product listing.
|
||
|
||
#### 4. **Observing the New Page State**
|
||
- **Action:** After clicking, I observe the new state of the page.
|
||
- **Page Changes:** The page now displays a broader selection of products under the "One Stop Market" branding, with various categories and items showcased.
|
||
- **Possible Purpose:** The purpose is to review the updated content, which now includes a wider range of products and categories.
|
||
|
||
### Summary
|
||
- **Actions Taken:** Hovered over and then clicked the "One Stop Market" text at the top of the page.
|
||
- **Resulting Changes:** Navigated from the "Health & Household" section to a more general "One Stop Market" page with a broader product selection.
|
||
- **Likely Purposes:** To return to a main or general view of the website for either broader browsing or to reset the navigation context.
|
||
|
||
This description strictly adheres to the observable actions within the provided video segment without incorporating any external context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Part 1: Navigation from "Health & Household" to "Living Room Furniture"
|
||
|
||
#### Step 1: Hover over the "Clothing, Shoes & Jewelry" menu
|
||
- **Action**: I hover my cursor over the "Clothing, Shoes & Jewelry" menu in the top navigation bar.
|
||
- **Page Changes**: A dropdown menu appears, displaying various subcategories such as "Women," "Men," "Novelty & More," etc.
|
||
- **Possible Purpose**: The likely intent is to explore the subcategories under "Clothing, Shoes & Jewelry," although this is not the final destination.
|
||
|
||
#### Step 2: Move cursor away from "Clothing, Shoes & Jewelry"
|
||
- **Action**: I move my cursor away from the "Clothing, Shoes & Jewelry" menu.
|
||
- **Page Changes**: The dropdown menu disappears.
|
||
- **Possible Purpose**: This action suggests a change in decision or accidental hover, redirecting focus elsewhere on the page.
|
||
|
||
#### Step 3: Hover over the "Electronics" menu
|
||
- **Action**: I hover my cursor over the "Electronics" menu in the top navigation bar.
|
||
- **Page Changes**: A dropdown menu appears, showing subcategories like "Cell Phones & Accessories," "Cases, Holsters & Sleeves," "Sport Specific Clothing," etc.
|
||
- **Possible Purpose**: The intent might be to explore electronics-related products, but this is also not the final destination.
|
||
|
||
#### Step 4: Move cursor away from "Electronics"
|
||
- **Action**: I move my cursor away from the "Electronics" menu.
|
||
- **Page Changes**: The dropdown menu disappears.
|
||
- **Possible Purpose**: This reinforces the idea of exploring different categories before making a selection.
|
||
|
||
#### Step 5: Click on "Home" in the breadcrumb navigation
|
||
- **Action**: I click on the "Home" link in the breadcrumb navigation at the top left of the page.
|
||
- **Page Changes**: The page transitions to the homepage of the website.
|
||
- **Possible Purpose**: The intent is to return to the main page, possibly to start a new search or navigate more broadly.
|
||
|
||
#### Step 6: Click on "Furniture" in the breadcrumb navigation
|
||
- **Action**: I click on the "Furniture" link in the breadcrumb navigation.
|
||
- **Page Changes**: The page transitions to the "Furniture" category page.
|
||
- **Possible Purpose**: The intent is to narrow down the search to furniture-related products.
|
||
|
||
#### Step 7: Click on "Living Room Furniture"
|
||
- **Action**: I click on the "Living Room Furniture" link in the breadcrumb navigation.
|
||
- **Page Changes**: The page transitions to the "Living Room Furniture" category page, displaying various living room furniture items with options to sort and filter.
|
||
- **Possible Purpose**: The final intent is to browse and potentially purchase living room furniture items.
|
||
|
||
### Summary
|
||
In this video segment, I navigated from the "Health & Household" section through various menus ("Clothing, Shoes & Jewelry" and "Electronics") before ultimately deciding to go back to the homepage and then specifically to the "Living Room Furniture" section. Each step involved hovering over menus, moving the cursor, and clicking links to refine the search and reach the desired product category.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the "Living Room Furniture" category link.
|
||
- **Page Changes**: The webpage transitions to display a list of living room furniture items. The title "Living Room Furniture" is prominently shown, and various product listings with images, names, and prices are visible.
|
||
- **Possible Purpose**: The likely intent is to navigate to a specific section of the website dedicated to living room furniture to browse available products.
|
||
|
||
#### 2. **Action**: I scroll down the page.
|
||
- **Page Changes**: Additional living room furniture products become visible as I scroll. The initial set of products at the top of the page moves upward, and new products appear below them.
|
||
- **Possible Purpose**: The purpose of scrolling is to view more product options that are not initially visible on the screen.
|
||
|
||
#### 3. **Action**: I hover over a specific product image (a sectional sofa).
|
||
- **Page Changes**: There is no significant change to the webpage layout, but the cursor changes to indicate interaction readiness.
|
||
- **Possible Purpose**: Hovering over the product image is likely to get a closer look at the item or to prepare for further interaction, such as clicking to view details.
|
||
|
||
#### 4. **Action**: I click on the "Add to Cart" button for the selected sectional sofa.
|
||
- **Page Changes**: A confirmation message appears, indicating that the item has been successfully added to the cart. The button may change its text or appearance to reflect this action (e.g., "Added" or a similar indicator).
|
||
- **Possible Purpose**: The intent is to add the selected sectional sofa to the shopping cart for potential purchase.
|
||
|
||
#### 5. **Action**: I scroll back up to the top of the page.
|
||
- **Page Changes**: The view returns to the top section of the "Living Room Furniture" page, showing the category title and initial product listings.
|
||
- **Possible Purpose**: Scrolling back up could be to review other categories, access navigation menus, or prepare for another action such as searching for additional items.
|
||
|
||
### Summary
|
||
In this video segment, I navigated to the "Living Room Furniture" section, browsed through various product listings, selected a sectional sofa by adding it to the cart, and then scrolled back to the top of the page. Each action was performed with the apparent intent of exploring and selecting items for purchase within the specified category.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "menuitem_\ue622 Health & Household",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"\ue622 Health & Household\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Living Room Furniture",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Living Room Furniture\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/health-household.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/health-household.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Sort By",
|
||
"idx": 2,
|
||
"action_repr": "frame.selectOptionget_by_label(\"Sort By\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/home-kitchen/furniture/living-room-furniture.html"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue613 Set Descending Direction",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue613 Set Descending Direction\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/home-kitchen/furniture/living-room-furniture.html?product_list_order=price"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/home-kitchen/furniture/living-room-furniture.html?product_list_order=price"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |