195 lines
9.0 KiB
Plaintext
195 lines
9.0 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
I have a lot of Nintendo Switch game cards now, help me find the best storage option to fit all 40 cards
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial Page Load and Search Initiation
|
||
|
||
#### Action:
|
||
- **I click** on the search bar located at the top right corner of the webpage.
|
||
- **I type** "nintendo switch card storage" into the search bar.
|
||
|
||
#### Page Changes:
|
||
- The webpage remains static until text is entered into the search bar.
|
||
- After typing, a dropdown suggestion list appears below the search bar, offering autocomplete options related to the typed query.
|
||
|
||
#### Possible Purpose:
|
||
- The likely intent of clicking the search bar is to activate it for input.
|
||
- Typing "nintendo switch card storage" aims to find products related to storage solutions for Nintendo Switch game cards.
|
||
|
||
---
|
||
|
||
#### Action:
|
||
- **I press** the Enter key after typing the query.
|
||
|
||
#### Page Changes:
|
||
- The webpage transitions from the homepage to a new page displaying search results for "nintendo switch card storage."
|
||
- The new page shows a list of products with images, names, prices, and an "Add to Cart" button for each item.
|
||
- The left sidebar updates to show filtering options based on the search query.
|
||
|
||
#### Possible Purpose:
|
||
- Pressing Enter submits the search query to the website.
|
||
- The transition to the search results page is intended to display relevant products that match the search term, allowing for selection and potential purchase.
|
||
|
||
---
|
||
|
||
### Summary of Actions in Part 1:
|
||
1. **Click** on the search bar to activate it.
|
||
2. **Type** "nintendo switch card storage" into the search bar.
|
||
3. **Press** Enter to submit the search query.
|
||
|
||
### Observed Page Changes:
|
||
- Activation of the search bar.
|
||
- Appearance of autocomplete suggestions.
|
||
- Transition to a search results page with relevant product listings and filtering options.
|
||
|
||
### Likely Purpose:
|
||
- To locate and view products specifically designed for storing Nintendo Switch game cards.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I hover over the product titled "Game Cartridge Holder Storage Case for 160 Nintendo 3DS 3DSXL 2DS 2DSXL DS DSL Storage Organizer Compatible with Nintendo Switch Game, Sony PS Vita Games, Playstation Vita Memory Card and SD Cards (Box Only)."
|
||
- **Page Changes**: A tooltip appears, displaying the full title of the product, which is truncated in the main view.
|
||
- **Possible Purpose**: The likely intent is to read the complete product title to ensure it matches my search criteria or interests.
|
||
|
||
#### 2. **Action**: I click on the product titled "Game Cartridge Holder Storage Case for 160 Nintendo 3DS 3DSXL 2DS 2DSXL DS DSL Storage Organizer Compatible with Nintendo Switch Game, Sony PS Vita Games, Playstation Vita Memory Card and SD Cards (Box Only)."
|
||
- **Page Changes**: The webpage transitions to the product detail page for the selected item. This new page provides a detailed description, images, price, and options to add the item to the cart.
|
||
- **Possible Purpose**: The intent is to view more detailed information about the product, such as specifications, customer reviews, and pricing, to make an informed purchasing decision.
|
||
|
||
#### 3. **Action**: I scroll down slightly on the product detail page.
|
||
- **Page Changes**: Additional sections of the product detail page become visible, including more images, a detailed description, and customer reviews.
|
||
- **Possible Purpose**: The intent is to gather more information about the product, such as its features, compatibility, and user feedback, to further evaluate its suitability.
|
||
|
||
#### 4. **Action**: I click on the "Add to Cart" button located below the product image and price.
|
||
- **Page Changes**: A confirmation message appears, indicating that the product has been successfully added to the cart. The cart icon at the top right corner updates to reflect the new item.
|
||
- **Possible Purpose**: The intent is to add the selected product to the shopping cart, moving forward with the purchase process.
|
||
|
||
### Summary
|
||
In this video segment, I interact with a product listing by first hovering over it to view the full title, then clicking to access the product detail page. After reviewing the details, I add the product to the cart. Each action is aimed at gathering necessary information and progressing towards making a purchase.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search entire store here...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.pressget_by_placeholder(\"Search entire store here...\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Game Cartridge Holder Case for 160 Nintendo 3DS 3DSXL 2DS 2DSXL DS DSi, Storage Organizer Compatible with Nintendo Switch Game, Sony PS Vita Games, Playstation Vita Memory Card and SD Cards (Box Only)",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Game Cartridge Holder Case for 160 Nintendo 3DS 3DSXL 2DS 2DSXL DS DSi, Storage Organizer Compatible with Nintendo Switch Game, Sony PS Vita Games, Playstation Vita Memory Card and SD Cards (Box Only)\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/catalogsearch/result/?q=nintendo+switch+card+storage"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/catalogsearch/result/?q=nintendo+switch+card+storage"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |