163 lines
7.2 KiB
Plaintext
163 lines
7.2 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
What do customers say about brush from sephora
|
||
|
||
# Annotation description
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the search bar located at the top center of the webpage.
|
||
- **Page Changes:** The search bar becomes active, and a blinking cursor appears inside it, indicating that it is ready for text input.
|
||
- **Possible Purpose:** The likely intent is to initiate a product search by entering specific keywords.
|
||
|
||
2. **Action:** I type "sephora brush" into the active search bar.
|
||
- **Page Changes:** As I type, the text "sephora brush" appears in the search bar. No immediate page changes occur until the search is executed.
|
||
- **Possible Purpose:** The purpose is to find products related to "sephora brush" within the website's inventory.
|
||
|
||
3. **Action:** I press the Enter key or click the search icon (magnifying glass) next to the search bar.
|
||
- **Page Changes:** The webpage transitions to a new page displaying search results for "sephora brush." The header now reads "Search results for: 'sephora brush'," and a list of relevant products is shown.
|
||
- **Possible Purpose:** The intent is to view and browse through the available products that match the search query.
|
||
|
||
4. **Action:** I scroll down the search results page.
|
||
- **Page Changes:** The visible portion of the webpage shifts downward, revealing more products related to "sephora brush."
|
||
- **Possible Purpose:** The likely intent is to explore additional products beyond those initially visible at the top of the search results.
|
||
|
||
5. **Action:** I click on the "Add to Cart" button for the "SEPHORA COLLECTION PRO Featherweight Powder Brush #91."
|
||
- **Page Changes:** A confirmation message may appear, indicating that the item has been added to the cart. The button might change its appearance or text to reflect the action (e.g., "Added" or "View Cart").
|
||
- **Possible Purpose:** The intent is to add the selected product to the shopping cart for potential purchase.
|
||
|
||
### Summary:
|
||
In this video segment, I perform a series of actions aimed at searching for and adding a specific product to the shopping cart. I start by activating and using the search bar to find "sephora brush" products. After viewing the search results, I scroll to find a suitable item and then add the "SEPHORA COLLECTION PRO Featherweight Powder Brush #91" to the cart. Each step is methodical, focusing on navigating the website to achieve the goal of selecting and preparing a product for purchase.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search entire store here...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.pressget_by_placeholder(\"Search entire store here...\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Image",
|
||
"idx": 2,
|
||
"action_repr": "frame.clicklocator(\"#product-item-info_58094\").get_by_role(\"link\", name=\"Image\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sephora-collection-skin-perfecting-brush-set.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/catalogsearch/result/?q=sephora+brush"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_SEPHORA COLLECTION PRO Featherweight Powder Brush #91",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"SEPHORA COLLECTION PRO Featherweight Powder Brush #91\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/catalogsearch/result/?q=sephora+brush"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/catalogsearch/result/?q=sephora+brush"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |