149 lines
6.6 KiB
Plaintext
149 lines
6.6 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Tell me the total cost of my latest complete order?
|
||
|
||
# Annotation description
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on "My Account" in the top-right corner of the webpage.
|
||
- **Page Changes:** The page transitions to a new section titled "My Account," displaying a sidebar menu with options like "My Orders," "My Wish List," "Account Information," etc. The main content area shows a list of past orders with details such as Order #, Date, Order Total, Status, and Action.
|
||
- **Possible Purpose:** The likely intent is to access personal account information and manage past orders or account settings.
|
||
|
||
#### 2. **Action:** I scroll down slightly on the "My Orders" page.
|
||
- **Page Changes:** The visible portion of the page shifts downward, revealing more of the order history list. The top part of the page, including the header and some initial orders, moves out of view.
|
||
- **Possible Purpose:** The purpose is to review additional orders that are not initially visible on the screen.
|
||
|
||
#### 3. **Action:** I hover over an order entry in the list (Order #000000180).
|
||
- **Page Changes:** There is no significant change in the page layout or content; however, the order entry might be highlighted or show a tooltip if such functionality is present.
|
||
- **Possible Purpose:** The intent is to focus on or inspect the details of this specific order, possibly to view more information or take further action related to it.
|
||
|
||
#### 4. **Action:** I click on the "View Order" link next to Order #000000180.
|
||
- **Page Changes:** The page transitions to a detailed order view for Order #000000180. This new page likely contains comprehensive information about the order, such as items ordered, quantities, prices, shipping details, and payment information.
|
||
- **Possible Purpose:** The goal is to access detailed information about this specific order, which may include verifying order contents, tracking status, or obtaining a receipt.
|
||
|
||
---
|
||
|
||
This description strictly adheres to the observable actions within the provided video segment, focusing solely on the interactions and resulting page changes without incorporating external context or assumptions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_My Account",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Account\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_My Orders",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000180 3/11/23 $65.32 Complete View Order Reorder",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000180 3/11/23 $65.32 Complete View Order Reorder\").get_by_role(\"link\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/180/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/180/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |