286 lines
12 KiB
Plaintext
286 lines
12 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Cancel order 307
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Provided Video Segment
|
||
|
||
#### 1. **Initial State: Dashboard Page**
|
||
- **Action:** I am on the Dashboard page of the Magento admin panel.
|
||
- **Page Changes:** The dashboard displays various metrics such as Lifetime Sales, Average Order, and Last Orders.
|
||
- **Possible Purpose:** The purpose is to provide an overview of the store's performance and recent activities.
|
||
|
||
#### 2. **Navigating to Orders Page**
|
||
- **Action:** I click on the "Sales" menu item in the left sidebar.
|
||
- **Page Changes:** A dropdown menu appears under "Sales."
|
||
- **Possible Purpose:** To access different sales-related sections within the admin panel.
|
||
|
||
- **Action:** I select the "Orders" option from the dropdown menu.
|
||
- **Page Changes:** The page transitions to the Orders page, displaying a list of orders with columns for ID, Purchase Point, Purchase Date, Bill-to Name, Ship-to Name, Grand Total (Base), Grand Total (Purchased), Status, Action, Allocated sources, and Braintree Transaction Source.
|
||
- **Possible Purpose:** To view and manage customer orders.
|
||
|
||
#### 3. **Searching for Specific Orders**
|
||
- **Action:** I click on the search bar at the top of the Orders page.
|
||
- **Page Changes:** The search bar becomes active, allowing text input.
|
||
- **Possible Purpose:** To find specific orders quickly by entering relevant keywords.
|
||
|
||
- **Action:** I type "Amanda Kim" into the search bar.
|
||
- **Page Changes:** The page updates to show 7 records found that match the search query "Amanda Kim."
|
||
- **Possible Purpose:** To locate orders associated with the customer "Amanda Kim."
|
||
|
||
#### 4. **Reviewing Search Results**
|
||
- **Action:** I observe the search results.
|
||
- **Page Changes:** The table now lists orders related to "Amanda Kim," including details like order IDs, purchase dates, and grand totals.
|
||
- **Possible Purpose:** To review the specific orders linked to "Amanda Kim" for further actions such as order management or analysis.
|
||
|
||
### Summary
|
||
- **Initial State:** Started on the Dashboard page.
|
||
- **Navigated to Orders Page:** Accessed via the "Sales" menu to manage orders.
|
||
- **Searched for Specific Orders:** Used the search bar to find orders related to "Amanda Kim."
|
||
- **Reviewed Search Results:** Examined the list of orders matching the search criteria.
|
||
|
||
Each step was performed methodically to navigate through the admin panel, search for specific data, and review the results, all while staying focused on the observable actions and their direct outcomes.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video starts with a view of an order management page for order #000000307.
|
||
- **Page Changes:** The page displays detailed information about the order, including order status (Pending), customer details, address information, and order totals.
|
||
- **Possible Purpose:** The initial state sets the context for managing or editing the order.
|
||
|
||
#### 2. **Click on "Edit" Button**
|
||
- **Action:** I click on the "Edit" button located at the top-right corner of the page.
|
||
- **Page Changes:** A confirmation dialog box appears with the message: "Are you sure? This order will be canceled and a new one will be created instead."
|
||
- **Possible Purpose:** The intent is to modify the order, which requires canceling the current order and creating a new one due to system constraints.
|
||
|
||
#### 3. **Confirm Edit Action**
|
||
- **Action:** I click on the "OK" button within the confirmation dialog box.
|
||
- **Page Changes:** The order status updates to "Canceled," and a notification at the top of the page confirms, "You have canceled the order."
|
||
- **Possible Purpose:** Confirming the edit action allows for the creation of a new order with potential modifications.
|
||
|
||
#### 4. **Order Status Update**
|
||
- **Action:** After confirming the edit, the order status changes from "Pending" to "Canceled."
|
||
- **Page Changes:** The order details reflect the cancellation, and the option to create a new order becomes relevant.
|
||
- **Possible Purpose:** The update ensures that the original order is properly canceled before proceeding with any new order creation.
|
||
|
||
### Summary
|
||
- **Actions Taken:** Clicked "Edit," confirmed with "OK."
|
||
- **Page Changes Observed:** Confirmation dialog appeared, order status changed to "Canceled," notification displayed.
|
||
- **Purpose of Actions:** To modify the existing order by canceling it and preparing to create a new one.
|
||
|
||
This description strictly adheres to the observable actions and their immediate consequences within the provided video segment.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_\ue60b Sales",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue60b Sales\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Orders",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "cell_\u2191 ID",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"cell\", name=\"\u2191 ID\").locator(\"span\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "cell_\u2193 ID",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"cell\", name=\"\u2193 ID\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_Search by keyword",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search by keyword\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_\ue620",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"\ue620\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000307 Main Website Main Website Store Default Store View Apr 19, 2023 7:42:25 PM Grace Nguyen Grace Nguyen $101.20 $101.20 Pending View",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000307 Main Website Main Website Store Default Store View Apr 19, 2023 7:42:25 PM Grace Nguyen Grace Nguyen $101.20 $101.20 Pending View\").get_by_role(\"link\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Edit",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Edit\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_\ue62f",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"\ue62f\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Cancel",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Cancel\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_OK",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"OK\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |