203 lines
9.5 KiB
Plaintext
203 lines
9.5 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Update order #307 with the DHL tracking number 24353446464
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State: Dashboard Page**
|
||
- **Action:** The video starts with me on the Dashboard page of a Magento admin panel.
|
||
- **Page Changes:** The dashboard displays various metrics such as Lifetime Sales, Average Order, and Last Orders.
|
||
- **Possible Purpose:** The initial state sets the context, indicating that I am about to perform actions related to order management or analysis.
|
||
|
||
#### 2. **Navigating to Orders Page**
|
||
- **Action:** I click on the "Sales" menu item in the left sidebar, then select "Orders" from the dropdown.
|
||
- **Page Changes:** The page transitions to the Orders grid, listing all orders with columns for ID, Purchase Point, Purchase Date, Bill-to Name, Ship-to Name, Grand Total (Base), Grand Total (Purchased), Status, Action, Allocated sources, and Braintree Transaction Source.
|
||
- **Possible Purpose:** My intention is to access detailed order information, possibly to review, manage, or analyze specific orders.
|
||
|
||
#### 3. **Searching for Specific Orders**
|
||
- **Action:** I click on the search bar at the top of the Orders page and type "sarah miller."
|
||
- **Page Changes:** The Orders grid updates to show only the orders associated with "sarah miller," filtering out other entries.
|
||
- **Possible Purpose:** I am specifically looking for orders placed by a customer named Sarah Miller, likely to review or manage these particular orders.
|
||
|
||
#### 4. **Selecting an Order**
|
||
- **Action:** I click on one of the orders listed under "sarah miller."
|
||
- **Page Changes:** The page navigates to the detailed view of the selected order, displaying comprehensive information including Order & Account Information, Address Information, and Account Information.
|
||
- **Possible Purpose:** I intend to examine the details of this specific order, which may involve verifying order status, customer details, shipping information, or financial transactions.
|
||
|
||
#### 5. **Reviewing Order Details**
|
||
- **Action:** I scroll through the detailed order view, observing sections like Order View, Information, Credit Memos, Invoices, Shipments, Comments History, and Address Information.
|
||
- **Page Changes:** The content within each section becomes visible as I scroll, providing a thorough overview of the order's lifecycle and associated data.
|
||
- **Possible Purpose:** My goal is to gather comprehensive information about the order, which could be for purposes such as troubleshooting, customer service, or record-keeping.
|
||
|
||
### Summary
|
||
In this video segment, I navigate from the Dashboard to the Orders page, search for orders by a specific customer ("sarah miller"), select one of these orders, and review its detailed information. Each step is methodical, suggesting a focused intent to access and analyze specific order data.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
In this video segment, the following actions are observed:
|
||
|
||
1. **Action**: I click on the "Order Status" dropdown menu.
|
||
- **Page Changes**: The dropdown menu expands, revealing a list of order status options.
|
||
- **Possible Purpose**: The likely intent is to change the current order status from "Canceled" to another available status.
|
||
|
||
2. **Action**: I select a new order status from the dropdown menu.
|
||
- **Page Changes**: The selected status appears in the "Order Status" field, replacing the previous "Canceled" status.
|
||
- **Possible Purpose**: The purpose is to update the order's status to reflect its current or intended processing stage.
|
||
|
||
3. **Action**: I click on the "Submit Comment" button.
|
||
- **Page Changes**: A confirmation message appears, indicating that the comment has been submitted successfully. The page may also refresh to display the updated order status and any associated comments.
|
||
- **Possible Purpose**: The intent is to save the changes made to the order status and potentially add a comment related to the status update for record-keeping or communication purposes.
|
||
|
||
4. **Action**: I hover over the "Edit" link next to the "Billing Address."
|
||
- **Page Changes**: No immediate changes occur on the page; however, the cursor changes to indicate interactivity with the "Edit" link.
|
||
- **Possible Purpose**: The likely intent is to prepare for editing the billing address information, though no actual edit is performed in this segment.
|
||
|
||
These actions are strictly based on the observable content of the provided video segment, without incorporating any additional context or assumptions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_\ue608 Catalog",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue608 Catalog\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue60b Sales",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue60b Sales\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Orders",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_\ue620",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"\ue620\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"tr:nth-child(2) > td:nth-child(3) > .data-grid-cell-content\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/sales/order/view/order_id/307/"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |