196 lines
10 KiB
Plaintext
196 lines
10 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Tell me the status of my latest order and when will it arrive
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on "My Orders" in the left-side menu.
|
||
- **Page Changes:** The page transitions to display a list of orders under the "My Orders" section. Each order is listed with details such as Order #, Date, Order Total, Status, and Action options like "View Order" and "Reorder."
|
||
- **Possible Purpose:** The likely intent is to review past orders, possibly to check order status, details, or to reorder items.
|
||
|
||
#### 2. **Action:** I click on the "View Order" link for Order #000000189.
|
||
- **Page Changes:** The page navigates to a detailed view of Order #000000189. This page includes sections for Order Information, Items Ordered, Shipping Address, Billing Address, Payment Method, and a summary of the order total.
|
||
- **Possible Purpose:** The purpose is to inspect the specifics of this particular order, such as the items purchased, shipping details, and payment information.
|
||
|
||
#### 3. **Action:** I scroll down to the "Payment Method" section.
|
||
- **Page Changes:** The view shifts downward, bringing the "Payment Method" section into focus. This section confirms the payment method used for the order (e.g., Check / Money order).
|
||
- **Possible Purpose:** The intent might be to verify the payment method used for this order or to gather information for record-keeping.
|
||
|
||
#### 4. **Action:** I hover over the "Items Ordered" section.
|
||
- **Page Changes:** No significant change occurs other than the cursor movement highlighting the "Items Ordered" section, which lists the product name, SKU, price, quantity, and subtotal.
|
||
- **Possible Purpose:** The likely reason is to review the specific items included in the order, their prices, and quantities.
|
||
|
||
### Summary
|
||
In this video segment, I navigate through the "My Orders" section to view the details of a specific order (Order #000000189). My actions include clicking to access the order details, scrolling to the payment method section, and hovering over the items ordered. These steps suggest a focused effort to review and confirm the specifics of the selected order, such as the items, addresses, and payment method.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "My Orders" link in the left-side menu.
|
||
- **Page Changes:** The page transitions to display a list of orders with details such as Order #, Date, Order Total, Status, and Action options.
|
||
- **Possible Purpose:** The likely intent is to access a summary of past orders for review or further action.
|
||
|
||
#### 2. **Action:** I hover over an order entry (Order # 000000189) in the list.
|
||
- **Page Changes:** No significant change occurs; the order entry remains highlighted as I hover.
|
||
- **Possible Purpose:** This action might be to prepare for selecting this specific order for detailed viewing or to ensure it is the correct one before proceeding.
|
||
|
||
#### 3. **Action:** I click on the order entry (Order # 000000189).
|
||
- **Page Changes:** The page navigates to a detailed view of Order # 000000189, showing specifics like items ordered, order information, shipping address, billing address, shipping method, and payment method.
|
||
- **Possible Purpose:** The intent is to examine the details of this particular order, possibly to verify information, track its status, or manage it further.
|
||
|
||
#### 4. **Action:** I scroll down slightly on the detailed order page.
|
||
- **Page Changes:** The visible section of the page shifts downward, revealing more details such as the "Grand Total" and additional order information.
|
||
- **Possible Purpose:** This action is likely to get a full view of the order details, ensuring all relevant information is reviewed.
|
||
|
||
#### 5. **Action:** I highlight the text "Grand Total" on the page.
|
||
- **Page Changes:** The text "Grand Total" becomes visually emphasized, indicating focus on this specific piece of information.
|
||
- **Possible Purpose:** The purpose could be to draw attention to the total cost of the order, perhaps for verification or documentation purposes.
|
||
|
||
#### 6. **Action:** I move the cursor to the "Order Date" section and highlight the date "May 2, 2023."
|
||
- **Page Changes:** The date "May 2, 2023" is now visually highlighted.
|
||
- **Possible Purpose:** This action suggests an intent to focus on the order date, possibly to confirm when the order was placed or for record-keeping.
|
||
|
||
#### 7. **Action:** I scroll back up to the top of the detailed order page.
|
||
- **Page Changes:** The view returns to the top section of the order details, including the order number and status.
|
||
- **Possible Purpose:** This might be to review other top-level details of the order or to navigate back to a previous section for additional context.
|
||
|
||
### Summary
|
||
In this video segment, my actions are focused on accessing and reviewing the details of a specific order (Order # 000000189). I navigate through the order list, select the desired order, and meticulously examine various details such as the grand total and order date. These steps suggest a thorough review process, possibly for verification, tracking, or documentation purposes. Each action is deliberate, aimed at ensuring all relevant information is accurately observed and understood.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_My Account",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Account\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_My Orders",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/customer/account/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000189 5/2/23 $754.99 Pending View Order Reorder",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000189 5/2/23 $754.99 Pending View Order Reorder\").get_by_role(\"link\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/189/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000180 3/11/23 $65.32 Complete View Order Reorder",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000180 3/11/23 $65.32 Complete View Order Reorder\").get_by_role(\"link\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/180/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |