262 lines
14 KiB
Plaintext
262 lines
14 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
How much refund I should expect from my order canlled in 2022, including shipping fee
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "My Account" link located at the top right corner of the webpage.
|
||
- **Page Changes:** The page transitions to the "My Account" section, displaying account-related options such as "My Orders," "Address Book," and "Account Information."
|
||
- **Possible Purpose:** The likely intent is to access personal account details and manage account settings or view order history.
|
||
|
||
#### 2. **Action:** I hover over the "Sports & Outdoors" menu item in the main navigation bar.
|
||
- **Page Changes:** A dropdown menu appears under "Sports & Outdoors," revealing subcategories like "Exercise & Fitness," "Fan Shop," and "Sports."
|
||
- **Possible Purpose:** The intention is to explore products within the "Sports & Outdoors" category, possibly to find specific items related to exercise, sports equipment, or fitness.
|
||
|
||
#### 3. **Action:** I click on the "My Orders" link in the left sidebar under the "My Account" section.
|
||
- **Page Changes:** The page navigates to the "My Orders" section, listing all previous orders with details such as Order #, Date, Order Total, Status, and Action.
|
||
- **Possible Purpose:** The goal is to review past orders, check their status, or possibly reorder items.
|
||
|
||
#### 4. **Action:** I scroll down slightly on the "My Orders" page.
|
||
- **Page Changes:** Additional orders become visible, extending the list of previous purchases.
|
||
- **Possible Purpose:** The purpose is to view more orders that were initially out of sight, ensuring a comprehensive review of the order history.
|
||
|
||
#### 5. **Action:** I click on the "View Order" link for a specific order (Order #000000170).
|
||
- **Page Changes:** The page transitions to a detailed view of the selected order, showing items purchased, quantities, prices, and shipping information.
|
||
- **Possible Purpose:** The intent is to inspect the details of this particular order, perhaps to verify the items ordered, check the total cost, or review shipping details.
|
||
|
||
### Summary
|
||
In this video segment, I navigate through the "My Account" section, explore the "Sports & Outdoors" category, and focus on reviewing my order history. Each action is deliberate, aimed at accessing specific account information and order details. The sequence reflects a user's typical behavior when managing their account and verifying past transactions on an e-commerce website.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the "View Order" link next to the order with Order # 000000184.
|
||
- **Page Changes:** The page transitions from the "My Orders" list to a detailed order view for Order # 000000184. This new page displays comprehensive information about the items ordered, their prices, quantities, and the total cost of the order. It also shows the shipping address and other order details.
|
||
- **Possible Purpose:** The likely intent is to review the specifics of this particular order, such as the items purchased, their costs, and delivery information.
|
||
|
||
#### Step 2:
|
||
- **Action:** I scroll down the detailed order view page.
|
||
- **Page Changes:** As I scroll, more details about the order become visible, including the "Items Ordered" section which lists each product with its name, SKU, price, quantity, and subtotal. Further scrolling reveals the "Order Information" section, which includes the shipping address and method.
|
||
- **Possible Purpose:** The purpose of scrolling is to gain a complete overview of the order, ensuring all relevant details are reviewed, such as the exact items ordered, their specifications, and the delivery location.
|
||
|
||
#### Step 3:
|
||
- **Action:** I hover over the "Grand Total" value at the bottom of the order summary.
|
||
- **Page Changes:** Hovering over the "Grand Total" highlights it, possibly indicating an interactive element or providing additional information (though no further pop-up or detail is shown in this segment).
|
||
- **Possible Purpose:** The intent might be to confirm the final amount paid for the order or to check if there is any additional information related to the total cost, such as a breakdown or explanation of charges.
|
||
|
||
### Summary of Observed Actions:
|
||
1. I clicked on the "View Order" link for Order # 000000184, transitioning to a detailed order view page.
|
||
2. I scrolled down the page to review the "Items Ordered" and "Order Information" sections.
|
||
3. I hovered over the "Grand Total" to possibly gather more information about the final cost of the order.
|
||
|
||
Each action is performed with the clear objective of thoroughly reviewing the details of a specific order, ensuring accuracy and completeness in understanding the transaction.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the "View Order" link for Order #000000172.
|
||
- **Page Changes:** A tooltip appears, displaying the text "get by role('cell', {name:'Complete'}).with{$(3)}".
|
||
- **Possible Purpose:** The likely intent is to inspect or verify details related to the order status of Order #000000172, which is marked as "Canceled". The appearance of the tooltip suggests an interaction with a debugging or inspection tool to examine the specific cell containing the order status.
|
||
|
||
#### Step 2:
|
||
- **Action:** I move the cursor away from the "View Order" link for Order #000000172.
|
||
- **Page Changes:** The tooltip disappears.
|
||
- **Possible Purpose:** The action of moving the cursor away indicates a completion of the inspection or a decision to not proceed further with this specific order at the moment.
|
||
|
||
#### Step 3:
|
||
- **Action:** I scroll down the page to view more orders.
|
||
- **Page Changes:** Additional orders (from Order #000000182 to Order #000000169) become visible.
|
||
- **Possible Purpose:** The intent is to review more orders, possibly looking for specific information or patterns across different orders.
|
||
|
||
#### Step 4:
|
||
- **Action:** I click on the "View Order" link for Order #000000159.
|
||
- **Page Changes:** A tooltip appears, displaying the text "get by role('cell', {name:'Complete'}).with{$(3)}".
|
||
- **Possible Purpose:** Similar to the previous inspection, the goal is to examine the order status of Order #000000159, which is also marked as "Complete". The tooltip again suggests the use of a debugging tool to focus on the order status cell.
|
||
|
||
#### Step 5:
|
||
- **Action:** I move the cursor away from the "View Order" link for Order #000000159.
|
||
- **Page Changes:** The tooltip disappears.
|
||
- **Possible Purpose:** This action signifies the end of the inspection for this particular order, allowing me to either review the information mentally or prepare for the next action.
|
||
|
||
#### Step 6:
|
||
- **Action:** I scroll further down the page.
|
||
- **Page Changes:** More orders become visible, specifically Orders #000000167 to #000000160.
|
||
- **Possible Purpose:** Continuing the review of additional orders, potentially searching for specific statuses, dates, or totals.
|
||
|
||
#### Step 7:
|
||
- **Action:** I click on the "View Order" link for Order #000000167.
|
||
- **Page Changes:** A tooltip appears with the text "get by role('cell', {name:'Complete'}).with{$(3)}".
|
||
- **Possible Purpose:** The intent is to inspect the order status of Order #000000167, which is marked as "Complete". The consistent use of the tooltip suggests a systematic approach to verifying order statuses.
|
||
|
||
#### Step 8:
|
||
- **Action:** I move the cursor away from the "View Order" link for Order #000000167.
|
||
- **Page Changes:** The tooltip disappears.
|
||
- **Possible Purpose:** Concluding the inspection of this order, preparing to either take further actions or continue reviewing other orders.
|
||
|
||
### Summary
|
||
In this video segment, I systematically inspected the "View Order" links for specific orders (Orders #000000172, #000000159, and #000000167) using a debugging tool that displayed tooltips. The primary focus appeared to be on verifying the order statuses, particularly those marked as "Complete" or "Canceled". Between inspections, I scrolled through the list to review additional orders, indicating a thorough examination of the order history.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I hover over the "My Orders" link in the left sidebar menu.
|
||
- **Page Changes**: There are no immediate changes to the webpage as a result of hovering.
|
||
- **Possible Purpose**: The likely intent is to prepare for selecting the "My Orders" link, ensuring it is the correct option before clicking.
|
||
|
||
2. **Action**: I click on the "My Orders" link in the left sidebar menu.
|
||
- **Page Changes**: The page remains on the "My Orders" section, confirming that this section is already active and no navigation is needed.
|
||
- **Possible Purpose**: The action confirms the current section and ensures that the "My Orders" page is displayed.
|
||
|
||
3. **Action**: I move the cursor to the "View Order" link associated with Order #000000181.
|
||
- **Page Changes**: There are no changes to the webpage as a result of this action.
|
||
- **Possible Purpose**: The intent is likely to inspect the details of Order #000000181 by viewing its specific order page.
|
||
|
||
4. **Action**: I click on the "View Order" link for Order #000000181.
|
||
- **Page Changes**: The webpage transitions to display the detailed information for Order #000000181, showing specifics such as items ordered, prices, and shipping details.
|
||
- **Possible Purpose**: The purpose is to review the complete details of Order #000000181, which may include verifying the order status, itemized costs, or shipment tracking.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions and resulting page changes within this specific part of the video.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_My Account",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Account\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_My Orders",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/customer/account/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue608 Page Next",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue608 Page Next\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=2"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000177 10/18/22 $2,126.32 Canceled View Order Reorder",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000177 10/18/22 $2,126.32 Canceled View Order Reorder\").get_by_role(\"cell\").nth(4)",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=3"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |