295 lines
15 KiB
Plaintext
295 lines
15 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
How much did I spend on shopping at One Stop Market on November 2022? They gave me a 20% discount on the total amount for orders exceeding $200 in cash
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the homepage of "One Stop Market" visible. The page displays various categories like "Beauty & Personal Care," "Sports & Outdoors," and others. There is a navigation bar at the top with options such as "My Account," "My Wish List," and "Sign Out."
|
||
- **Page Changes:** No changes occur as this is the starting point.
|
||
- **Possible Purpose:** The initial state sets the context for the user's interaction with the website.
|
||
|
||
#### 2. **Hovering Over "My Account"**
|
||
- **Action:** I hover over the "My Account" link in the top navigation bar.
|
||
- **Page Changes:** A dropdown menu appears under "My Account," displaying options such as "My Orders," "My Downloadable Products," "My Address Book," "Account Information," "Stored Payment Methods," "My Product Reviews," and "Newsletter Subscriptions."
|
||
- **Possible Purpose:** Hovering over "My Account" is likely to explore the available account-related options or to navigate to a specific section within the account settings.
|
||
|
||
#### 3. **Clicking "My Account"**
|
||
- **Action:** I click on the "My Account" link in the top navigation bar.
|
||
- **Page Changes:** The page transitions to the "My Account" dashboard. This new page includes sections for "Account Information," "Contact Information," "Address Book," and "Default Billing Address." Additionally, there is a sidebar on the left with the same options that were in the dropdown menu.
|
||
- **Possible Purpose:** Clicking "My Account" is intended to access the main account dashboard, where the user can manage various aspects of their account.
|
||
|
||
#### 4. **Navigating to "My Orders"**
|
||
- **Action:** I click on the "My Orders" link in the sidebar menu on the left.
|
||
- **Page Changes:** The page updates to display the "My Orders" section. This page lists all previous orders with details such as "Order #," "Date," "Order Total," "Status," and "Action." Each order has options to "View Order" and "Reorder."
|
||
- **Possible Purpose:** Navigating to "My Orders" allows the user to review past purchases, check order statuses, or reorder items.
|
||
|
||
#### 5. **Scrolling Down the "My Orders" Page**
|
||
- **Action:** I scroll down the "My Orders" page.
|
||
- **Page Changes:** As I scroll, more orders become visible, showing additional entries in the order list.
|
||
- **Possible Purpose:** Scrolling down is likely to view more orders that are not initially visible on the page, providing a comprehensive overview of all past transactions.
|
||
|
||
### Summary
|
||
In this video segment, I start from the homepage of "One Stop Market," hover over and then click the "My Account" link to access the account dashboard. From there, I navigate to the "My Orders" section by clicking the corresponding link in the sidebar. Finally, I scroll down to view more of my past orders. Each action is performed to progressively access and review detailed account and order information.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a webpage titled "My Orders" displayed. The page lists various orders with details such as Order #, Date, Order Total, Status, and Action.
|
||
- **Page Changes:** No changes occur at this point; the page is static.
|
||
- **Possible Purpose:** The initial state sets the context for the subsequent actions, indicating that the user is reviewing their order history.
|
||
|
||
#### 2. **Highlighting an Order**
|
||
- **Action:** I highlight a specific order in the list (Order # 000000163) by hovering over or selecting it.
|
||
- **Page Changes:** The highlighted order becomes more prominent, possibly through a change in background color or border.
|
||
- **Possible Purpose:** This action suggests that I am focusing on this particular order, perhaps to view more details or perform an action related to it.
|
||
|
||
#### 3. **Inspecting the Order Element**
|
||
- **Action:** I right-click on the highlighted order and select an inspection tool (likely a browser's developer tools) to examine the HTML element.
|
||
- **Page Changes:** A small tooltip or inspection box appears, showing the HTML code for the selected order element.
|
||
- **Possible Purpose:** The inspection is likely to understand the structure of the webpage, possibly to locate specific attributes or classes for automation or debugging purposes.
|
||
|
||
#### 4. **Navigating to a Calculator Page**
|
||
- **Action:** The scene transitions to a new webpage featuring a calculator interface.
|
||
- **Page Changes:** The new page displays a calculator with input fields for expressions and results, along with buttons for operations like "Clear" and "=".
|
||
- **Possible Purpose:** This transition suggests a shift in focus, possibly to perform a calculation related to the order data previously inspected.
|
||
|
||
#### 5. **Inputting an Expression**
|
||
- **Action:** I enter a mathematical expression ("218.17*0.8") into the calculator's input field.
|
||
- **Page Changes:** The expression appears in the input field, and the result is displayed below after pressing the "=" button.
|
||
- **Possible Purpose:** The calculation might be related to determining a discounted price, subtotal, or another value derived from the order data.
|
||
|
||
#### 6. **Returning to the Orders Page**
|
||
- **Action:** The scene transitions back to the "My Orders" page.
|
||
- **Page Changes:** The orders list is visible again, with the same highlighted order as before.
|
||
- **Possible Purpose:** Returning to the orders page indicates that the calculation was relevant to the order data, and I may need to verify or use the result in the context of the orders.
|
||
|
||
#### 7. **Highlighting Another Order**
|
||
- **Action:** I highlight a different order in the list (Order # 000000174).
|
||
- **Page Changes:** The new order becomes highlighted, drawing attention to its details.
|
||
- **Possible Purpose:** This action suggests a continuation of reviewing or analyzing specific orders, possibly comparing them or performing similar actions as before.
|
||
|
||
#### 8. **Inspecting the New Order Element**
|
||
- **Action:** I right-click on the newly highlighted order and use the inspection tool to examine its HTML element.
|
||
- **Page Changes:** Another inspection box appears, displaying the HTML code for the selected order element.
|
||
- **Possible Purpose:** Similar to the previous inspection, this action aims to analyze the webpage structure, potentially for consistency or further automation tasks.
|
||
|
||
### Summary
|
||
In this video segment, I interact with a "My Orders" webpage by highlighting and inspecting specific orders, transitioning to a calculator page to perform a calculation, and then returning to the orders page to highlight and inspect another order. These actions suggest a process of analyzing order data, possibly for verification, comparison, or automation purposes. Each step is methodical, focusing on understanding and manipulating the webpage elements.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the text box where the expression is entered.
|
||
- **Page Changes**: The text box becomes active, and a cursor appears inside it, indicating that it is ready for input or editing.
|
||
- **Possible Purpose**: The likely intent is to either modify the existing expression or prepare to enter a new one.
|
||
|
||
2. **Action**: I type `+51.94` into the text box.
|
||
- **Page Changes**: The expression in the text box updates to `218.17*0.8+133.07+51.94`.
|
||
- **Possible Purpose**: The purpose is to add a new term to the existing mathematical expression to modify the calculation.
|
||
|
||
3. **Action**: I click the green button with the equals sign (`=`).
|
||
- **Page Changes**: The result displayed below the text box updates to `350.536`.
|
||
- **Possible Purpose**: The intent is to evaluate the new expression and display the updated result.
|
||
|
||
4. **Action**: I hover over the calculator interface, specifically near the text elements.
|
||
- **Page Changes**: A tooltip or inspection tool appears, highlighting the text "Calculator Enter the expression and get the results - Clear Result: 350.536".
|
||
- **Possible Purpose**: The action seems to be aimed at inspecting or verifying the current state of the calculator, possibly for debugging or documentation purposes.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, without any additional context or assumptions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_My Account",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Account\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_My Orders",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/account/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Scratchpad",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_label(\"Scratchpad\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "heading_History",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"heading\", name=\"History\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Page 2",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Page 2\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Calculator",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Calculator\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Enter expression\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_=",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"=\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_8",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Enter expression\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/calculator.html"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |