288 lines
14 KiB
Plaintext
288 lines
14 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
I recently moved, my address is 111 Magnolia Path, Atlanta, GA, 30303, update my information on OneStopShopping accordingly
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on "Address Book" in the left-side menu under "My Account."
|
||
- **Page Changes:** The page transitions to the "Address Book" section within "My Account." The main content area now displays options related to address management, including "Default Addresses," "Additional Address Entries," and buttons like "Add New Address."
|
||
- **Possible Purpose:** The likely intent is to manage or add new shipping or billing addresses. This action suggests a need to update or review existing address information.
|
||
|
||
#### 2. **Action:** I hover over the "Add New Address" button.
|
||
- **Page Changes:** There are no immediate changes to the page content, but the button becomes highlighted, indicating it is interactive and ready to be clicked.
|
||
- **Possible Purpose:** Hovering over the button is typically done to confirm its functionality or to prepare for clicking it. This could indicate an intention to add a new address entry.
|
||
|
||
#### 3. **Action:** I click on the "Add New Address" button.
|
||
- **Page Changes:** The page transitions to a new form where details for a new address can be entered. This form includes fields for entering the name, address, city, state, zip code, country, and other relevant information.
|
||
- **Possible Purpose:** The action of clicking the "Add New Address" button indicates the intent to input and save a new address, possibly for future orders or shipments.
|
||
|
||
### Summary of Observed Actions:
|
||
- **Initial State:** The video starts with the "My Account" page open, specifically highlighting the "Address Book" option in the sidebar menu.
|
||
- **Sequence of Actions:**
|
||
1. I navigate to the "Address Book" section by clicking on it.
|
||
2. I interact with the "Add New Address" button by first hovering over it and then clicking it.
|
||
3. As a result, the page changes to display a form for adding a new address.
|
||
- **Final State:** The video ends with the new address form displayed, ready for input.
|
||
|
||
This sequence strictly follows the observable actions without incorporating any assumptions beyond what is directly seen in the video segment.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
#### 1. **Action:** I click on the "Street Address" text box under the "Address" section.
|
||
- **Page Changes:** The text box is highlighted, indicating it is active and ready for input or editing.
|
||
- **Possible Purpose:** The likely intent is to either enter a new street address or modify the existing one. This action suggests a need to update or verify the address information.
|
||
|
||
#### 2. **Action:** I type "101 S San Mateo Dr" into the "Street Address" text box.
|
||
- **Page Changes:** The text "101 S San Mateo Dr" replaces any previous text in the "Street Address" field.
|
||
- **Possible Purpose:** The purpose is to input a specific street address, possibly to update the address to a new location or correct an existing entry.
|
||
|
||
#### 3. **Action:** I click on the "Street Address Line 2" text box under the "Address" section.
|
||
- **Page Changes:** The "Street Address Line 2" text box is highlighted, indicating it is now active.
|
||
- **Possible Purpose:** The intent might be to enter additional address information, such as a suite number, apartment number, or other relevant details that do not fit in the primary street address field.
|
||
|
||
#### 4. **Action:** I type "111 Magnolia Path" into the "Street Address Line 2" text box.
|
||
- **Page Changes:** The text "111 Magnolia Path" appears in the "Street Address Line 2" field.
|
||
- **Possible Purpose:** This action aims to provide supplementary address information, which could be necessary for more precise location identification or delivery instructions.
|
||
|
||
#### 5. **Action:** I click on the "State/Province" dropdown menu.
|
||
- **Page Changes:** The dropdown menu expands, displaying a list of states or provinces.
|
||
- **Possible Purpose:** The intent is to select the appropriate state or province associated with the entered address.
|
||
|
||
#### 6. **Action:** I scroll through the dropdown menu and select "California."
|
||
- **Page Changes:** The "State/Province" field updates to display "California."
|
||
- **Possible Purpose:** This ensures the address is correctly associated with the right state, which is essential for accurate geographical and logistical purposes, such as shipping or billing.
|
||
|
||
### Summary:
|
||
In this video segment, I am actively engaged in editing an address within the "Edit Address" section of the website. My actions include clicking on and typing into the "Street Address" and "Street Address Line 2" fields, as well as selecting "California" from the "State/Province" dropdown menu. These steps are performed to update or verify the address details accurately. Each action is deliberate, focusing on ensuring the address information is complete and correct.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the text box labeled "City."
|
||
- **Page Changes**: The text box is highlighted, indicating it is active and ready for input.
|
||
- **Possible Purpose**: My likely intent is to either edit or verify the city information in the address form.
|
||
|
||
2. **Action**: I type "Atlanta" into the "City" text box.
|
||
- **Page Changes**: The text "Atlanta" appears in the "City" field, replacing any previous text.
|
||
- **Possible Purpose**: I am updating or confirming the city name as part of editing the address details.
|
||
|
||
3. **Action**: I click on the text box labeled "Zip/Postal Code."
|
||
- **Page Changes**: The "Zip/Postal Code" text box is highlighted, indicating it is now active.
|
||
- **Possible Purpose**: My intention is to either edit or verify the zip code information in the address form.
|
||
|
||
4. **Action**: I type "30303" into the "Zip/Postal Code" text box.
|
||
- **Page Changes**: The text "30303" appears in the "Zip/Postal Code" field, replacing any previous text.
|
||
- **Possible Purpose**: I am updating or confirming the zip code as part of editing the address details.
|
||
|
||
5. **Action**: I hover over the "Save Address" button.
|
||
- **Page Changes**: The "Save Address" button is highlighted, indicating it is being hovered over.
|
||
- **Possible Purpose**: I am preparing to save the changes made to the address form.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions within this specific part of the video.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_My Account",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"My Account\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Address Book",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Address Book\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/account/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Emma Lopez 101 S San Mateo Dr San Mateo, California, 94010 United States T: 6505",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_text(\"Emma Lopez 101 S San Mateo Dr San Mateo, California, 94010 United States T: 6505\").nth(1)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Add New Address",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Add New Address\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/new/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Change Shipping Address",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Change Shipping Address\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Street Address: Line 1",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_label(\"Street Address: Line 1\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_State/Province",
|
||
"idx": 9,
|
||
"action_repr": "frame.selectOptionget_by_label(\"State/Province\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_City",
|
||
"idx": 11,
|
||
"action_repr": "frame.pressget_by_label(\"City\")CapsLock",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Zip/Postal Code",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_label(\"Zip/Postal Code\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Save Address",
|
||
"idx": 13,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Save Address\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7770/customer/address/edit/id/26/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |