340 lines
22 KiB
Plaintext
340 lines
22 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
What is the estimated driving time between the hometown of Joe Biden and Bridgeport?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action: Hover over the "Learn More" button**
|
||
- **Description**: I move the cursor over the "Learn More" button, which is located below the introductory text on the OpenStreetMap welcome modal.
|
||
- **Page Changes**: There are no immediate changes to the webpage as a result of this action; the button remains in its original state.
|
||
- **Possible Purpose**: The likely intent is to either prepare for clicking the button or to view any hover effects or tooltips that might provide additional information about the "Learn More" option.
|
||
|
||
#### 2. **Action: Click on the "Learn More" button**
|
||
- **Description**: I click on the "Learn More" button.
|
||
- **Page Changes**: After clicking, the welcome modal disappears, and the main map interface of OpenStreetMap becomes fully visible without any overlay.
|
||
- **Possible Purpose**: The purpose of this action is to close the welcome modal and access the full functionality of the OpenStreetMap website. This allows for interaction with the map and other features without the introductory message obstructing the view.
|
||
|
||
#### 3. **Action: Hover over the search bar at the top left corner**
|
||
- **Description**: I move the cursor over the search bar located at the top left corner of the page.
|
||
- **Page Changes**: No immediate changes occur as a result of this action; the search bar remains in its default state.
|
||
- **Possible Purpose**: The likely intent is to prepare for entering a search query into the search bar, possibly to find a specific location on the map.
|
||
|
||
#### 4. **Action: Click inside the search bar**
|
||
- **Description**: I click inside the search bar at the top left corner.
|
||
- **Page Changes**: The search bar becomes active, indicated by a blinking cursor inside it, ready for text input.
|
||
- **Possible Purpose**: The purpose of this action is to activate the search bar so that a location or keyword can be typed in to search for a specific area on the map.
|
||
|
||
#### 5. **Action: Type "New York" into the search bar**
|
||
- **Description**: I type the text "New York" into the active search bar.
|
||
- **Page Changes**: As I type, the text "New York" appears in the search bar. No search results are displayed yet since the search has not been initiated.
|
||
- **Possible Purpose**: The intent is to search for the location "New York" on the OpenStreetMap. This will likely center the map on New York once the search is executed.
|
||
|
||
#### 6. **Action: Press the "Enter" key on the keyboard**
|
||
- **Description**: I press the "Enter" key on the keyboard while the search bar is active and contains the text "New York".
|
||
- **Page Changes**: The map updates and centers on the location "New York". The map zooms in to provide a detailed view of the New York area.
|
||
- **Possible Purpose**: The purpose of pressing "Enter" is to execute the search query and display the corresponding location on the map. This allows for a focused view of New York and its surrounding areas.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website by first closing the welcome modal to access the full map interface. I then use the search functionality to find and display the location "New York" on the map. Each step is performed methodically to achieve the goal of viewing a specific area on the map.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the browser tab labeled "Wikipedia."
|
||
- **Page Changes:** The webpage transitions from displaying the OpenStreetMap interface to the Wikipedia homepage.
|
||
- **Possible Purpose:** The likely intent is to navigate away from the current page (OpenStreetMap) to access information or content available on Wikipedia.
|
||
|
||
#### 2. **Action:** I hover over the search bar on the Wikipedia homepage.
|
||
- **Page Changes:** No immediate changes occur; the cursor is positioned over the search bar, indicating readiness to input text.
|
||
- **Possible Purpose:** The intention is to prepare for a search query on Wikipedia, possibly to find specific information.
|
||
|
||
#### 3. **Action:** I click inside the search bar on the Wikipedia homepage.
|
||
- **Page Changes:** The search bar becomes active, and the cursor blinks inside it, ready for text input.
|
||
- **Possible Purpose:** The action aims to activate the search bar so that a search term can be entered.
|
||
|
||
#### 4. **Action:** I type "Geography" into the Wikipedia search bar.
|
||
- **Page Changes:** As I type, suggestions related to "Geography" appear below the search bar, offering autocomplete options.
|
||
- **Possible Purpose:** The goal is to search for information related to "Geography" on Wikipedia.
|
||
|
||
#### 5. **Action:** I press the Enter key after typing "Geography."
|
||
- **Page Changes:** The Wikipedia page navigates to the article titled "Geography," displaying detailed information about the subject.
|
||
- **Possible Purpose:** The intent is to access and read the Wikipedia article on Geography for information or research purposes.
|
||
|
||
---
|
||
|
||
This description strictly adheres to the observable actions within the provided video segment, focusing solely on the interactions with the UI elements and the resulting page changes.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a webpage open, displaying search results for "Joe Biden" on Wikipedia.
|
||
- **Page Changes:** The page shows various links related to Joe Biden, including articles about his family, political positions, and other related topics.
|
||
- **Possible Purpose:** The initial state sets the context, indicating that the user is researching information about Joe Biden, possibly to find specific details or verify information.
|
||
|
||
#### 2. **Hovering Over a Link**
|
||
- **Action:** I hover over a link titled "Family of Joe Biden."
|
||
- **Page Changes:** No immediate changes occur on the page; however, a tooltip or preview might appear, providing a brief summary or snippet of the linked page's content.
|
||
- **Possible Purpose:** The purpose of this action is likely to preview the content of the "Family of Joe Biden" page before deciding to click on it. This helps in assessing whether the linked page contains the desired information.
|
||
|
||
#### 3. **Clicking on the "Family of Joe Biden" Link**
|
||
- **Action:** I click on the "Family of Joe Biden" link.
|
||
- **Page Changes:** The webpage transitions to the "Family of Joe Biden" Wikipedia page. This new page provides detailed information about Joe Biden's family members, their backgrounds, and connections.
|
||
- **Possible Purpose:** The intent here is to access comprehensive information about Joe Biden's family. This could be for research purposes, personal interest, or to gather specific details about his family history and relationships.
|
||
|
||
#### 4. **Scrolling Down the Page**
|
||
- **Action:** I scroll down the "Family of Joe Biden" page.
|
||
- **Page Changes:** As I scroll, different sections of the page become visible, including details about Joe Biden's immediate family, extended family, and historical background.
|
||
- **Possible Purpose:** The scrolling action suggests that I am looking for specific information that is not immediately visible at the top of the page. This could involve finding details about particular family members, historical context, or other relevant data.
|
||
|
||
#### 5. **Pausing on the "Immediate Family" Section**
|
||
- **Action:** I pause at the "Immediate Family" section.
|
||
- **Page Changes:** The "Immediate Family" section becomes the focal point, displaying information about Joe Biden's spouse, children, and other close relatives.
|
||
- **Possible Purpose:** The pause indicates that I am reading and possibly analyzing the information in this section. The focus on the "Immediate Family" suggests an interest in understanding Joe Biden's closest familial relationships and their significance.
|
||
|
||
#### 6. **Highlighting Text**
|
||
- **Action:** I highlight text within the "Immediate Family" section.
|
||
- **Page Changes:** The highlighted text stands out, making it easier to focus on specific details within the section.
|
||
- **Possible Purpose:** Highlighting text is often done to emphasize important information, make notes, or prepare to copy the text. This action suggests that I find the highlighted information particularly relevant or noteworthy.
|
||
|
||
#### 7. **Continuing to Scroll**
|
||
- **Action:** I continue scrolling down the page.
|
||
- **Page Changes:** Further sections of the page become visible, including additional details about Joe Biden's family history and connections.
|
||
- **Possible Purpose:** The continued scrolling indicates an ongoing search for more information. This could involve looking for comprehensive details, verifying information, or exploring related topics mentioned on the page.
|
||
|
||
### Summary
|
||
In this video segment, I begin by hovering over and then clicking on the "Family of Joe Biden" link from a search results page. After navigating to the corresponding Wikipedia page, I scroll through the content, pausing to focus on the "Immediate Family" section and highlighting specific text. The actions suggest a deliberate effort to gather and analyze detailed information about Joe Biden's family, with a particular emphasis on his immediate family relationships. Each step is methodical, indicating a focused research or information-gathering process.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action**: I click on the text "Scranton, Lackawanna County, United States" within the Wikipedia page about Joe Biden.
|
||
- **Page Changes**: The webpage transitions to a new section or opens a new tab displaying information related to Scranton, Lackawanna County, United States.
|
||
- **Possible Purpose**: The likely intent is to gather more detailed information about the location mentioned in Joe Biden's biography, possibly to understand its geographical or historical context.
|
||
|
||
2. **Action**: I navigate to the OpenStreetMap website.
|
||
- **Page Changes**: The webpage now displays the OpenStreetMap interface, featuring a map of the world with various interactive elements such as a search bar, buttons for "Learn More" and "Start Mapping," and a map centered on a default location.
|
||
- **Possible Purpose**: The transition to OpenStreetMap suggests an intention to visualize or explore the geographical location of Scranton, Lackawanna County, United States, in more detail using a mapping tool.
|
||
|
||
3. **Action**: I click on the search bar located at the top-left corner of the OpenStreetMap page.
|
||
- **Page Changes**: The search bar becomes active, allowing text input. A small dropdown menu appears below the search bar, ready to provide suggestions based on the entered text.
|
||
- **Possible Purpose**: The purpose is to enter a specific location into the search bar to find and display it on the map.
|
||
|
||
4. **Action**: I type "Scranton, Lackawanna County, United States" into the active search bar.
|
||
- **Page Changes**: As I type, the dropdown menu updates with suggested locations that match the entered text. Once the full location name is typed, the suggestion "Scranton, Lackawanna County, United States" appears in the dropdown list.
|
||
- **Possible Purpose**: The intent is to locate and select the exact geographical area of interest on the map for further examination.
|
||
|
||
5. **Action**: I click on the suggestion "Scranton, Lackawanna County, United States" in the dropdown menu.
|
||
- **Page Changes**: The map centers on the location of Scranton, Lackawanna County, United States, with a marker pinpointing the exact area. The surrounding region is also displayed, providing context for the location.
|
||
- **Possible Purpose**: The goal is to visually confirm the location's position and get a better understanding of its geographical context using the map.
|
||
|
||
6. **Action**: I click on the "Reverse Directions" button located below the search bar.
|
||
- **Page Changes**: The interface rearranges to show two input fields for directions: "From" and "To." The previously searched location "Scranton, Lackawanna County, United States" is automatically populated in the "To" field, and the "From" field is left blank, ready for input.
|
||
- **Possible Purpose**: The action suggests an intention to plan a route or direction leading to Scranton, Lackawanna County, United States, from another unspecified starting point. This could be for planning travel or understanding the accessibility of the location.
|
||
|
||
### Summary:
|
||
The sequence of actions involves transitioning from reading about Joe Biden's birthplace on Wikipedia to locating and examining this place (Scranton, Lackawanna County, United States) on OpenStreetMap. The steps include searching for and selecting the location on the map and then initiating a direction query to or from this location, indicating a focus on understanding its geographical context and potentially planning a route.
|
||
|
||
---
|
||
|
||
## Part 5
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the search bar at the top of the OpenStreetMap webpage.
|
||
- **Page Changes**: The cursor is now active inside the search bar, ready for input.
|
||
- **Possible Purpose**: The likely intent is to enter a location or address to find its position on the map.
|
||
|
||
#### 2. **Action**: I type "Bridgeport" into the search bar.
|
||
- **Page Changes**: As I type, a dropdown menu appears below the search bar, displaying search suggestions related to "Bridgeport."
|
||
- **Possible Purpose**: The purpose is to select a specific location named "Bridgeport" from the suggestions provided by the map service.
|
||
|
||
#### 3. **Action**: I select "Bridgeport, Greater Bridgeport Planning Region, Connecticut, United States" from the dropdown menu.
|
||
- **Page Changes**: The map centers on Bridgeport, Connecticut, and a marker appears at the selected location. Additionally, the directions panel updates to reflect this destination.
|
||
- **Possible Purpose**: The intent is to set Bridgeport as the destination for route planning.
|
||
|
||
#### 4. **Action**: I click on the "Directions" button located near the search result.
|
||
- **Page Changes**: A directions panel opens on the left side of the screen, prompting for a starting point.
|
||
- **Possible Purpose**: The goal is to initiate the process of getting directions to Bridgeport.
|
||
|
||
#### 5. **Action**: I click on the starting point field in the directions panel.
|
||
- **Page Changes**: The starting point field becomes active, allowing for input.
|
||
- **Possible Purpose**: The intention is to specify the starting location for the route.
|
||
|
||
#### 6. **Action**: I type "Scranton" into the starting point field.
|
||
- **Page Changes**: Another dropdown menu appears with suggestions related to "Scranton."
|
||
- **Possible Purpose**: The aim is to select Scranton as the starting point for the journey.
|
||
|
||
#### 7. **Action**: I select "Scranton, Lackawanna County, United States" from the dropdown menu.
|
||
- **Page Changes**: The map now displays a route from Scranton to Bridgeport, with the directions panel updating to show the step-by-step instructions and total distance/time.
|
||
- **Possible Purpose**: The purpose is to finalize the route setup from Scranton to Bridgeport and view the detailed directions.
|
||
|
||
#### 8. **Action**: I scroll through the directions listed in the panel.
|
||
- **Page Changes**: The directions text scrolls, revealing the full list of steps for the journey.
|
||
- **Possible Purpose**: The intent is to review the complete set of directions for the planned route.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website to plan a route from Scranton to Bridgeport. My actions include searching for both the starting and destination locations, selecting them from the suggestions, initiating the directions feature, and reviewing the generated route details. Each step is performed methodically to achieve the goal of obtaining clear navigation instructions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "label_Search 'Wikipedia'",
|
||
"idx": 1,
|
||
"action_repr": "frame.pressget_by_label(\"Search 'Wikipedia'\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Family of Joe Biden",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Family of Joe Biden\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/search?content=wikipedia_en_all_maxi_2022-05&pattern=joe+biden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/search?content=wikipedia_en_all_maxi_2022-05&pattern=joe+biden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "cell_Joe Biden\u2022 Jill Biden\u2022 Beau Biden\u2022 Hunter Biden\u2022 Naomi Biden\u2022 Ashley Biden",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"cell\", name=\"Joe Biden\u2022 Jill Biden\u2022 Beau Biden\u2022 Hunter Biden\u2022 Naomi Biden\u2022 Ashley Biden\").get_by_title(\"Joe Biden\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Joe_Biden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Family_of_Joe_Biden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Joe_Biden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Joe_Biden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 6,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Joe_Biden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Joe_Biden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 8,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |