337 lines
20 KiB
Plaintext
337 lines
20 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Show me the way from Carnegie Mellon University to the home stadium of Boston home NBA team
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar labeled "Search."
|
||
- **Page Changes:** The search bar becomes active, and a blinking cursor appears inside it, indicating that it is ready for text input.
|
||
- **Possible Purpose:** The likely intent is to enter a location or keyword to search for specific places or features on the OpenStreetMap.
|
||
|
||
#### 2. **Action:** I hover over the "Sign Up" button located in the top-right corner of the page.
|
||
- **Page Changes:** There are no immediate visual changes to the page; however, the "Sign Up" button may become slightly highlighted or underlined, depending on the website's design.
|
||
- **Possible Purpose:** The action suggests an intention to create a new account on OpenStreetMap, possibly to access additional features or contribute to the map.
|
||
|
||
#### 3. **Action:** I move the cursor to the "Start Mapping" button and click on it.
|
||
- **Page Changes:** The page transitions to a new view, likely directing me to a section where I can begin contributing to the map or accessing mapping tools. This change is not fully visible in the provided segment but is a typical response to clicking such a button.
|
||
- **Possible Purpose:** The intent is to start engaging with the mapping features of OpenStreetMap, either by editing the map, adding data, or exploring mapping functionalities.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website by activating the search bar, hovering over the "Sign Up" button, and then clicking the "Start Mapping" button. Each action is performed with the apparent purpose of navigating the site to either search for locations, create an account, or begin using the mapping tools. The observed actions are straightforward and align with common user behaviors on a mapping platform.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
In this video segment, the following actions are observed:
|
||
|
||
1. **Action**: I click on the "Sign Up" link located in the top-right corner of the webpage.
|
||
- **Page Changes**: The page does not navigate away or display a new window; instead, it highlights the "Sign Up" link, indicating an interaction but no immediate transition to a different page.
|
||
- **Possible Purpose**: The likely intent is to initiate the process of creating a new account on OpenStreetMap. However, since there is no navigation to a sign-up form, the action might be demonstrating the location of the "Sign Up" link or testing its functionality.
|
||
|
||
2. **Action**: I move the cursor over the map area without clicking.
|
||
- **Page Changes**: There are no changes to the webpage as a result of this action.
|
||
- **Possible Purpose**: The purpose could be to explore the map visually or to prepare for a subsequent interaction with the map, such as zooming or selecting a specific location.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the described part without incorporating details from other segments.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I observe the following actions being performed on the OpenStreetMap website:
|
||
|
||
### 1. **Action**: I click on the "Sign Up" button located in the top-right corner of the webpage.
|
||
- **Page Changes**: A small dropdown menu appears below the "Sign Up" button, displaying the text "get_by_role("link", name="Sign Up")".
|
||
- **Possible Purpose**: The likely intent is to initiate the process of creating a new account on OpenStreetMap. This action is typically taken by users who want to contribute to the map or access features that require an account.
|
||
|
||
### 2. **Action**: I move the cursor away from the "Sign Up" button.
|
||
- **Page Changes**: The dropdown menu with the text "get_by_role("link", name="Sign Up")" disappears, returning the page to its original state.
|
||
- **Possible Purpose**: This action might be exploratory, where the user is assessing the functionality of the "Sign Up" button without immediately proceeding to create an account. It could also be an accidental hover that was corrected by moving the cursor away.
|
||
|
||
### Summary:
|
||
- The primary action in this segment is interacting with the "Sign Up" button, which triggers a brief display of a dropdown menu with specific text.
|
||
- The secondary action involves moving the cursor away, reverting the page to its initial state.
|
||
- These actions suggest an interest in the account creation process but do not result in actually starting the sign-up procedure.
|
||
|
||
This description strictly adheres to the observable actions within the provided video segment, without incorporating any additional context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the search bar labeled "Search Wikipedia".
|
||
- **Page Changes:** The search bar becomes active, and a dropdown menu appears below it, suggesting search terms as I begin to type.
|
||
- **Possible Purpose:** The likely intent is to initiate a search for specific information or a topic on Wikipedia.
|
||
|
||
2. **Action:** I type "Boston NBA" into the search bar.
|
||
- **Page Changes:** As I type, the dropdown menu updates in real-time, displaying suggestions related to "Boston NBA", such as "Boston Celtics (NBA)" and other relevant entries.
|
||
- **Possible Purpose:** The purpose is to find information specifically related to the Boston Celtics or other NBA-related topics associated with Boston.
|
||
|
||
3. **Action:** I select the suggestion "Boston Celtics (NBA)" from the dropdown menu.
|
||
- **Page Changes:** The webpage transitions to the Wikipedia page for the "Boston Celtics (NBA)", displaying detailed information about the team, including history, players, and statistics.
|
||
- **Possible Purpose:** The intent is to access detailed information about the Boston Celtics, a professional basketball team in the NBA.
|
||
|
||
### Summary:
|
||
In this video segment, I interact with the Wikipedia search functionality by clicking on the search bar, typing "Boston NBA", and selecting the suggestion "Boston Celtics (NBA)" from the dropdown menu. This sequence of actions results in navigating to the specific Wikipedia page for the Boston Celtics, indicating a focused search for information about this NBA team.
|
||
|
||
---
|
||
|
||
## Part 5
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a webpage open on "OpenStreetMap," which is a map of the world created by user contributions.
|
||
- **Page Changes:** The page displays a welcome message, "Welcome to OpenStreetMap!" along with a search bar, various menu options (like "Learn More" and "Start Mapping"), and a map centered around the northeastern United States.
|
||
- **Possible Purpose:** The initial state sets the context for the actions that follow, indicating the user is about to interact with this mapping website.
|
||
|
||
#### 2. **Clicking the Search Bar**
|
||
- **Action:** I click on the search bar located at the top left corner of the page.
|
||
- **Page Changes:** The cursor focuses inside the search bar, allowing text input.
|
||
- **Possible Purpose:** The purpose of this action is to initiate a search query, likely to find a specific location or feature on the map.
|
||
|
||
#### 3. **Typing in the Search Bar**
|
||
- **Action:** I type "Boston Celtics" into the search bar.
|
||
- **Page Changes:** As I type, the webpage does not immediately change, but it is preparing for a search submission.
|
||
- **Possible Purpose:** The intent is to search for information related to "Boston Celtics" on the OpenStreetMap, possibly to locate a place associated with the team or to see if there are any mapped features tagged with this name.
|
||
|
||
#### 4. **Submitting the Search Query**
|
||
- **Action:** I press the "Enter" key or click the search button next to the search bar.
|
||
- **Page Changes:** The page transitions to display search results. However, instead of showing relevant map locations, it appears to redirect to a Wikipedia page about the Boston Celtics, as indicated by the visible text and layout.
|
||
- **Possible Purpose:** The redirection to a Wikipedia page suggests that the search functionality might have linked to an external resource for detailed information about the Boston Celtics, rather than finding a specific location on the map.
|
||
|
||
#### 5. **Observing the Redirected Page**
|
||
- **Action:** I observe the content of the redirected Wikipedia page.
|
||
- **Page Changes:** The page now shows detailed information about the Boston Celtics, including their history, achievements, and related statistics.
|
||
- **Possible Purpose:** The purpose of observing this page could be to gather additional context or information about the Boston Celtics, even though it was not the initial expectation from the OpenStreetMap search.
|
||
|
||
### Summary
|
||
In this video segment, I start on the OpenStreetMap homepage, use the search bar to look for "Boston Celtics," submit the search, and end up on a Wikipedia page about the Boston Celtics. The sequence of actions reflects an attempt to find map-related information, but the result leads to textual information instead. Each step is clearly documented based on the observable UI interactions and page changes.
|
||
|
||
---
|
||
|
||
## Part 6
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "From" text box.
|
||
- **Page Changes:** The "From" text box is highlighted, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a starting location for a route or direction.
|
||
|
||
#### 2. **Action:** I click on the "Car (OSRM)" dropdown menu.
|
||
- **Page Changes:** The dropdown menu expands, showing different transportation options.
|
||
- **Possible Purpose:** The intent is to select a mode of transportation for the route planning.
|
||
|
||
#### 3. **Action:** I click on the "Go" button next to the "Car (OSRM)" dropdown.
|
||
- **Page Changes:** There is no immediate visible change in the main map area, but the system might be processing the request or waiting for additional inputs.
|
||
- **Possible Purpose:** The intent is to initiate a route search or direction calculation based on the selected transportation mode.
|
||
|
||
#### 4. **Action:** I hover over the "Share" button located in the top-right corner of the map.
|
||
- **Page Changes:** A tooltip or additional options related to sharing the map view may appear, though this is not explicitly shown.
|
||
- **Possible Purpose:** The intent is to explore options for sharing the current map view or route information with others.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap interface by activating the "From" text box, selecting a transportation mode from the dropdown menu, initiating a route search, and exploring the "Share" button options. Each action appears to be part of setting up and potentially sharing a route or map view.
|
||
|
||
---
|
||
|
||
## Part 7
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "From" textbox.
|
||
- **Page Changes:** The textbox becomes active, and a dropdown menu appears below it, suggesting recent or popular locations.
|
||
- **Possible Purpose:** The likely intent is to enter a starting location for a route.
|
||
|
||
#### 2. **Action:** I type "Carnegie Mellon University, Schenley Drive" into the "From" textbox.
|
||
- **Page Changes:** As I type, the dropdown menu updates with search suggestions that match the input text.
|
||
- **Possible Purpose:** The purpose is to specify the exact starting point for the route by selecting it from the suggestions.
|
||
|
||
#### 3. **Action:** I select "Carnegie Mellon University, Schenley Drive" from the dropdown menu.
|
||
- **Page Changes:** The selected location is confirmed in the "From" textbox, and the map updates to center around this location.
|
||
- **Possible Purpose:** This action finalizes the starting point for the route planning.
|
||
|
||
#### 4. **Action:** I click on the "To" textbox.
|
||
- **Page Changes:** The "To" textbox becomes active, and a dropdown menu appears, similar to the "From" textbox.
|
||
- **Possible Purpose:** The intent is to enter a destination location for the route.
|
||
|
||
#### 5. **Action:** I type "Boston Celtics" into the "To" textbox.
|
||
- **Page Changes:** The dropdown menu updates with search suggestions that match "Boston Celtics."
|
||
- **Possible Purpose:** The goal is to specify the destination for the route by selecting an appropriate suggestion.
|
||
|
||
#### 6. **Action:** I select "td_garden, 100, Legends Way, West End, I..." from the dropdown menu under the "To" textbox.
|
||
- **Page Changes:** The selected location is confirmed in the "To" textbox, and the map updates to show both the starting and ending points of the route.
|
||
- **Possible Purpose:** This action sets the destination point for the route planning.
|
||
|
||
#### 7. **Action:** I click the "Go" button next to the "To" textbox.
|
||
- **Page Changes:** The map displays a calculated route from Carnegie Mellon University to TD Garden, with step-by-step directions appearing on the left side of the screen.
|
||
- **Possible Purpose:** The intent is to generate and display the optimal route between the specified start and end points.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to plan a route using OpenStreetMap. I begin by specifying a starting location ("Carnegie Mellon University, Schenley Drive") and a destination ("TD Garden, Boston Celtics"). After confirming these points, I initiate the route calculation, resulting in a displayed route and detailed directions. Each step is methodically executed to achieve the goal of obtaining navigation instructions from the chosen start to the end point.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "label_Search 'Wikipedia'",
|
||
"idx": 1,
|
||
"action_repr": "frame.pressget_by_label(\"Search 'Wikipedia'\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Boston Celtics all-time roster",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Boston Celtics all-time roster\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/search?content=wikipedia_en_all_maxi_2022-05&pattern=Boston+NBA"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/search?content=wikipedia_en_all_maxi_2022-05&pattern=Boston+NBA"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Boston Celtics",
|
||
"idx": 3,
|
||
"action_repr": "frame.clicklocator(\"#mf-section-0\").get_by_role(\"link\", name=\"Boston Celtics\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Boston_Celtics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Boston_Celtics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"p\").filter(has_text=\"The Boston Celtics (/\u02c8s\u025blt\u026aks/ SEL-tiks) are an American professional basketball\").get_by_title(\"TD Garden\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/Boston_Celtics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 7,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Tab",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 9,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8888/wikipedia_en_all_maxi_2022-05/A/TD_Garden"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |