427 lines
19 KiB
Plaintext
427 lines
19 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Given the following locations, ['Massachusetts Institute of Technology', 'Harvard University', 'Boston Logan International Airport'], what would be the optimal route to travel through them all in order to minimize total travel time? Please note the journey begins at the first place listed.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Actions and Observations
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the search bar located at the top left corner of the page.
|
||
- **Page Changes:** The search bar becomes active, allowing text input.
|
||
- **Possible Purpose:** The likely intent is to enter a location or address to find it on the map.
|
||
|
||
#### Step 2:
|
||
- **Action:** I type "Where is this?" into the search bar.
|
||
- **Page Changes:** No immediate changes occur on the page as the text is being typed.
|
||
- **Possible Purpose:** The purpose is to query the map for information about a specific location.
|
||
|
||
#### Step 3:
|
||
- **Action:** I press the "Go" button next to the search bar.
|
||
- **Page Changes:** The map updates to show a new location based on the search query. However, since "Where is this?" is not a specific location, the result might be unexpected or default.
|
||
- **Possible Purpose:** To execute the search and display the corresponding location on the map.
|
||
|
||
#### Step 4:
|
||
- **Action:** I click on the "Directions" tab or icon, which appears on the left side of the screen.
|
||
- **Page Changes:** A directions panel opens, displaying fields for entering start and destination locations.
|
||
- **Possible Purpose:** To set up a route between two points on the map.
|
||
|
||
#### Step 5:
|
||
- **Action:** I click on the "From" textbox in the directions panel.
|
||
- **Page Changes:** The "From" textbox becomes active, ready for input.
|
||
- **Possible Purpose:** To specify the starting point of the route.
|
||
|
||
#### Step 6:
|
||
- **Action:** I type "MIT" into the "From" textbox.
|
||
- **Page Changes:** As I type, suggestions appear below the textbox, listing possible matches for "MIT."
|
||
- **Possible Purpose:** To select "Massachusetts Institute of Technology" as the starting point.
|
||
|
||
#### Step 7:
|
||
- **Action:** I select "Massachusetts Institute of Technology, Allston Street, Cambridge, MA" from the suggestion list.
|
||
- **Page Changes:** The selected location is filled into the "From" textbox, and a marker appears on the map at the specified location.
|
||
- **Possible Purpose:** To confirm the exact starting point for the route.
|
||
|
||
#### Step 8:
|
||
- **Action:** I click on the "To" textbox in the directions panel.
|
||
- **Page Changes:** The "To" textbox becomes active, ready for input.
|
||
- **Possible Purpose:** To specify the destination point of the route.
|
||
|
||
#### Step 9:
|
||
- **Action:** I type "Harvard University" into the "To" textbox.
|
||
- **Page Changes:** Suggestions appear below the textbox, listing possible matches for "Harvard University."
|
||
- **Possible Purpose:** To select "Harvard University" as the destination.
|
||
|
||
#### Step 10:
|
||
- **Action:** I select "Harvard University, Hurilbut Street, Avon Hill, Cambridge, MA" from the suggestion list.
|
||
- **Page Changes:** The selected location is filled into the "To" textbox, and a marker appears on the map at the specified location.
|
||
- **Possible Purpose:** To confirm the exact destination for the route.
|
||
|
||
#### Step 11:
|
||
- **Action:** I click the "Go" button in the directions panel.
|
||
- **Page Changes:** The map displays a route between MIT and Harvard University, with step-by-step directions listed in the panel.
|
||
- **Possible Purpose:** To generate and display the route and directions from the starting point to the destination.
|
||
|
||
### Summary
|
||
In this segment, I interact with the OpenStreetMap website to search for a location and then set up and display directions between two specific points (MIT and Harvard University). Each action is focused on navigating the interface to achieve these goals, with clear responses from the webpage indicating successful execution of the tasks.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the "My Notes" section.
|
||
- **Page Changes**: A note-taking interface appears, overlaying part of the map and directions panel. The interface includes a placeholder text: "Type your note here...".
|
||
- **Possible Purpose**: The likely intent is to add personal notes or annotations related to the current map view or directions.
|
||
|
||
#### 2. **Action**: I click inside the "Type your note here..." text box.
|
||
- **Page Changes**: The text box becomes active, allowing for text input. However, no text is typed in this segment.
|
||
- **Possible Purpose**: The intention is to prepare for typing a note, possibly to document specific details about the route or locations shown on the map.
|
||
|
||
#### 3. **Action**: I click on the "History" tab located in the top navigation bar.
|
||
- **Page Changes**: The main content area transitions to display the history page of OpenStreetMap. This page provides information about the project's background, including its creation by UCL, Fastly, Bytemark Hosting, and other partners. It also features options like "Learn More" and "Start Mapping".
|
||
- **Possible Purpose**: The goal is to access historical information or background details about OpenStreetMap, which might be relevant for understanding the context or reliability of the map data being used.
|
||
|
||
#### 4. **Action**: I click on the "From" textbox in the Directions panel.
|
||
- **Page Changes**: The "From" textbox is activated, indicating readiness for input or modification of the starting location for the directions.
|
||
- **Possible Purpose**: The intention is to either edit the current starting location or confirm it before proceeding with further actions, such as recalculating the route or adding a new destination.
|
||
|
||
---
|
||
|
||
These actions are described based solely on the observable elements within the provided video segment, without incorporating any external context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Go" button located next to the destination input field.
|
||
- **Page Changes:** The page transitions to display a route map with directions from Harvard University to Logan International Airport. The left panel now shows detailed step-by-step directions for the journey, including distance and estimated time.
|
||
- **Possible Purpose:** The likely intent is to generate and view the driving directions between the specified starting point (Harvard University) and destination (Logan International Airport).
|
||
|
||
#### 2. **Action:** I click on the text area within the "My Notes" section.
|
||
- **Page Changes:** The text area becomes active, allowing for text input. The cursor appears inside the text area, indicating readiness for typing.
|
||
- **Possible Purpose:** The intention is to add or edit notes related to the current task or information displayed on the page.
|
||
|
||
#### 3. **Action:** I type the text "MIT - Harvard: 7 mins" into the active text area.
|
||
- **Page Changes:** The typed text appears in the text area under the heading "My Notes."
|
||
- **Possible Purpose:** The purpose is to record specific information, possibly the travel time between MIT and Harvard, for reference or future use.
|
||
|
||
#### 4. **Action:** I continue typing "Harvard - Airport 19 mins" into the same text area.
|
||
- **Page Changes:** The additional text is appended below the previously entered note, updating the content of the "My Notes" section.
|
||
- **Possible Purpose:** This action aims to document another piece of relevant information, in this case, the travel time from Harvard to the airport, for organizational or planning purposes.
|
||
|
||
#### 5. **Action:** I click on the "Add Note" button located below the text area in the "My Notes" section.
|
||
- **Page Changes:** There is no immediate visible change to the page layout or content after clicking the button. However, the action might be intended to save or confirm the entered notes.
|
||
- **Possible Purpose:** The likely intent is to save the entered notes, ensuring they are stored or confirmed within the application.
|
||
|
||
---
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website by generating driving directions and documenting relevant travel times in a notes section. Each action is focused on obtaining and recording specific route information, suggesting a workflow centered around planning or organizing travel details.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:**
|
||
I click on the text box within the "My Notes" section.
|
||
|
||
**Page Changes:**
|
||
The text box becomes active, allowing me to type or edit the content inside it.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to update or add information to the notes. This could involve correcting existing text, adding new details, or organizing the information for clarity.
|
||
|
||
---
|
||
|
||
#### 2. **Action:**
|
||
I type "MIT - Airport 14 mins" into the active text box.
|
||
|
||
**Page Changes:**
|
||
The text "MIT - Airport 14 mins" appears as part of the existing content in the "My Notes" section.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to record the travel time between MIT and the Airport, possibly for future reference or planning purposes.
|
||
|
||
---
|
||
|
||
#### 3. **Action:**
|
||
I click on the "Add Note" button located below the text box.
|
||
|
||
**Page Changes:**
|
||
The newly typed note "MIT - Airport 14 mins" is confirmed and added to the list of notes. The text box clears, ready for a new entry.
|
||
|
||
**Possible Purpose:**
|
||
This action confirms the addition of the note to the list, ensuring it is saved and visible for future access.
|
||
|
||
---
|
||
|
||
#### 4. **Action:**
|
||
I click on the text box again to make another entry.
|
||
|
||
**Page Changes:**
|
||
The text box becomes active once more, ready for input.
|
||
|
||
**Possible Purpose:**
|
||
The intention is to add another piece of information to the notes, continuing the process of documenting relevant details.
|
||
|
||
---
|
||
|
||
#### 5. **Action:**
|
||
I type "Airport - Harvard 18 mins" into the active text box.
|
||
|
||
**Page Changes:**
|
||
The text "Airport - Harvard 18 mins" appears in the text box, ready to be added to the notes.
|
||
|
||
**Possible Purpose:**
|
||
This action records the travel time between the Airport and Harvard, further compiling travel information for reference.
|
||
|
||
---
|
||
|
||
#### 6. **Action:**
|
||
I click the "Add Note" button again.
|
||
|
||
**Page Changes:**
|
||
The note "Airport - Harvard 18 mins" is added to the list of notes in the "My Notes" section. The text box clears, indicating readiness for the next entry.
|
||
|
||
**Possible Purpose:**
|
||
This confirms and saves the new note, ensuring all relevant travel times are documented and organized.
|
||
|
||
---
|
||
|
||
#### 7. **Action:**
|
||
I scroll down slightly to view the "History" section below the "My Notes" section.
|
||
|
||
**Page Changes:**
|
||
The "History" section becomes visible, showing previous entries or actions related to the notes.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to review past entries or actions, possibly to verify information, check for duplicates, or understand previous updates.
|
||
|
||
---
|
||
|
||
### Summary
|
||
In this video segment, I interact with the "My Notes" section by adding two new notes regarding travel times ("MIT - Airport 14 mins" and "Airport - Harvard 18 mins"). Each note is confirmed using the "Add Note" button. Finally, I scroll to view the "History" section, likely to review or reference past entries. These actions suggest a focus on documenting and organizing travel-related information efficiently.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 78,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Tab",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 82,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Scratchpad",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Scratchpad\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Type your note here...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.clicklocator(\"#sidebar > .search_forms > .directions_form > .d-flex\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Close",
|
||
"idx": 8,
|
||
"action_repr": "frame.clicklocator(\"#sidebar form\").filter(has_text=\"Bicycle (OSRM)Car (OSRM)Foot (OSRM) Go Reverse Directions Loading...\").get_by_label(\"Close\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/?query=MIT#map=15/42.3609/-71.1201"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/?query=MIT#map=15/42.3609/-71.1201"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_16",
|
||
"idx": 16,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/?query=MIT#map=15/42.3609/-71.1201"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/?query=MIT#map=15/42.3609/-71.1201"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_17",
|
||
"idx": 17,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_car&route=42.3679%2C-71.1268%3B42.3632%2C-71.0136#map=13/42.3576/-71.0742"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_car&route=42.3679%2C-71.1268%3B42.3632%2C-71.0136#map=13/42.3576/-71.0742"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_79",
|
||
"idx": 79,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Type your note here...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Reverse Directions",
|
||
"idx": 80,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Reverse Directions\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:4399/scratchpad.html"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |