223 lines
11 KiB
Plaintext
223 lines
11 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
What is the zip code of Chatham University?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State and First Action
|
||
|
||
**Action:**
|
||
I click on the search bar located at the top left corner of the OpenStreetMap webpage.
|
||
|
||
**Page Changes:**
|
||
The search bar becomes active, allowing text input. A placeholder text "Where is this?" is visible inside the search bar.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to initiate a search for a specific location or point of interest on the map.
|
||
|
||
---
|
||
|
||
### Second Action
|
||
|
||
**Action:**
|
||
I type "chatham university" into the active search bar.
|
||
|
||
**Page Changes:**
|
||
As I type, the webpage processes the input. After completing the entry, a list of search results appears below the search bar under the heading "Search Results." The results are from "OpenStreetMap Nominatim," showing two entries:
|
||
1. "University Chatham University, North Woodland Road, Squirrel Hill North, Pittsburgh, Allegheny County, 15232, United States"
|
||
2. "University Building Chatham University, Fifth Avenue, Larimer, Pittsburgh, Allegheny County, 15208, United States"
|
||
|
||
**Possible Purpose:**
|
||
The purpose of typing "chatham university" is to find and select the specific location of Chatham University on the map.
|
||
|
||
---
|
||
|
||
### Third Action
|
||
|
||
**Action:**
|
||
I click on the second search result titled "University Building Chatham University, Fifth Avenue, Larimer, Pittsburgh, Allegheny County, 15208, United States."
|
||
|
||
**Page Changes:**
|
||
Upon clicking the result, the map shifts focus to the selected location. An orange outline appears around the area representing Chatham University on Fifth Avenue. Additionally, a sidebar on the left side of the screen displays detailed information about the selected location, including tags such as "amenity," "name," "website," and other metadata.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to view the exact location and details of the Chatham University building on Fifth Avenue within the map interface.
|
||
|
||
---
|
||
|
||
### Fourth Action
|
||
|
||
**Action:**
|
||
I scroll down in the sidebar to view more details about the selected location.
|
||
|
||
**Page Changes:**
|
||
Scrolling reveals additional information in the sidebar, such as node details, Wikipedia links, and other relevant data associated with the mapped location.
|
||
|
||
**Possible Purpose:**
|
||
The purpose of scrolling is to gather comprehensive information about the selected location, ensuring a thorough understanding of its attributes and context within OpenStreetMap.
|
||
|
||
---
|
||
|
||
### Summary of Observed Actions:
|
||
1. **Click on the search bar** to activate it.
|
||
2. **Type "chatham university"** into the search bar.
|
||
3. **Click on the second search result** for "University Building Chatham University, Fifth Avenue."
|
||
4. **Scroll down in the sidebar** to view more details about the selected location.
|
||
|
||
Each action is performed sequentially to locate and examine the specific details of Chatham University on OpenStreetMap.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the OpenStreetMap website open, displaying a map centered on Chatham University. The left panel shows detailed information about "Way: Chatham University (172206707) Version #4."
|
||
- **Page Changes:** The map highlights the area corresponding to Chatham University with an orange outline. The left panel lists tags such as `amenity`, `ele`, `gnis:county_id`, etc., associated with this specific version of the way.
|
||
- **Possible Purpose:** The initial state suggests that the user is reviewing or analyzing the data for Chatham University in OpenStreetMap, possibly for verification or editing purposes.
|
||
|
||
#### 2. **Clicking on "History"**
|
||
- **Action:** I click on the "History" tab located at the top of the left panel.
|
||
- **Page Changes:** The page transitions to display different versions of the "Way: Chatham University." Specifically, it shows "Version #3" with its own set of tags and edit details.
|
||
- **Possible Purpose:** By accessing the history, I am likely intending to compare different versions of the Chatham University entry to understand changes over time or to revert to a previous version if necessary.
|
||
|
||
#### 3. **Observing "Version #3" Details**
|
||
- **Action:** I review the details of "Version #3" presented in the left panel.
|
||
- **Page Changes:** The panel now displays tags and metadata specific to Version #3, such as `building`, `name`, and `Nodes` listed below the tags.
|
||
- **Possible Purpose:** This action indicates an intent to closely examine the specifics of Version #3, perhaps to identify differences from the current version or to assess the quality and accuracy of the data at that point in time.
|
||
|
||
#### 4. **Using the Search Function**
|
||
- **Action:** I click on the search bar at the top left corner of the screen and type "chatham university."
|
||
- **Page Changes:** As I type, a dropdown menu appears with search suggestions. After completing the entry, I press the "Go" button next to the search bar.
|
||
- **Possible Purpose:** The purpose of this action is to perform a new search for "chatham university" within OpenStreetMap, potentially to find related entries or to verify the consistency of data across different search results.
|
||
|
||
#### 5. **Reviewing Search Results**
|
||
- **Action:** I observe the search results displayed below the search bar.
|
||
- **Page Changes:** The search results show multiple entries related to "chatham university," including a specific result for "University Chatham University, North Woodland Road, Squirrel Hill North, Pittsburgh, Allegheny County, 15208, United States."
|
||
- **Possible Purpose:** By reviewing the search results, I am likely ensuring that the correct entity is being referenced or exploring additional related data points within OpenStreetMap.
|
||
|
||
### Summary
|
||
In this video segment, I begin by examining "Version #4" of Chatham University on OpenStreetMap. I then access the history to view "Version #3," comparing its details. Following this, I use the search function to look up "chatham university," reviewing the provided search results. Each step appears methodical, aimed at verifying, comparing, or gathering comprehensive information about Chatham University within the OpenStreetMap database.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 11,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"Search\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Chatham University, North Woodland Road, Squirrel Hill North, Pittsburgh, Allegheny County, 15232, United States",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Chatham University, North Woodland Road, Squirrel Hill North, Pittsburgh, Allegheny County, 15232, United States\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/way/172206707"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/way/172206707"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Chatham University, Fifth Avenue, Larimer, Pittsburgh, Allegheny County, 15208, United States",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Chatham University, Fifth Avenue, Larimer, Pittsburgh, Allegheny County, 15208, United States\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/way/109563490"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/way/109563490"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |