157 lines
6.8 KiB
Plaintext
157 lines
6.8 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Which US states border New Hampshire?
|
||
|
||
# Annotation description
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the OpenStreetMap website open, displaying a welcome modal dialog titled "Welcome to OpenStreetMap!".
|
||
- **Page Changes:** The main map is visible in the background, but partially obscured by the welcome modal.
|
||
- **Possible Purpose:** The initial state sets the context for interacting with the OpenStreetMap interface.
|
||
|
||
#### 2. **Closing the Welcome Modal**
|
||
- **Action:** I click the close button (an "X" icon) on the top-right corner of the welcome modal dialog.
|
||
- **Page Changes:** The welcome modal disappears, revealing the full view of the map and the search bar at the top left of the page.
|
||
- **Possible Purpose:** To clear the screen and gain full access to the map and its functionalities without the modal obstructing the view.
|
||
|
||
#### 3. **Focusing on the Search Bar**
|
||
- **Action:** I move the cursor to the search bar located at the top left corner of the page and click inside it.
|
||
- **Page Changes:** The search bar becomes active, indicated by a blinking cursor inside it, ready for text input.
|
||
- **Possible Purpose:** To prepare for entering a location or keyword to search within the map.
|
||
|
||
#### 4. **Typing in the Search Bar**
|
||
- **Action:** I type "new hampshire" into the search bar.
|
||
- **Page Changes:** As I type, a dropdown menu appears below the search bar, suggesting search results related to the entered text.
|
||
- **Possible Purpose:** To find and focus the map on the area corresponding to "New Hampshire".
|
||
|
||
#### 5. **Selecting a Search Result**
|
||
- **Action:** I use the mouse to hover over the dropdown menu and click on the first suggested result labeled "State: New Hampshire, United States".
|
||
- **Page Changes:** The map updates to center on the state of New Hampshire, and a marker or highlight may appear to indicate the selected region.
|
||
- **Possible Purpose:** To zoom and center the map display on the specific location of New Hampshire for further exploration or mapping activities.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website by first closing a welcome modal to access the full map interface. I then use the search functionality to find and focus on the state of New Hampshire, United States, by typing the name into the search bar and selecting the appropriate result from the dropdown suggestions. Each action is performed with the intent of navigating and utilizing the map features effectively.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |