trace_synthesis/summary/381_prompt_debug.txt
yuyr a84d51a101 1. 增加r1生成综合策略代码和输出;
2. 增加tasks;
3. 增加analysis部分,对策略进行归纳分类,然后进行评测。
2025-04-17 17:40:15 +08:00

141 lines
6.0 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Instruction
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
whic includes action and url before and after the action.
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
- For each strategy, give a clear list of the low level action sequence.
# Task
Find the hotel around Carnegie Music Hall
# Annotation description
### Step-by-Step Actions:
1. **Action:** I click on the search bar labeled "Search" at the top left corner of the page.
- **Page Changes:** The search bar becomes active, allowing text input.
- **Possible Purpose:** The likely intent is to enter a search query to find specific locations or information on the map.
2. **Action:** I type "hotels near Carnegie M" into the search bar.
- **Page Changes:** As I type, there are no immediate changes to the page, but the text appears in the search bar.
- **Possible Purpose:** The purpose is to search for hotels located near a place associated with "Carnegie M," possibly Carnegie Mellon University or Carnegie Museum.
3. **Action:** I click on the "Go" button next to the search bar.
- **Page Changes:** A dropdown list appears below the search bar, displaying search results related to "hotels near Carnegie M."
- **Possible Purpose:** The intent is to execute the search query and view the results for the specified location.
4. **Action:** I hover over the search results dropdown.
- **Page Changes:** The dropdown remains visible, listing various hotel options near the searched location.
- **Possible Purpose:** The purpose is to review the search results and potentially select one of the listed options for more detailed information.
5. **Action:** I click on one of the search result entries in the dropdown list.
- **Page Changes:** The map updates to show the selected location, and additional details about the chosen hotel may appear.
- **Possible Purpose:** The intent is to focus on a specific hotel from the search results, likely to view its exact location on the map and any other relevant details.
### Summary:
In this video segment, I interact with the OpenStreetMap website by using the search functionality to find hotels near a location associated with "Carnegie M." I activate the search bar, enter the query, execute the search, and then select a result from the dropdown list. Each step is aimed at narrowing down and viewing specific location details on the map.
# Playwright action
[
{
"action_uid": "textbox_Search",
"idx": 0,
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search\")",
"before": {
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
},
"after": {
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
}
},
{
"action_uid": "button_Go",
"idx": 1,
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
"before": {
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
},
"after": {
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
}
}
]
# Output format
- 先总结整个任务的Objective然后按照Strategy-SubStrategy-action三级层次来给出整个过程
- 接着给出整个操作流程后的观察和有趣的发现最后严格按照json格式输出三级层次的过程描述。
- 最后的输出json应该是包在```{json}```之间最底层动作需要包含描述、对应的playwright动作指令顺序编号以及具体指令内容。
# Example
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
---
#### **Strategy 1: Navigate to the Repository**
**Low-Level Action Sequence:**
1. **Search for the user "kkroening"**
- Click the global search bar (placeholder: "Search GitLab").
- Type "kkroening" and press `Enter`.
2. **Select the user from results**
- Click the "Users" tab in search results.
- Click on "Karl Kroening @kkroening" in the user list.
3. **Access the repository**
- Navigate to the "Personal projects" section.
- Click on the "ffmpeg-python" project.
---
#### **Strategy 2: Filter Issues by Label**
**Low-Level Action Sequence:**
1. **Open the Issues tab**
- Scroll to the left sidebar menu.
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
2. **Apply label filtering**
- Click the search/filter bar in the issues list.
- Select the "Label" dropdown from the filter options.
- Type or select "question" from the label dropdown.
- Click the search/apply button to confirm the filter.
---
#### **Final Oberservation**
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
`.../ffmpeg-python/-/issues/?label_name[]=question`.
---
### Key Observations from Playwright Trace
- The final URL after filtering:
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
confirms the "question" label filter is applied.
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
### Final output
```json
[{
"strategy" : "Navigate to the Repository",
"substrategies": [
{
"substrategy": "Search for the user \"kkroening\"",
"actions" : [
{
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
"playwright_idx" : 18,
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
}
]
},
{
"substrategy": "Select the user from results",
"actions" : [
]
}
]
},
{
"strategy" : "Filter Issues by Label",
"substrategies" : [
]
}]
```