274 lines
11 KiB
Plaintext
274 lines
11 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Post my question, "is car necessary in NYC", in a subreddit where I'm likely to get an answer
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Navigating to the List of Forums
|
||
|
||
#### Action:
|
||
- I click on the "Forums" link in the navigation bar at the top of the webpage.
|
||
|
||
#### Page Changes:
|
||
- The page transitions from displaying a list of submissions and comments to showing an alphabetical list of forums. Each forum is listed with its name, description, and subscriber count.
|
||
|
||
#### Possible Purpose:
|
||
- The likely intent is to explore different forums available on the platform. This action suggests a desire to either find a specific forum or browse through various community discussions.
|
||
|
||
---
|
||
|
||
### Part 2: Scrolling Through the List of Forums
|
||
|
||
#### Action:
|
||
- I scroll down the page to view more forums in the list.
|
||
|
||
#### Page Changes:
|
||
- As I scroll, additional forums become visible, expanding the range of topics and communities displayed. The forums are organized alphabetically, making it easier to locate specific ones.
|
||
|
||
#### Possible Purpose:
|
||
- The purpose of this action is to continue exploring the available forums. Scrolling allows for a broader overview of the community options, potentially to find a forum of interest or relevance.
|
||
|
||
---
|
||
|
||
### Part 3: Selecting a Forum
|
||
|
||
#### Action:
|
||
- I click on the "nyc" forum link from the list.
|
||
|
||
#### Page Changes:
|
||
- The page transitions to display the contents of the "nyc" forum. It shows a list of recent submissions related to New York City, along with options to subscribe, hide the forum, and access moderation tools.
|
||
|
||
#### Possible Purpose:
|
||
- The intent here is to enter the "nyc" forum to view and possibly engage with discussions specific to New York City. This action indicates an interest in content related to this particular topic or location.
|
||
|
||
---
|
||
|
||
### Part 4: Browsing Submissions in the "nyc" Forum
|
||
|
||
#### Action:
|
||
- I scroll through the list of submissions in the "nyc" forum.
|
||
|
||
#### Page Changes:
|
||
- As I scroll, more submissions appear, each with details such as the title, author, date, and number of comments. The content varies, covering topics like events, personal experiences, and city-related discussions.
|
||
|
||
#### Possible Purpose:
|
||
- The purpose of this action is to review the recent activity and discussions within the "nyc" forum. Scrolling helps in assessing the type of content shared by the community members and identifying any posts of particular interest.
|
||
|
||
---
|
||
|
||
### Summary of Actions:
|
||
1. **Navigated to the Forums List**: Clicked the "Forums" link to explore available community options.
|
||
2. **Scrolled Through Forums**: Continued browsing to see a wider range of forum topics.
|
||
3. **Selected the "nyc" Forum**: Clicked on the "nyc" forum link to focus on New York City-related discussions.
|
||
4. **Browsed Submissions**: Scrolled through the "nyc" forum to review recent posts and community activity.
|
||
|
||
Each step reflects a deliberate exploration of the platform's features, aiming to find and engage with relevant content.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click inside the "Title" text box.
|
||
- **Page Changes**: The cursor appears in the "Title" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose**: My intention is to enter a title for the submission.
|
||
|
||
2. **Action**: I type "Is car necessary in NYC" into the "Title" text box.
|
||
- **Page Changes**: The text "Is car necessary in NYC" appears in the "Title" text box.
|
||
- **Possible Purpose**: I am providing a specific title for the submission, likely to frame the topic or question I want to discuss.
|
||
|
||
3. **Action**: I move the cursor to the "Body" text box area but do not click or type anything.
|
||
- **Page Changes**: There are no changes to the webpage as no action (click or type) is taken in the "Body" text box.
|
||
- **Possible Purpose**: I might be considering what content to add to the body of the submission but have not yet decided or started typing.
|
||
|
||
4. **Action**: I click on the "Forum" dropdown menu.
|
||
- **Page Changes**: The dropdown menu expands, showing available forum options.
|
||
- **Possible Purpose**: I am selecting the appropriate forum for my submission to ensure it is posted in the correct category.
|
||
|
||
5. **Action**: I select "nyc" from the dropdown menu.
|
||
- **Page Changes**: The "Forum" field updates to display "nyc," confirming the selection.
|
||
- **Possible Purpose**: I am specifying that this submission should be posted in the "nyc" forum, aligning with the topic of the title.
|
||
|
||
6. **Action**: I hover over the "Create submission" button but do not click it.
|
||
- **Page Changes**: There are no changes to the webpage as the button is not clicked.
|
||
- **Possible Purpose**: I might be reviewing the information I have entered before finalizing and submitting the post.
|
||
|
||
These actions collectively show the process of preparing a submission titled "Is car necessary in NYC" for the "nyc" forum, with careful attention to the title and forum selection. No further actions, such as typing in the body or final submission, are observed in this segment.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Alphabetical",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Alphabetical\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_nyc",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"nyc\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums/all"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums/all"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Submit",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_label(\"Submit\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"#submission_url\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title *",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_label(\"Title *\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Body",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_label(\"Body\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create submission",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create submission\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/nyc"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |