380 lines
17 KiB
Plaintext
380 lines
17 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Re-post the image of costume contest in this page to funny subreddit and note "from /f/pics"
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Create submission" button.
|
||
- **Page Changes:** The page transitions from the main forum view to a new form titled "Create submission."
|
||
- **Possible Purpose:** The intent is to start the process of submitting a new post or thread within the forum.
|
||
|
||
2. **Action:** I click inside the "URL" text box.
|
||
- **Page Changes:** The cursor focuses on the "URL" text box, ready for input.
|
||
- **Possible Purpose:** The intent is to enter a web address, likely to link an external resource or image to the submission.
|
||
|
||
3. **Action:** I type a URL into the "URL" text box.
|
||
- **Page Changes:** The URL appears in the text box as it is typed.
|
||
- **Possible Purpose:** The intent is to provide a specific web link that will be associated with the submission, possibly for reference or content display.
|
||
|
||
4. **Action:** I click inside the "Title" text box.
|
||
- **Page Changes:** The cursor moves to the "Title" text box, indicating readiness for text entry.
|
||
- **Possible Purpose:** The intent is to enter a title for the submission, which will serve as the heading or summary of the post.
|
||
|
||
5. **Action:** I type text into the "Title" text box.
|
||
- **Page Changes:** The text appears in the "Title" text box as it is typed.
|
||
- **Possible Purpose:** The intent is to provide a descriptive or attention-grabbing title for the submission to inform other users about its content.
|
||
|
||
6. **Action:** I click inside the "Body" text area.
|
||
- **Page Changes:** The cursor moves to the "Body" text area, preparing for additional text input.
|
||
- **Possible Purpose:** The intent is to add more detailed information or context to the submission beyond just the title and URL.
|
||
|
||
7. **Action:** I begin typing in the "Body" text area.
|
||
- **Page Changes:** The text starts appearing in the "Body" text area as it is typed.
|
||
- **Possible Purpose:** The intent is to provide supplementary information, elaboration, or personal commentary related to the submission.
|
||
|
||
8. **Action:** I click on the "Forum" dropdown menu.
|
||
- **Page Changes:** A list of forum options becomes visible, allowing selection of a specific forum.
|
||
- **Possible Purpose:** The intent is to choose the appropriate forum category for the submission to ensure it reaches the relevant audience.
|
||
|
||
9. **Action:** I scroll through the list of forum options.
|
||
- **Page Changes:** The list of forums scrolls, revealing more options.
|
||
- **Possible Purpose:** The intent is to find and select the most suitable forum for the submission based on its content or theme.
|
||
|
||
10. **Action:** I click on a specific forum option from the list.
|
||
- **Page Changes:** The selected forum option is highlighted, confirming the choice.
|
||
- **Possible Purpose:** The intent is to finalize the forum selection, ensuring the submission is posted in the correct category.
|
||
|
||
11. **Action:** I click the "Create submission" button.
|
||
- **Page Changes:** The form is submitted, and the page likely transitions to a confirmation or the newly created submission view.
|
||
- **Possible Purpose:** The intent is to complete the submission process and publish the post within the chosen forum.
|
||
|
||
This sequence strictly follows the observable actions in the provided video segment without incorporating any additional context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Video Segment Description
|
||
|
||
#### Initial State:
|
||
The video begins with a webpage open to a "Create submission" form on a forum-like platform. The page includes fields for URL, Title, Body, and Forum selection. On the right side, there are sections labeled "pics," "Hide this forum," and "Toolbox."
|
||
|
||
---
|
||
|
||
### Action 1:
|
||
- **Action:** I click inside the "Body" text box.
|
||
- **Page Changes:** The cursor appears in the "Body" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to start typing the main content or description for the submission.
|
||
|
||
---
|
||
|
||
### Action 2:
|
||
- **Action:** I type the letter "l" into the "Body" text box.
|
||
- **Page Changes:** The letter "l" appears in the "Body" text box.
|
||
- **Possible Purpose:** This action suggests the beginning of writing the submission content. The specific purpose of typing "l" could be the start of a word or phrase relevant to the submission.
|
||
|
||
---
|
||
|
||
### Action 3:
|
||
- **Action:** I move the cursor outside the "Body" text box.
|
||
- **Page Changes:** The cursor is no longer in the "Body" text box, but the letter "l" remains typed.
|
||
- **Possible Purpose:** Moving the cursor might be to review the current input, prepare to interact with another element on the page, or pause before continuing to type.
|
||
|
||
---
|
||
|
||
### Final State:
|
||
The video segment ends with the "Body" text box containing the letter "l," and the cursor positioned outside the text box. No other actions are observed in this segment.
|
||
|
||
---
|
||
|
||
### Summary:
|
||
In this video segment, the primary actions involve interacting with the "Body" text box of a submission form. Specifically, I activate the text box, type the letter "l," and then move the cursor away. These actions suggest the initial steps of composing a submission post on the platform. No other elements on the page are interacted with during this segment.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the webpage open on the "Forums" section of a site called Postmill. The user is logged in as "MarvelsGrantMan136," as indicated by the username in the top-right corner.
|
||
- **Page Changes:** The page displays a list of forums with various topics and recent posts. The right sidebar shows the user's profile options and statistics, such as the number of submissions and hidden forums.
|
||
- **Possible Purpose:** The initial state sets the context for the user's interaction with the forums, suggesting they are about to engage with or manage their forum activity.
|
||
|
||
#### 2. **Scrolling Through the Forums List**
|
||
- **Action:** I scroll down the list of forums.
|
||
- **Page Changes:** As I scroll, different forum topics come into view, including ones like "AskReddit," "relationship_advice," "worldnews," "Nvidia RTX 4090," "news," "movies," and "memes."
|
||
- **Possible Purpose:** The purpose of scrolling is likely to browse through the available forums to find a specific topic of interest or to review recent activity across different forums.
|
||
|
||
#### 3. **Hovering Over the " relationship_advice" Forum**
|
||
- **Action:** I hover over the "relationship_advice" forum entry.
|
||
- **Page Changes:** No significant changes occur on the page, but the hovering action highlights the forum entry, indicating it is clickable.
|
||
- **Possible Purpose:** Hovering suggests an intention to either read more about this forum or to enter it. It could be a preliminary step before clicking to access the forum's content.
|
||
|
||
#### 4. **Clicking on the " relationship_advice" Forum**
|
||
- **Action:** I click on the "relationship_advice" forum entry.
|
||
- **Page Changes:** The page transitions to the "relationship_advice" forum, displaying the latest posts and discussions related to relationship advice.
|
||
- **Possible Purpose:** The click action indicates a clear intent to engage with the content within the "relationship_advice" forum, possibly to read discussions, participate in conversations, or find specific advice.
|
||
|
||
#### 5. **Reviewing the "relationship_advice" Forum Content**
|
||
- **Action:** After clicking, I review the content displayed in the "relationship_advice" forum.
|
||
- **Page Changes:** The forum page shows a list of posts with titles, timestamps, and the number of comments. Each post is associated with a brief description or excerpt.
|
||
- **Possible Purpose:** Reviewing the forum content allows me to understand the current discussions, identify relevant threads, or decide whether to contribute to any of the ongoing conversations.
|
||
|
||
### Summary
|
||
In this video segment, I begin by browsing through a list of forums, specifically focusing on the "relationship_advice" forum. I scroll to locate it, hover to highlight it, and then click to access its content. My actions suggest an intent to engage with discussions related to relationship advice, either by reading existing posts or potentially contributing to the conversation. The sequence of actions is methodical, moving from general browsing to specific engagement with a chosen forum.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_We won first place in the costume contest!",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"We won first place in the costume contest!\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.clicklocator(\"li\").filter(has_text=\"134 comments\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submission_images/bd8bc5f4c846aac4df08626faa3a34a7d47c8f3bdd92bf615a54afd939f063a7.jpg"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submission_images/bd8bc5f4c846aac4df08626faa3a34a7d47c8f3bdd92bf615a54afd939f063a7.jpg"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Submit",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_label(\"Submit\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_pics",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_label(\"pics\").get_by_text(\"pics\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_funny",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"#select2-submission_forum-result-kte3-10046\").get_by_text(\"funny\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"#submission_url\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title *",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_label(\"Title *\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Body",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"Body\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create submission",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create submission\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Go to home page",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Go to home page\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submit/pics"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_MarvelsGrantMan136",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"MarvelsGrantMan136\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Profile",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Profile\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_From /f/pics",
|
||
"idx": 13,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"From /f/pics\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submission_images/bd8bc5f4c846aac4df08626faa3a34a7d47c8f3bdd92bf615a54afd939f063a7.jpg"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/submission_images/bd8bc5f4c846aac4df08626faa3a34a7d47c8f3bdd92bf615a54afd939f063a7.jpg"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_14",
|
||
"idx": 14,
|
||
"action_repr": "frame.clicklocator(\".submission__nav > .unlistify > li > a\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/funny/5/from-f-pics"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/funny/5/from-f-pics"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |