289 lines
14 KiB
Plaintext
289 lines
14 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Open my latest created issue that has homepage content in its title to check if it is closed
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a view of a GitLab Projects page.
|
||
- **Page Changes:** The page displays a list of projects under "Yours," with details such as project names, descriptions, and update times.
|
||
- **Possible Purpose:** The initial state sets the context for navigating within the user's projects on GitLab.
|
||
|
||
#### 2. **Navigating to a Specific Project**
|
||
- **Action:** I click on the project named "a11y-webbing.club."
|
||
- **Page Changes:** The page transitions to the project's main dashboard, showing sections like "Project information," "Repository," "Issues," etc.
|
||
- **Possible Purpose:** The intent is to access the specific project "a11y-webbing.club" to perform actions related to this project.
|
||
|
||
#### 3. **Accessing the Issues Section**
|
||
- **Action:** I click on the "Issues" link in the left sidebar menu.
|
||
- **Page Changes:** The page updates to display a list of issues associated with the "a11y-webbing.club" project.
|
||
- **Possible Purpose:** The goal is to review or manage the issues related to this project.
|
||
|
||
#### 4. **Selecting a Specific Issue**
|
||
- **Action:** I click on the issue titled "[Feature suggestion] Support linking to an accessibility statement."
|
||
- **Page Changes:** The page navigates to the detailed view of the selected issue, showing its description, comments, and other details.
|
||
- **Possible Purpose:** The intention is to examine the specifics of this particular issue, possibly to understand its details, comment on it, or take further action.
|
||
|
||
#### 5. **Reviewing the Issue Details**
|
||
- **Action:** I scroll through the issue page to review the content.
|
||
- **Page Changes:** The page scrolls down, revealing more details about the issue, including the feature suggestion, discussion points, and any comments or tasks associated with it.
|
||
- **Possible Purpose:** The purpose is to thoroughly understand the issue's context, requirements, and any ongoing discussions or tasks.
|
||
|
||
### Summary
|
||
In this video segment, I navigate from the GitLab Projects page to a specific project ("a11y-webbing.club"), then to its Issues section, and finally to a detailed view of a particular issue. Each step involves clicking on relevant links or sections to progressively narrow down to the specific issue of interest. The actions are focused on accessing and reviewing the details of the selected issue.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a view of a GitLab project's "Issues" page. The page displays a list of issues, each with details such as title, assignee, labels, and status.
|
||
- **Page Changes:** No immediate changes occur as this is the starting point.
|
||
- **Possible Purpose:** The initial state sets the context for managing or reviewing issues within the project.
|
||
|
||
#### 2. **Filtering Issues by Assignee**
|
||
- **Action:** I click on the "Assignee" filter dropdown located at the top of the issues list.
|
||
- **Page Changes:** A dropdown menu appears, listing available options for filtering issues by assignee.
|
||
- **Possible Purpose:** The intent is to narrow down the list of issues to those assigned to a specific user, likely for focused review or management.
|
||
|
||
#### 3. **Selecting an Assignee**
|
||
- **Action:** I select an assignee from the dropdown menu. The chosen assignee is "Byte Blazea11project.contributor.me."
|
||
- **Page Changes:** The issues list updates to display only the issues assigned to the selected user.
|
||
- **Possible Purpose:** This action aims to view and manage the tasks specifically assigned to the chosen user, facilitating task oversight or personal workload management.
|
||
|
||
#### 4. **Sorting Issues by Created Date**
|
||
- **Action:** I click on the "Created date" button located near the top-right corner of the issues list.
|
||
- **Page Changes:** A sorting dropdown menu appears, offering options to sort issues by different criteria.
|
||
- **Possible Purpose:** The goal is to organize the filtered issues based on their creation date, which can help in prioritizing newer or older tasks.
|
||
|
||
#### 5. **Confirming Sorting Option**
|
||
- **Action:** I select "Created date" from the sorting dropdown menu.
|
||
- **Page Changes:** The issues list reorders, now displaying issues sorted by their creation date.
|
||
- **Possible Purpose:** This ensures that the issues are presented in a chronological order, aiding in tracking the timeline of task assignments or identifying recently added tasks.
|
||
|
||
#### 6. **Reviewing Sorted and Filtered Issues**
|
||
- **Action:** I scroll through the updated issues list.
|
||
- **Page Changes:** The screen moves vertically, showing different issues as I scroll.
|
||
- **Possible Purpose:** The intent is to review the sorted and filtered list, possibly to assess the status, prioritize tasks, or find a specific issue for further action.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to filter and sort the issues on a GitLab project page. Specifically, I filter the issues by a particular assignee and then sort them by their creation date. These actions result in a more organized and manageable list, which I then review by scrolling through the updated display. The purpose appears to be efficient task management and oversight within the project.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Action 1:
|
||
- **Action:** I click on the "Assignee" filter dropdown in the issues list.
|
||
- **Page Changes:** The dropdown menu appears, listing available assignees.
|
||
- **Possible Purpose:** To filter the issues by a specific assignee, likely to focus on tasks assigned to a particular person.
|
||
|
||
#### Action 2:
|
||
- **Action:** I select "Byte Blazea11yproject.contributor.me" from the assignee dropdown.
|
||
- **Page Changes:** The issues list updates to show only the issues assigned to "Byte Blazea11yproject.contributor.me."
|
||
- **Possible Purpose:** To view and manage the issues specifically assigned to this user.
|
||
|
||
#### Action 3:
|
||
- **Action:** I type "homepage content" into the search bar at the top of the issues page.
|
||
- **Page Changes:** The search results update to display issues that contain the text "homepage content."
|
||
- **Possible Purpose:** To find specific issues related to the homepage content for further review or action.
|
||
|
||
#### Action 4:
|
||
- **Action:** I hover over an issue titled "Rethink the homepage’s content."
|
||
- **Page Changes:** A tooltip or additional information about the issue may appear (though not explicitly shown in the image).
|
||
- **Possible Purpose:** To gather more information about the issue before selecting it.
|
||
|
||
#### Action 5:
|
||
- **Action:** I click on the issue titled "Rethink the homepage’s content."
|
||
- **Page Changes:** The page navigates to the detailed view of the selected issue.
|
||
- **Possible Purpose:** To view the full details of the issue, including comments, status, and other relevant information.
|
||
|
||
#### Action 6:
|
||
- **Action:** I scroll down slightly on the issue detail page.
|
||
- **Page Changes:** Additional content of the issue becomes visible, such as comments or task lists.
|
||
- **Possible Purpose:** To review more details or comments related to the issue.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to filter and locate a specific issue related to "homepage content." I begin by filtering issues based on an assignee, then use the search functionality to narrow down the results. After identifying the relevant issue, I navigate to its detailed view to review further information. Each step is aimed at efficiently locating and examining the specific issue for potential action or analysis.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_label(\"Issues\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_[Feature suggestion] Support linking to an accessibility statement",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"[Feature suggestion] Support linking to an accessibility statement\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_All 66",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"All 66\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "base-dropdown-toggle",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_test_id(\"base-dropdown-toggle\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"group\").get_by_role(\"link\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&sort=created_asc&state=all"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&sort=created_asc&state=all"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_8",
|
||
"idx": 8,
|
||
"action_repr": "frame.clicklocator(\"#filtered-search-issues\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&sort=created_asc&state=all"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&sort=created_asc&state=all"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_9",
|
||
"idx": 9,
|
||
"action_repr": "frame.presslocator(\"#filtered-search-issues\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze&search=homepage+content"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze&search=homepage+content"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Rethink the homepage's content",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Rethink the homepage's content\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze&search=homepage+content"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=all&assignee_username=byteblaze&search=homepage+content"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |