233 lines
12 KiB
Plaintext
233 lines
12 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Open my latest updated issue that has keyword "better" in its title to check if it is closed
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Issues" tab located at the top of the page.
|
||
- **Page Changes:** The page transitions from the "Projects" view to the "Issues" view, displaying a list of open issues associated with the selected projects.
|
||
- **Possible Purpose:** The likely intent is to navigate to the section where project issues are managed and reviewed.
|
||
|
||
#### 2. **Action:** I use the search bar by typing "Byte Blaze" into the "Assignee" filter field.
|
||
- **Page Changes:** The list of issues updates to show only those assigned to "Byte Blaze."
|
||
- **Possible Purpose:** The purpose is to filter the issues specifically to those assigned to the user or entity named "Byte Blaze," allowing for focused review or management of their tasks.
|
||
|
||
#### 3. **Action:** I type the word "better" into the main search bar within the Issues page.
|
||
- **Page Changes:** The issue list further narrows down to include only those issues whose titles or descriptions contain the word "better."
|
||
- **Possible Purpose:** This action aims to refine the search results even more specifically, possibly to find issues related to improvements or enhancements suggested by the term "better."
|
||
|
||
#### 4. **Action:** I hover over the "Created date" column header.
|
||
- **Page Changes:** No immediate change occurs, but this action might be preparatory for sorting.
|
||
- **Possible Purpose:** The intent could be to sort the issues based on their creation date, either to prioritize newer or older issues.
|
||
|
||
#### 5. **Action:** I click on the "Created date" column header.
|
||
- **Page Changes:** The issues list is sorted by their creation date, either in ascending or descending order.
|
||
- **Possible Purpose:** Sorting by creation date helps in organizing the issues temporally, which can be useful for tracking the timeline of reported issues or focusing on the most recent ones.
|
||
|
||
#### 6. **Action:** I click on the "Updated date" column header.
|
||
- **Page Changes:** The issues list is now sorted by their last updated date.
|
||
- **Possible Purpose:** Sorting by the updated date allows me to focus on the most recently active issues, which might be the ones currently being worked on or have had recent developments.
|
||
|
||
### Summary
|
||
In this video segment, I navigate from the "Projects" page to the "Issues" page, filter issues by assignee and specific keywords, and sort the issues by their creation and update dates. These actions suggest a systematic approach to managing and reviewing specific issues assigned to "Byte Blaze," with a focus on finding relevant, recent, or updated tasks.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Provided Video Segment
|
||
|
||
#### 1. **Action**: I click on the "Edit" button located next to the "Assignee" section.
|
||
- **Page Changes**: The page transitions into an editing mode for the assignee section, allowing me to select or change the assignee.
|
||
- **Possible Purpose**: The likely intent is to modify or confirm the assignee for this issue, ensuring it is correctly assigned to the responsible individual.
|
||
|
||
#### 2. **Action**: I click on the "Eric Bailey" option from the dropdown menu that appears after clicking "Edit."
|
||
- **Page Changes**: The assignee is updated to "Eric Bailey," and the editing interface reverts to its standard display mode.
|
||
- **Possible Purpose**: The purpose is to assign the issue to Eric Bailey, presumably because he is the appropriate person to handle or oversee this task.
|
||
|
||
#### 3. **Action**: I click on the "Edit" button next to the "Labels" section.
|
||
- **Page Changes**: The labels section enters an editable state, displaying a list of available labels that can be added or removed.
|
||
- **Possible Purpose**: The intention is to modify the labels associated with this issue, which helps in categorizing and prioritizing the task effectively.
|
||
|
||
#### 4. **Action**: I click on the "Type: Enhancement" label within the editable labels section.
|
||
- **Page Changes**: The "Type: Enhancement" label is highlighted or confirmed, indicating it is selected or deselected (depending on its previous state).
|
||
- **Possible Purpose**: The goal is to ensure that the "Type: Enhancement" label is appropriately applied to this issue, aiding in its classification and management.
|
||
|
||
#### 5. **Action**: I click on the "Create merge request" button located near the top-right of the issue details.
|
||
- **Page Changes**: A new window or overlay appears, prompting me to create a merge request related to this issue.
|
||
- **Possible Purpose**: The aim is to initiate the process of creating a merge request, which is typically done to propose changes that address the issue at hand.
|
||
|
||
#### 6. **Action**: I type the text ".get_by_test_id('by-labels').get_by_test_id('collapsed')" into the search bar at the top of the page.
|
||
- **Page Changes**: The search bar reflects the entered text, but no immediate page changes occur as a result of this action (search results are not shown in this segment).
|
||
- **Possible Purpose**: The likely intent is to perform a search using a specific query, possibly to locate related issues, code snippets, or documentation that includes this exact phrase.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions focused on managing an issue within a GitLab project. These actions include assigning the issue to a specific user, editing and confirming labels, initiating a merge request, and entering a specific search query. Each step appears to be part of a workflow aimed at organizing, addressing, and potentially resolving the issue efficiently.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_label(\"Issues\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.clicklocator(\"#filtered-search-issues\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_2",
|
||
"idx": 2,
|
||
"action_repr": "frame.presslocator(\"#filtered-search-issues\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=opened&assignee_username=byteblaze&search=better"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "base-dropdown-toggle",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_test_id(\"base-dropdown-toggle\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=opened&assignee_username=byteblaze&search=better"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=opened&assignee_username=byteblaze&search=better"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Updated date",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_text(\"Updated date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&search=better&sort=updated_desc&state=opened"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?scope=all&state=opened&assignee_username=byteblaze&search=better"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"div\").filter(has_text=\"Issues Select project to create issue Search for project Open 2 Closed 4 All 6 S\").nth(1)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&search=better&sort=updated_desc&state=opened"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&search=better&sort=updated_desc&state=opened"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Better initial load experience",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Better initial load experience\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&search=better&sort=updated_desc&state=opened"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/dashboard/issues?assignee_username=byteblaze&scope=all&search=better&sort=updated_desc&state=opened"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |