325 lines
16 KiB
Plaintext
325 lines
16 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Post "Good idea" for the merge request related to color ulitity in a11yproject.com project
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a view of the GitLab Projects page, specifically under the "Personal" section.
|
||
- **Page Changes:** The page displays a list of projects with details such as project names, descriptions, star counts, and last updated dates.
|
||
- **Possible Purpose:** The initial state sets the context for navigating through personal projects on GitLab.
|
||
|
||
#### 2. **Hovering Over Project Names**
|
||
- **Action:** I hover over the project named "Byte Blaze / a11y-syntax-highlighting."
|
||
- **Page Changes:** No significant changes occur; the project name is highlighted as part of the hover effect.
|
||
- **Possible Purpose:** Hovering might be to identify or prepare to select this specific project for further actions.
|
||
|
||
#### 3. **Selecting a Project**
|
||
- **Action:** I click on the project named "The A11Y Project / a11yproject.com."
|
||
- **Page Changes:** The page transitions to the detailed view of the selected project, showing project information such as commit history, branches, tags, and merge requests.
|
||
- **Possible Purpose:** The intent is to access the detailed information and settings of the "a11yproject.com" project.
|
||
|
||
#### 4. **Navigating Project Details**
|
||
- **Action:** On the project's main page, I scroll down slightly to view more details.
|
||
- **Page Changes:** Additional sections of the project page become visible, including recent activity, files, and project statistics.
|
||
- **Possible Purpose:** Scrolling allows for a comprehensive view of the project's current status and recent updates.
|
||
|
||
#### 5. **Accessing Merge Requests**
|
||
- **Action:** I click on the "Merge requests" link in the left sidebar menu.
|
||
- **Page Changes:** The page navigates to the "Merge requests" section, displaying a list of open, merged, and closed merge requests.
|
||
- **Possible Purpose:** The goal is to review or manage merge requests associated with the project.
|
||
|
||
#### 6. **Inspecting Merge Requests**
|
||
- **Action:** I scroll through the list of merge requests.
|
||
- **Page Changes:** Different merge requests become visible, showing their titles, authors, and statuses.
|
||
- **Possible Purpose:** Scrolling helps in reviewing the list to find specific merge requests or assess their overall status.
|
||
|
||
### Summary
|
||
In this video segment, I navigate from the GitLab Projects page to a specific project ("a11yproject.com") and then to its "Merge requests" section. The actions include hovering over project names, selecting a project, viewing project details, and inspecting merge requests. Each step is performed to gather information and manage aspects of the selected GitLab project.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Step 1: Initial State
|
||
- **Action**: I begin by observing the current state of the webpage, which is the "Merge requests" section of a GitLab project.
|
||
- **Page Changes**: The page displays a list of open merge requests with details such as titles, authors, creation dates, and update statuses.
|
||
- **Possible Purpose**: To familiarize myself with the current context and available options before performing any actions.
|
||
|
||
#### Step 2: Searching for Specific Merge Requests
|
||
- **Action**: I click on the search bar located at the top of the merge requests list.
|
||
- **Page Changes**: The search bar becomes active, allowing me to input text.
|
||
- **Possible Purpose**: To filter the list of merge requests based on specific criteria for easier navigation.
|
||
|
||
#### Step 3: Entering Search Query
|
||
- **Action**: I type "color utility" into the search bar.
|
||
- **Page Changes**: As I type, the page dynamically updates to show merge requests that match the search query "color utility."
|
||
- **Possible Purpose**: To locate merge requests related to "color utility" quickly.
|
||
|
||
#### Step 4: Reviewing Search Results
|
||
- **Action**: I review the filtered list of merge requests that appear after entering the search query.
|
||
- **Page Changes**: The list now shows only the merge requests containing "color utility" in their titles or descriptions.
|
||
- **Possible Purpose**: To identify the specific merge request I am interested in from the filtered results.
|
||
|
||
#### Step 5: Selecting a Merge Request
|
||
- **Action**: I click on the merge request titled "Add color utility classes."
|
||
- **Page Changes**: The page transitions to the detailed view of the selected merge request, displaying information such as the overview, commits, pipelines, and changes.
|
||
- **Possible Purpose**: To access more detailed information about the "Add color utility classes" merge request.
|
||
|
||
#### Step 6: Observing Merge Request Details
|
||
- **Action**: I scroll through the detailed view of the merge request to observe various sections like the overview, activity, and comments.
|
||
- **Page Changes**: The page scrolls to reveal different sections of the merge request, including the description, commit history, and any discussions or comments.
|
||
- **Possible Purpose**: To gather comprehensive information about the merge request, including its status, changes made, and any associated discussions.
|
||
|
||
### Summary
|
||
In this video segment, I performed a series of actions to locate and review a specific merge request titled "Add color utility classes" within a GitLab project. I began by searching for relevant merge requests using the search bar, then selected the desired merge request from the filtered results. Finally, I reviewed the detailed information and sections of the selected merge request to understand its contents and status. Each action was aimed at efficiently navigating and gathering information about the targeted merge request.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the text area under the "Activity" section where it says "Write a comment or drag your files here..."
|
||
- **Page Changes:** The text area becomes active, and a blinking cursor appears inside it, indicating that I can now type.
|
||
- **Possible Purpose:** The likely intent is to add a comment or note related to the merge request, possibly to communicate with other team members or document changes.
|
||
|
||
2. **Action:** I begin typing in the active text area.
|
||
- **Page Changes:** As I type, the text appears in the text area. Specifically, I type `@byteblaze ju`.
|
||
- **Possible Purpose:** The purpose of typing this text is to mention a specific user (@byteblaze) and possibly start a message or command. This could be to draw their attention to the comment or to use a quick action feature.
|
||
|
||
3. **Action:** I continue typing by adding the text `nowZ` after `ju`.
|
||
- **Page Changes:** The text in the text area now reads `@byteblaze junowZ`.
|
||
- **Possible Purpose:** The continued typing might be to complete a thought, command, or message. The addition of `nowZ` could be part of a specific instruction or note.
|
||
|
||
4. **Action:** I press the spacebar after `nowZ`.
|
||
- **Page Changes:** A space is added after `nowZ`, and the text now reads `@byteblaze junowZ `.
|
||
- **Possible Purpose:** Adding a space typically indicates the end of a word or the preparation to start a new word or command.
|
||
|
||
5. **Action:** I type the phrase `good idea` after the space.
|
||
- **Page Changes:** The text in the text area now reads `@byteblaze junowZ good idea`.
|
||
- **Possible Purpose:** The phrase `good idea` suggests that I am expressing agreement or approval regarding something mentioned earlier, possibly in the context of the merge request.
|
||
|
||
6. **Action:** I move the cursor to the "Comment" button below the text area.
|
||
- **Page Changes:** The "Comment" button is highlighted, indicating it is ready to be clicked.
|
||
- **Possible Purpose:** The intention is to submit the typed comment so that it is posted in the activity section of the merge request.
|
||
|
||
7. **Action:** I hover over the text area again, possibly to make an edit or review the comment before submitting.
|
||
- **Page Changes:** No immediate changes occur; the text remains as `@byteblaze junowZ good idea`.
|
||
- **Possible Purpose:** This action suggests a final review of the comment to ensure it is correct and conveys the intended message before submission.
|
||
|
||
### Summary:
|
||
In this video segment, I interact with the "Activity" section of a GitLab merge request page. I activate the comment text area, type a message that includes a user mention and a supportive phrase, and prepare to submit the comment. The actions suggest a focus on communication within the context of the merge request, likely to provide feedback or instructions to another user.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_The A11Y Project\\n/ a11yproject.com",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"The A11Y Project\\n/ a11yproject.com\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Merge requests",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_label(\"Project navigation\").get_by_role(\"link\", name=\"Merge requests\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_2",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search or filter results...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_3",
|
||
"idx": 3,
|
||
"action_repr": "frame.pressget_by_placeholder(\"Search or filter results...\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"#filtered-search-merge_requests\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"#content-body div\").filter(has_text=\"Select all Recent searches You don't have any recent searches Author Assignee Re\").nth(1)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clicklocator(\"#filtered-search-merge_requests\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.presslocator(\"#filtered-search-merge_requests\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests?scope=all&state=opened&search=color+utility"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests?scope=all&state=opened&search=color+utility"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Add color utility classes",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Add color utility classes\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests/1071"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests?scope=all&state=opened&search=color+utility"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "comment-field",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_test_id(\"comment-field\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests/1071"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests/1071"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "comment-button",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_test_id(\"comment-button\").get_by_role(\"button\", name=\"Comment\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests/1071"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/merge_requests/1071"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |