180 lines
9.4 KiB
Plaintext
180 lines
9.4 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Checkout merge requests requiring my review
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Observing Actions in the Provided Video Segment
|
||
|
||
#### Step 1: Initial State
|
||
- **Action**: The video begins with a view of the "Projects" page on GitLab.
|
||
- **Page Changes**: The page displays a list of projects under "Yours," with details such as project names, descriptions, and update statuses.
|
||
- **Possible Purpose**: The initial state sets the context for navigating through personal projects on GitLab.
|
||
|
||
#### Step 2: Hovering Over Project Details
|
||
- **Action**: I hover over a specific project titled "Byte Blaze / a11y-syntax-highlighting."
|
||
- **Page Changes**: No significant changes occur; the project details remain visible.
|
||
- **Possible Purpose**: This action might be to inspect the project details more closely or prepare for further interaction with this project.
|
||
|
||
#### Step 3: Clicking on a Project
|
||
- **Action**: I click on the project titled "Byte Blaze / a11y-syntax-highlighting."
|
||
- **Page Changes**: The page transitions to the project's main dashboard, displaying various sections like activity, repository, issues, and merge requests.
|
||
- **Possible Purpose**: The intent is to access the detailed dashboard of the selected project for further actions or information.
|
||
|
||
#### Step 4: Navigating to Merge Requests
|
||
- **Action**: I click on the "Merge requests" tab within the project dashboard.
|
||
- **Page Changes**: The page updates to show a list of merge requests associated with the project, including their titles, statuses, and additional details.
|
||
- **Possible Purpose**: The goal is to review or manage merge requests related to the project.
|
||
|
||
#### Step 5: Hovering Over Merge Request Details
|
||
- **Action**: I hover over a specific merge request titled "WIP - Post: Pitfalls of accessible components."
|
||
- **Page Changes**: A tooltip or additional details about the merge request may appear, though no significant change is noted in the provided frames.
|
||
- **Possible Purpose**: This action could be to gather more information about the merge request before taking further action.
|
||
|
||
#### Step 6: Selecting a Merge Request
|
||
- **Action**: I click on the merge request titled "WIP - Post: Pitfalls of accessible components."
|
||
- **Page Changes**: The page transitions to the detailed view of the selected merge request, showing its description, comments, and other relevant information.
|
||
- **Possible Purpose**: The intention is to review the specifics of this merge request, possibly to provide feedback, approve changes, or track its progress.
|
||
|
||
### Summary of Observed Actions
|
||
In this video segment, I start by viewing the "Projects" page on GitLab, then navigate to a specific project titled "Byte Blaze / a11y-syntax-highlighting." From there, I access the "Merge requests" tab and select a particular merge request for detailed review. Each step is methodical, focusing on accessing and examining specific elements within the GitLab interface. The actions suggest a workflow aimed at project management and code review within the GitLab platform.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar at the top of the page.
|
||
- **Page Changes:** The cursor is now active inside the search bar, and a dropdown menu appears below it, suggesting recent searches or search options.
|
||
- **Possible Purpose:** The likely intent is to initiate a search within the GitLab interface, possibly to find specific merge requests, issues, or other elements related to the project.
|
||
|
||
#### 2. **Action:** I type `locator(".sidebar-collapse").first` into the search bar.
|
||
- **Page Changes:** As I type, the text appears in the search bar. However, no immediate search results are displayed until the search is executed.
|
||
- **Possible Purpose:** The typed text appears to be a code snippet or a query related to locating an element, possibly for automation or testing purposes. The intent might be to search for this specific phrase within the repository or project documentation.
|
||
|
||
#### 3. **Action:** I highlight the text `feature: add WCAG levels` in the list of merge requests.
|
||
- **Page Changes:** The text `feature: add WCAG levels` is visually highlighted, indicating selection.
|
||
- **Possible Purpose:** Highlighting this specific merge request could be for the purpose of reviewing its details, copying the text, or preparing to perform an action related to this particular request (e.g., opening it, commenting on it).
|
||
|
||
#### 4. **Action:** I scroll down slightly on the page.
|
||
- **Page Changes:** The view shifts downward, revealing more merge requests in the list.
|
||
- **Possible Purpose:** Scrolling allows me to view additional merge requests that were not initially visible, potentially to find a specific request or to review more items in the list.
|
||
|
||
#### 5. **Action:** I highlight the text `Fix card focus edge cases` in the list of merge requests.
|
||
- **Page Changes:** The text `Fix card focus edge cases` is now visually highlighted.
|
||
- **Possible Purpose:** Similar to the previous highlighting action, this could be for reviewing, copying, or preparing to interact with this specific merge request.
|
||
|
||
### Summary
|
||
In this video segment, my actions primarily involve interacting with the search functionality and the list of merge requests on the GitLab page. Specifically, I enter a query into the search bar, highlight two different merge requests in the list, and scroll to view more content. These actions suggest a focused effort on locating and reviewing specific merge requests, possibly for detailed examination or further interaction.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_label(\"Merge requests\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Review requests for you 5",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Review requests for you 5\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |