368 lines
20 KiB
Plaintext
368 lines
20 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Tell me the full names of the repositories where I made contributions and they got the most stars?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar at the top of the page.
|
||
- **Page Changes:** The search bar becomes active, and a blinking cursor appears inside it, indicating that it is ready for input.
|
||
- **Possible Purpose:** The likely intent is to initiate a search within the GitLab platform, possibly to find specific projects, issues, or other resources.
|
||
|
||
#### 2. **Action:** I type "Search GitLab" into the search bar.
|
||
- **Page Changes:** As I type, the text "Search GitLab" appears in the search bar. No immediate page changes occur beyond the text input.
|
||
- **Possible Purpose:** The purpose is to either test the functionality of the search bar or to prepare for an actual search query.
|
||
|
||
#### 3. **Action:** I press the Enter key while the search bar is active.
|
||
- **Page Changes:** The page transitions to a new view displaying search results related to "Search GitLab." The results include various categories such as Projects, Issues, Merge Requests, etc., that match the search term.
|
||
- **Possible Purpose:** The intent is to execute the search query and view the results that match "Search GitLab."
|
||
|
||
#### 4. **Action:** I hover over the "Projects" tab in the navigation menu.
|
||
- **Page Changes:** The "Projects" tab is highlighted, indicating that it is being hovered over. However, no further page changes occur until a selection is made.
|
||
- **Possible Purpose:** The likely intent is to navigate to the Projects section to view or manage projects.
|
||
|
||
#### 5. **Action:** I click on the "Projects" tab in the navigation menu.
|
||
- **Page Changes:** The page updates to display a list of projects. The projects are categorized under "Yours," "Starred," "Explore," and "Topics." The "Yours" tab is selected by default, showing a list of personal projects.
|
||
- **Possible Purpose:** The purpose is to access the Projects section to view or manage the projects associated with my account.
|
||
|
||
#### 6. **Action:** I scroll down the list of projects.
|
||
- **Page Changes:** As I scroll, more projects become visible on the page. The projects are listed with details such as the project name, description, owner, and last updated time.
|
||
- **Possible Purpose:** The intent is to browse through the list of projects to find a specific project or to review the available projects.
|
||
|
||
#### 7. **Action:** I click on the project named "The A11Y Project / a11yproject.com."
|
||
- **Page Changes:** The page transitions to the detailed view of the selected project, "The A11Y Project / a11yproject.com." This view includes project details, files, branches, commits, and other relevant information.
|
||
- **Possible Purpose:** The purpose is to access the specific project "The A11Y Project / a11yproject.com" to view its details, files, or to perform actions related to this project.
|
||
|
||
#### 8. **Action:** I click on the "New project" button located in the top-right corner of the page.
|
||
- **Page Changes:** A modal window or a new page appears, providing options to create a new project. This includes fields for entering the project name, description, and other settings.
|
||
- **Possible Purpose:** The intent is to start the process of creating a new project within the GitLab platform.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the GitLab platform by using the search functionality, navigating to the Projects section, browsing through a list of projects, selecting a specific project, and initiating the creation of a new project. Each action is performed with the intent to explore, manage, or create projects within the platform.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar at the top of the GitLab page.
|
||
- **Page Changes:** A dropdown menu appears below the search bar, displaying various search options such as "Issues assigned to me," "Merge requests assigned to me," "Merge requests that I'm a reviewer," and "Merge requests I've created."
|
||
- **Possible Purpose:** The likely intent is to initiate a search or navigate quickly to specific sections related to issues and merge requests within GitLab.
|
||
|
||
#### 2. **Action:** I hover over the "Projects" section on the left sidebar.
|
||
- **Page Changes:** No immediate changes occur on the page, but the cursor highlights the "Projects" section, indicating focus.
|
||
- **Possible Purpose:** This action suggests an intention to interact with the projects list or to ensure the correct section is selected before proceeding with further actions.
|
||
|
||
#### 3. **Action:** I click on the "Yours" tab under the "Projects" section.
|
||
- **Page Changes:** The projects list updates to display only the projects associated with "Yours," which is indicated by the number "23" next to it.
|
||
- **Possible Purpose:** The purpose is to filter and view the projects that are specifically assigned or related to the user's account.
|
||
|
||
#### 4. **Action:** I use the browser's developer tools (as indicated by the highlighted code snippets) to inspect elements on the page.
|
||
- **Page Changes:** Developer tool overlays appear, showing CSS selectors and HTML structure for different elements on the page, such as project names and their corresponding details.
|
||
- **Possible Purpose:** The intent is to analyze the webpage's structure, possibly for debugging, automation scripting, or understanding the layout for further interaction.
|
||
|
||
#### 5. **Action:** I highlight and possibly copy a CSS selector from the developer tools.
|
||
- **Page Changes:** The highlighted selector corresponds to a specific project entry, suggesting an interest in that particular element.
|
||
- **Possible Purpose:** This action is likely aimed at using the selector for tasks such as automated testing, custom scripting, or detailed inspection of that project's properties.
|
||
|
||
#### 6. **Action:** I continue to navigate through the projects list, focusing on different project entries.
|
||
- **Page Changes:** The cursor moves over various project titles and descriptions, with no clicks but potential preparation for selection.
|
||
- **Possible Purpose:** The user is reviewing the list to locate a specific project or to assess the available projects before taking further action.
|
||
|
||
### Summary
|
||
In this video segment, I primarily interact with the search functionality and the projects list on the GitLab page. My actions include searching for specific items, filtering projects to those assigned to me, and using developer tools to inspect and analyze the page's structure. These steps suggest a focus on navigating, analyzing, and potentially automating interactions with the GitLab interface.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video starts with a view of a GitLab dashboard displaying a list of projects.
|
||
- **Page Changes:** The page shows various project names, their descriptions, and additional details like the number of stars, forks, and last updated times.
|
||
- **Possible Purpose:** The initial state sets the context, indicating that the user is navigating through their GitLab projects.
|
||
|
||
#### 2. **Scrolling Through Projects**
|
||
- **Action:** I scroll down the list of projects.
|
||
- **Page Changes:** As I scroll, different projects come into view, revealing more options and details about each project.
|
||
- **Possible Purpose:** The purpose of scrolling is to locate a specific project or to review the available projects.
|
||
|
||
#### 3. **Selecting a Project**
|
||
- **Action:** I click on the project named "The A11Y Project / a11yproject.com."
|
||
- **Page Changes:** The page transitions to the project's main repository view, showing detailed information such as the project description, commit history, and repository files.
|
||
- **Possible Purpose:** The intent is to access the specific project for further actions, such as viewing its contents, making changes, or interacting with its features.
|
||
|
||
#### 4. **Viewing Project Details**
|
||
- **Action:** I observe the project details, including the project description and the main repository files.
|
||
- **Page Changes:** The page remains on the project's main view, providing information about the project's purpose, recent updates, and available actions (e.g., clone, fork).
|
||
- **Possible Purpose:** The purpose is to gather information about the project or to prepare for further interactions, such as cloning the repository or exploring its contents.
|
||
|
||
#### 5. **Hovering Over Elements**
|
||
- **Action:** I hover over certain elements on the page, such as the project name and description.
|
||
- **Page Changes:** Hovering over these elements may highlight them or provide tooltips, but no significant page changes occur.
|
||
- **Possible Purpose:** The intent is to focus on specific parts of the project details, possibly to ensure the correct project is selected or to read additional information provided by tooltips.
|
||
|
||
#### 6. **Ending State**
|
||
- **Action:** The video segment ends with the project's main repository view still displayed.
|
||
- **Page Changes:** No further changes occur as the segment concludes.
|
||
- **Possible Purpose:** The ending state confirms that the user has successfully navigated to the desired project and is ready for the next set of actions, which are not shown in this segment.
|
||
|
||
### Summary
|
||
In this video segment, I navigate through a list of GitLab projects, scroll to find a specific project, select "The A11Y Project / a11yproject.com," and view its details. My actions are focused on locating and accessing the project for further interaction, with careful attention to the project's main repository view. Each step is deliberate, ensuring the correct project is selected and its details are reviewed.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial Observation**
|
||
- **Action:** I observe the GitLab project page for a repository named "design."
|
||
- **Page Changes:** The page displays various sections including project information, repository details, issues, merge requests, and more. A notification at the top suggests adding an SSH key to enable SSH-based repository operations.
|
||
- **Possible Purpose:** The initial observation sets the context for interacting with the project's repository settings and features.
|
||
|
||
#### 2. **Hovering Over the Project Name**
|
||
- **Action:** I move the cursor over the project name "design" located near the top of the page.
|
||
- **Page Changes:** A tooltip appears, providing additional information about the project's visibility status ("Public - the project can be accessed without any authentication") and other details such as the number of commits, branches, tags, and project storage size.
|
||
- **Possible Purpose:** Hovering over the project name is likely to gather more detailed information about the project's current settings and statistics, aiding in understanding its scope and access level.
|
||
|
||
#### 3. **Moving Cursor Away from the Project Name**
|
||
- **Action:** I move the cursor away from the project name "design."
|
||
- **Page Changes:** The tooltip that appeared when hovering over the project name disappears, returning the view to the standard project page layout.
|
||
- **Possible Purpose:** Moving the cursor away clears the additional information provided by the tooltip, possibly to focus on other elements or actions on the page.
|
||
|
||
### Summary
|
||
In this video segment, my actions are focused on observing and interacting with the project name "design" on a GitLab project page. Specifically, I hover over the project name to reveal a tooltip with detailed project information and then move the cursor away to dismiss the tooltip. These actions suggest an intent to quickly review and understand the project's basic details and settings. No other interactions or page navigations are observed in this segment.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clicklocator(\"a\").filter(has_text=re.compile(r\"^2$\"))",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"1\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&page=2&sort=name_asc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&page=2&sort=name_asc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_2",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search GitLab\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Projects New project",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_text(\"Projects New project\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"[id=\\\"__BVID__10__BV_toggle_\\\"]\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menu-sidebar",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_test_id(\"menu-sidebar\").get_by_label(\"Projects\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/?non_archived=true&sort=name_asc"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Starred 3",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Starred 3\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/projects/starred"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/projects/starred"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Yours 23",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Yours 23\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/projects/starred"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/dashboard/projects/starred"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_The A11Y Project\\n/ a11yproject.com",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"The A11Y Project\\n/ a11yproject.com\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "heading_a11yproject.com Public - The project can be accessed without any authentication.",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_role(\"heading\", name=\"a11yproject.com Public - The project can be accessed without any authentication.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_12",
|
||
"idx": 12,
|
||
"action_repr": "frame.presslocator(\"body\")Control+c",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Primer\\n/ design",
|
||
"idx": 13,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Primer\\n/ design\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/primer/design"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/primer/design"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "heading_design Public - The project can be accessed without any authentication.",
|
||
"idx": 24,
|
||
"action_repr": "frame.clickget_by_role(\"heading\", name=\"design Public - The project can be accessed without any authentication.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/primer/design"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:8023/primer/design"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |