238 lines
11 KiB
Plaintext
238 lines
11 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Follow ['Jakub Klinkovský', 'Koushik', 'Vinta Chen'] on Gitlab
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with a view of a GitLab dashboard, specifically the "Projects" section under "Yours."
|
||
- **Page Changes:** The page displays a list of projects with details such as project names, descriptions, update times, and interaction metrics (stars, forks, issues, etc.).
|
||
- **Possible Purpose:** The initial state sets the context for managing or navigating through personal projects on GitLab.
|
||
|
||
#### 2. **Navigating to User Profiles**
|
||
- **Action:** I click on a user profile link located in the top-right corner of the screen.
|
||
- **Page Changes:** The page transitions from the "Projects" section to a user profile page.
|
||
- **Possible Purpose:** The action aims to access detailed information about a specific user's activities, contributions, and personal projects on GitLab.
|
||
|
||
#### 3. **Observing the User Profile Page**
|
||
- **Action:** I review the user profile page, which includes sections like "Overview," "Activity," "Groups," "Contributed projects," "Personal projects," etc.
|
||
- **Page Changes:** The page shows the user's username, join date, follower count, following count, and recent activity.
|
||
- **Possible Purpose:** The purpose is to gather information about the user's engagement, contributions, and personal projects on the platform.
|
||
|
||
#### 4. **Switching Between User Profiles**
|
||
- **Action:** I navigate to another user profile by clicking on a different user link, likely found in a dropdown menu or another part of the interface.
|
||
- **Page Changes:** The page updates to display the new user's profile, including their username, location, bio, and activity details.
|
||
- **Possible Purpose:** This action is intended to compare or review the details of multiple user profiles, possibly for collaboration, following, or analysis purposes.
|
||
|
||
#### 5. **Using Developer Tools**
|
||
- **Action:** I open the developer tools (as indicated by the visible code snippets and highlighted elements) and interact with the search bar on the user profile page.
|
||
- **Page Changes:** The developer tools panel appears, showing code snippets that seem to target UI elements like the search bar.
|
||
- **Possible Purpose:** The use of developer tools suggests an intent to inspect, test, or manipulate the webpage elements, possibly for debugging, automation, or learning purposes.
|
||
|
||
### Summary
|
||
In this video segment, I start by viewing my projects on GitLab, then navigate to and observe two different user profiles, and finally use developer tools to interact with the webpage elements. Each step is methodical, focusing on accessing and analyzing user-specific information and webpage functionality.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action: I click on the search bar at the top of the GitLab page.**
|
||
- **Page Changes:** The search bar becomes active, and a dropdown menu appears below it, suggesting search options such as "Users."
|
||
- **Possible Purpose:** The likely intent is to initiate a search for a specific user or item within GitLab.
|
||
|
||
#### 2. **Action: I type "Vinta" into the search bar.**
|
||
- **Page Changes:** As I type, the dropdown menu updates to show search results that match the input "Vinta." A user named "Vinta Chen" appears in the results.
|
||
- **Possible Purpose:** The purpose is to locate the user "Vinta Chen" within GitLab.
|
||
|
||
#### 3. **Action: I select "Vinta Chen" from the search results.**
|
||
- **Page Changes:** The page transitions to Vinta Chen's profile page.
|
||
- **Possible Purpose:** The intent is to view the details of Vinta Chen's GitLab profile.
|
||
|
||
#### 4. **Action: I hover over the "Follow" button on Vinta Chen's profile page.**
|
||
- **Page Changes:** No immediate changes occur, but the button is highlighted, indicating it is interactive.
|
||
- **Possible Purpose:** The likely intent is to follow Vinta Chen's activities on GitLab.
|
||
|
||
#### 5. **Action: I click on the "Follow" button.**
|
||
- **Page Changes:** The "Follow" button changes to "Unfollow," indicating that I am now following Vinta Chen.
|
||
- **Possible Purpose:** The purpose is to start receiving updates about Vinta Chen's activities on GitLab.
|
||
|
||
#### 6. **Action: I scroll down to the "Activity" section on Vinta Chen's profile page.**
|
||
- **Page Changes:** The view shifts to display Vinta Chen's recent activity, including contributions and project updates.
|
||
- **Possible Purpose:** The intent is to review the recent activities and contributions made by Vinta Chen.
|
||
|
||
#### 7. **Action: I hover over a specific date in the contributions calendar.**
|
||
- **Page Changes:** A tooltip appears, showing the number of contributions made by Vinta Chen on that specific date.
|
||
- **Possible Purpose:** The likely intent is to check the details of Vinta Chen's contributions on a particular day.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to locate and follow a user named Vinta Chen on GitLab, and then review their recent activities and contributions. Each action is focused on navigating the GitLab interface to gather specific information about the user's profile and activities.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search GitLab\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_lahwaacz",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_label(\"lahwaacz\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Follow",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Follow\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_3",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search GitLab\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_label(\"koush\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/koush"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/lahwaacz"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search GitLab\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/koush"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/koush"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"vinta\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/koush"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/koush"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |