267 lines
13 KiB
Plaintext
267 lines
13 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Fill the "contact us" form in the site for a refund on the speaker I bought, stating that it broke after just three days of use. Also, ensure to include the order number #148 and the product SKU. Don't submit yet, I will check.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the "Contact Us" link.
|
||
- **Page Changes**: The page transitions to a "Contact Us" form where I can input my details and message.
|
||
- **Possible Purpose**: The likely intent is to reach out to the customer support team for assistance or inquiries.
|
||
|
||
2. **Action**: I click inside the "Name" text box and type "Emma231 Willow Way, Suite 100, Chicago, IL, 606012EmmaEmmaEn".
|
||
- **Page Changes**: The text I entered appears in the "Name" field.
|
||
- **Possible Purpose**: The purpose is to fill in the required "Name" field with the specified text, possibly as part of a test or specific instruction.
|
||
|
||
3. **Action**: I click inside the "Email" text box and type "emma.lopez@gmail.com".
|
||
- **Page Changes**: The email address is displayed in the "Email" field.
|
||
- **Possible Purpose**: This action aims to provide a valid email address for contact purposes.
|
||
|
||
4. **Action**: I click inside the "What's on your mind?" text box and type "I would like to refund for order #148".
|
||
- **Page Changes**: The message text appears in the "What's on your mind?" field.
|
||
- **Possible Purpose**: The intention is to draft a message requesting a refund for a specific order, indicating a need for customer service related to a purchase.
|
||
|
||
5. **Action**: I click the "Submit" button.
|
||
- **Page Changes**: There is no immediate visual feedback shown in the video regarding the submission status (e.g., confirmation message or error).
|
||
- **Possible Purpose**: The goal is to send the filled form to the customer support team for processing.
|
||
|
||
These actions are performed sequentially as observed in the video segment, focusing solely on the interactions within the "Contact Us" page.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Contact Us" link.
|
||
- **Page Changes:** The webpage transitions from the "My Orders" section to the "Contact Us" page. The new page displays a form with fields for "Name," "Email," "Phone Number," and "What's on your mind?" along with a "Submit" button.
|
||
- **Possible Purpose:** The likely intent is to navigate to the "Contact Us" page to initiate communication, possibly to address an issue or request information.
|
||
|
||
#### 2. **Action:** I fill in the "Name" field with "Emma Lopez."
|
||
- **Page Changes:** The "Name" field now contains the text "Emma Lopez."
|
||
- **Possible Purpose:** The purpose is to provide the required personal information as part of the contact form submission process.
|
||
|
||
#### 3. **Action:** I fill in the "Email" field with "emma.lopez@gmail.com."
|
||
- **Page Changes:** The "Email" field now contains the text "emma.lopez@gmail.com."
|
||
- **Possible Purpose:** The purpose is to provide a valid email address for the sender, which is necessary for any follow-up communication.
|
||
|
||
#### 4. **Action:** I click inside the "What's on your mind?" text area and type "I would like to refund for order #148."
|
||
- **Page Changes:** The text "I would like to refund for order #148" appears in the "What's on your mind?" text area.
|
||
- **Possible Purpose:** The purpose is to draft a message requesting a refund for a specific order, indicating the user's intention to communicate this request to customer support.
|
||
|
||
#### 5. **Action:** I hover over the "Submit" button.
|
||
- **Page Changes:** There are no immediate changes to the page, but the cursor changes to indicate interactivity with the "Submit" button.
|
||
- **Possible Purpose:** The likely intent is to prepare to submit the filled-out form, finalizing the request for a refund.
|
||
|
||
### Summary
|
||
In this video segment, I navigated from the "My Orders" section to the "Contact Us" page, filled out the contact form with my name, email, and a specific refund request, and prepared to submit the form. Each action was methodical, aimed at initiating a customer service request for a refund.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the text box labeled "What's on your mind?".
|
||
- **Page Changes**: The text box becomes active, and a blinking cursor appears inside it, indicating that it is ready for text input.
|
||
- **Possible Purpose**: The likely intent is to enter a message or query into the text box as part of the contact form.
|
||
|
||
2. **Action**: I type the message: "I would like to refund for the speaker I bought in order #148, because it was broken after 3 days of use."
|
||
- **Page Changes**: The text appears within the text box, filling the space as I type.
|
||
- **Possible Purpose**: The purpose is to communicate a specific request for a refund due to a product issue, providing necessary details such as the order number and the reason for the request.
|
||
|
||
3. **Action**: I click on the "Submit" button below the text box.
|
||
- **Page Changes**: There are no visible changes to the page layout or content immediately after clicking the "Submit" button in this segment.
|
||
- **Possible Purpose**: The intent is to send the entered message to the recipient, completing the action of submitting the contact form.
|
||
|
||
These actions are focused on filling out and submitting a contact form with a specific refund request.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the text box labeled "What's on your mind?".
|
||
- **Page Changes**: The text box becomes active, and a blinking cursor appears inside it, indicating that it is ready for text input.
|
||
- **Possible Purpose**: The likely intent is to enter a message or query into the text box as part of filling out the contact form.
|
||
|
||
2. **Action**: I type the following message into the text box:
|
||
> "I would like to refund for the speaker I bought in order #148, because it was broken after 3 days of use. The sku is B003FWW3VA."
|
||
- **Page Changes**: The text appears within the text box, replacing any previous content.
|
||
- **Possible Purpose**: The purpose is to provide a detailed request for a refund, including relevant order information and the reason for the request.
|
||
|
||
3. **Action**: I highlight the text box by clicking and dragging over the text area.
|
||
- **Page Changes**: The entire text box is highlighted, showing a visual selection around it.
|
||
- **Possible Purpose**: This action might be intended to review, edit, or ensure the accuracy of the entered text before proceeding.
|
||
|
||
4. **Action**: I click on the "Submit" button below the text box.
|
||
- **Page Changes**: There are no immediate visible changes to the page, suggesting that the submission process may be processing or that feedback is not instantly displayed.
|
||
- **Possible Purpose**: The intent is to send the filled-out form with the entered information to the recipient for further action.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions and their immediate effects on the webpage.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Contact Us",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Contact Us\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_What\u2019s on your mind?",
|
||
"idx": 32,
|
||
"action_repr": "frame.pressget_by_label(\"What\u2019s on your mind?\")ArrowRight",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/contact/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/contact/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_View All",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"View All\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/customer/account/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue608 Page Next",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue608 Page Next\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "cell_000000148",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"cell\", name=\"000000148\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=2"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=2"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "row_000000148 1/29/23 $440.64 Complete View Order Reorder",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"row\", name=\"000000148 1/29/23 $440.64 Complete View Order Reorder\").get_by_role(\"link\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=2"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/history/?p=2"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "cell_B003FVW3VA",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"cell\", name=\"B003FVW3VA\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/148/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7770/sales/order/view/order_id/148/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |