Sherlog - Gmail Live AI Feature Evaluation Project

Truelancer AI is recruiting participants for a research project evaluating a new AI-powered feature in Gmail called Ask Gmail.

What is Ask Gmail?

Ask Gmail allows you to ask questions about information available in your Gmail account using natural language.

For example, you may ask:

• “How many flights did I take in the last 30 days?”
• “When is my next meeting?”
• “Find the email where I discussed my upcoming travel plans.”

The AI analyzes information from your emails and provides an answer.

What will you do?

As part of the study, you will evaluate two different AI-generated responses to the same question.

For each task:

1. You will be given a prompt/question to ask in Gmail.
2. You will ask the same prompt using two different AI model options.
3. You will review the AI-generated responses (AI Overviews).
4. You will provide feedback on each response individually.
5. You will then compare both responses and indicate which response better answers the question and why.

The purpose is to understand how well different AI responses answer questions based on information available in Gmail.

Time Commitment

• Time per task: 1 hour

• You will need to complete all assigned tasks and provide feedback for each evaluation.

What participants need to do

Participants should be comfortable with:

• Using Gmail
• Following detailed task instructions
• Asking questions/prompts in Gmail
• Reviewing AI-generated responses
• Providing thoughtful and accurate feedback
• Comparing two responses and explaining which one better addresses the question

Important

This is a research and evaluation project. Your participation involves using the assigned Gmail environment/features and evaluating AI-generated responses based on the provided instructions.

Please follow all project instructions carefully and do not share screenshots, email content, prompts, responses, or other project information outside the study.

Participation

Compensation and payment details will be shared separately with eligible participants during onboarding.

Who should participate?

We are looking for participants who:

• Have access to and regularly use Gmail
• Are comfortable using new Gmail/AI features
• Can commit approximately 6 hours across 2 days
• Can carefully evaluate and compare AI-generated answers
• Can complete all 6 assigned tasks within the project timeline

If you are interested, please complete the registration/onboarding process to confirm your eligibility.

Guidelines

Steps to Proceed

Step 1: Join Google Workspace Labs
https://workspace.google.com/experiments/

Click “Join to try” → Check all 3 boxes → Save and take a screenshot showing your email address.

Tutorial:
https://drive.google.com/file/d/1huBkPBKxAQKi8v1n8i8z-sT3xG9Kamj1/view

Step 2: Enable Smart Features

Go to Gmail → Settings → General and turn ON both:

• Smart features and personalization
• Google Workspace smart features and personalization

Tutorial:
https://drive.google.com/file/d/1z8LKJP5DAWDeaWNWgX22QJpxPAaQxtLW/view

Step 3: Join the WhatsApp Group

Please join the WhatsApp group to receive project updates and important coordination messages quickly.

WhatsApp Group Link - https://chat.whatsapp.com/G2awLLCfTeGHOtbsbULYcB

Important: This is an urgent project, so we need everyone to stay aligned and receive updates promptly. We will only consider participants who have joined the WhatsApp group.

Please complete the steps as soon as possible and let us know once they are completed.

Thank you for your cooperation.