Description
About HumanSignal
Real-world data is the competitive edge in AI.
HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery.
We design and create datasets from scratch, recruit and manage the domain experts who evaluate model output, and run everything through our own platform, Label Studio, the open-source standard for data labeling and evaluation, used by over 1 million practitioners worldwide.
We specialize in the operationally complex: real-world data collection, multimodal pipelines, and multi-step workflows. Advanced ML and AI teams use our enterprise platform to run their own data factories, and our services team to extend their reach where in-house capacity runs out.
If you want to do work that materially shapes how the next generation of AI products gets built, we'd love to talk.
Graphic Design Expert
Location: Remote
Type: Part-Time Contract (15+ hours per week for 2 weeks | 30 hour minimum commitment)
Duration: 2 weeks with potential for ongoing work
Compensation: Up to $20/hour for US based candidates
About the Project
You'll evaluate and rate graphic design elements on standardized quality scales to train AI models that assess design effectiveness—contributing to cutting-edge technology that advances how AI understands and evaluates visual content quality.
What You'll Do
- Review and rate graphic design elements using our Label Studio Enterprise platform across multiple quality dimensions:
- Typography quality (readability, hierarchy, font pairing appropriateness)
- Visual hierarchy effectiveness (clarity, balance, focal point strength)
- Design element execution (icon quality, illustration polish, image selection)
- Color palette effectiveness (harmony, contrast, accessibility)
- Layout and composition quality (grid usage, spacing, alignment)
- Overall design effectiveness and professional polish
- Apply consistent 1-5 rating scales across hundreds of design examples
- Provide clear, objective assessments based on established design principles
- Follow detailed rating guidelines and rubrics to maintain consistency
- Process high volumes of design samples with sustained attention to detail
- Participate in weekly calibration sessions to align rating standards
- Meet quality benchmarks for rating consistency and inter-rater reliability
Basic Qualifications
Required Experience
- Formal training in graphic design, visual arts, or related field (degree, certificate, or bootcamp completion) OR 2-3 years of professional graphic design experience
- Strong proficiency in design principles (composition, color theory, typography, visual hierarchy)
- Ability to objectively critique and evaluate design work against industry standards
- Understanding of what constitutes effective vs. ineffective design choices
- Experience reviewing or providing design feedback
- Exceptional attention to detail and ability to maintain focus during repetitive tasks
Skills & Attributes
- Comfort with highly repetitive work—ability to maintain quality and engagement while rating similar content for extended periods
- Strong visual discrimination and consistent aesthetic judgment
- Reliability and consistency in applying rating standards
- Self-motivated with ability to work independently on repetitive tasks
- Excellent focus and concentration for sustained periods
- Strong communication skills for flagging edge cases or unclear examples
- Comfortable with feedback and recalibration
- Minimum 10 hours per week availability for at least 3 months
Nice-to-Haves
Preferred Background
- Portfolio demonstrating diverse design work and critical eye
- Experience as a design reviewer, art director, or in quality assurance roles
- Understanding of accessibility standards (WCAG, inclusive design principles)
- Knowledge of design across multiple contexts (branding, editorial, digital, UX/UI)
- Experience teaching or mentoring design concepts
Technical Experience
- Experience with Figma, Sketch, or other modern design tools
- Prior data annotation, content moderation, or quality evaluation experience
- Exposure to machine learning or computer vision concepts
- Familiarity with design systems and component libraries
Work Style
- High tolerance for repetitive evaluation tasks—finds rhythm and satisfaction in systematic work
- Ability to maintain consistent standards across thousands of ratings
- Strong pattern recognition—quickly identifying quality patterns across examples
- Disciplined, methodical approach with minimal variation in judgment
- Problem-solving mindset for handling ambiguous or borderline cases
- Interest in AI/ML applications in design and creative fields
- Previous remote work experience with self-directed tasks
What Makes Someone Successful in This Role
You thrive on consistency and repetition - You find satisfaction in systematic work and can maintain quality standards while evaluating similar content repeatedly without losing focus or accuracy.
You have a calibrated design eye - You can quickly assess design quality against objective criteria and apply the same standards consistently across hundreds or thousands of examples.
You're objective and principled - You separate personal taste from professional assessment, basing ratings on established design principles rather than subjective preference.
You maintain sustained focus - You can work through high volumes of repetitive tasks while staying sharp and attentive to subtle quality differences.
You're a clear communicator - You flag confusing examples or rating criteria early, ask clarifying questions, and document edge cases clearly.
You're curious about technology - You're genuinely interested in how AI learns to evaluate design quality and see value in contributing to machine learning training data.
Similar jobs
Est. 41,600 USD
About HumanSignal Real-world data is the competitive edge in AI. HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their rese…
About HumanSignal Real-world data is the competitive edge in AI. HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their rese…
About HumanSignal Real-world data is the competitive edge in AI. HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their rese…
A career that’s the whole package! At Conga, we’ve built a community where our colleagues can thrive. Here you’ll find opportunities to innovate and support growth through individual and team development, all within an e…
Brandtech+ (formerly known as OLIVER+) is a global team of creative thinkers, tech-savvy trendsetters, and production pros specialising in film, CGI, automation, AI, motion design, and digital/print content. We partner w…
Company Overview APPLICATIONS FROM OUTSIDE COLOMBIA AND ARGENTINA WILL NOT BE CONSIDERED FOR THIS ROLE Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design…
Brandtech+ (formerly known as OLIVER+) is a global team of creative thinkers, tech-savvy trendsetters, and production pros specialising in film, CGI, automation, AI, motion design, and digital/print content. We partner w…
Established in 2004, OLIVER is the world’s first and only specialist in designing, building, and running bespoke in-house agencies and marketing ecosystems for brands. We partner with over 300 clients in 40+ countries an…
Est. 140,000 USD
About Us Red Cell Partners is an incubation firm building and investing in rapidly scalable technology-led companies that are bringing revolutionary advancements to market in three distinct practice areas: healthcare, cy…
Brandtech+ (formerly known as OLIVER+) is a global team of creative thinkers, tech-savvy trendsetters, and production pros specialising in film, CGI, automation, AI, motion design, and digital/print content. We partner w…
Established in 2004, OLIVER is the world’s first and only specialist in designing, building, and running bespoke in-house agencies and marketing ecosystems for brands. We partner with over 300 clients in 40+ countries an…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Est. 140,000 USD
Location: US or CanadaType: US Applicants - Full-Time; Canadian Applicants - Independent Contractor **A note before you read further** The world is changing fast, and so is this role. We can't fully tell you what this jo…
Brandtech+ (formerly known as OLIVER+) is a global team of creative thinkers, tech-savvy trendsetters, and production pros specialising in film, CGI, automation, AI, motion design, and digital/print content. We partner w…
APPLICATIONS FROM OUTSIDE COLOMBIA WILL NOT BE CONSIDERED FOR THIS ROLERobots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that int…
Brandtech+ (formerly known as OLIVER+) is a global team of creative thinkers, tech-savvy trendsetters, and production pros specialising in film, CGI, automation, AI, motion design, and digital/print content. We partner w…
Est. 45,000 EUR
About Remedy Edge Barcelona: As part of one of the world’s most prestigious healthcare advertising agencies, Remedy Edge Barcelona brings people, brands, ideas and innovation together to change real lives in extraordinar…
Est. 115,000 USD
About Judi Health Judi Health is a health technology company providing benefit administration solutions to employers, unions, health plans, and government entities. Judi Health replaces fragmented, outdated systems with…
Est. 105,000 USD
At OneSpan, we specialize in digital identity and anti-fraud solutions that create exceptional and secure experiences.Elevate the Brand. Build the Foundation. Shape the Experience. Great brands are constantly evolving. W…
Est. 115,000 USD
About Us SharkNinja is a global product design and technology company, with a diversified portfolio of 5-star rated lifestyle solutions that positively impact people’s lives in homes around the world. Powered by two trus…
Est. 84,000 PLN
Location: Poland We Are: RTB House is a next-generation performance demand-side platform (DSP) that uses proprietary Deep Learning AI algorithms to help brands grow. The company is the market leader in driving performanc…
Est. 125,000 USD
About Us SharkNinja is a global product design and technology company, with a diversified portfolio of 5-star rated lifestyle solutions that positively impact people’s lives in homes around the world. Powered by two trus…
Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for ou…