Skip to main content
A quality score is a score assigned on a scale from 0 to 100 and is skill-based. For instance, an expert that did an amazing job in cardiology projects can be scored at 90, but if they also have internal medicine skill — it would require a separate work history to be evaluated. Each submission from an expert goes through an internal segmentation process, which adds (or removes) points based on the category of a submission. For instance, if an expert submitted a prompt that’s oncology-specific but has a part that talks about cardiology, an expert will see their score improve largely for oncology, and less so (if at all) for cardiology. We treat quality as a threefold equation. First, the quality of an expert. Based on their results on training sessions, we adjust their scores and leveling. Second, the quality of their work. This is a combination of their efforts across multiple projects. It has both positive and negative signals that are taken into account. Positive signals include:
  • Acceptance rate — how many of their submissions are getting accepted.
  • Execution speed — how fast they can start working from the moment they’ve been whitelisted on a project.
  • Survival rate — how long they’ve been working with us.
  • And other signals.
For the negative ones, we check:
  • Failure rate — how many submissions have been rejected.
  • Fraud flags — if we suspect fraudulent behavior from an expert (we’re happy to discuss it in detail with you, but would prefer not to share this information as a fraud prevention measure). Overall, we track about 12 signals, including communication fingerprints to ensure validity of an expert.
Third, it’s a project-specific score. While an expert can come from a leading medical school with a decade of experience, that doesn’t necessarily translate into them being the best candidate for an AI training project, which is why on top of the 0–100 score, we add another layer of a project-specific quality. Similarly to the above, there are positive and negative signals we take into account. Main factors that contribute to the score are:
  • Evaluation of their submission — a reviewer rates the submission for overall quality, and separately based on project-specific human-expert rubrics.
  • The richness of the provided feedback.
  • The feedback (if available) from a customer.
Combining the detailed sourcing pipelines with our own data and the detailed quality profile, we’re able to find the right people fast and replace candidates that don’t meet the bar before it’s too late.