Skip to main content

How to Provide Feedback on AI Models and Scoring in PyjamaHR: A Complete Guide for Recruiters and Hiring Teams

Updated this week

Executive Summary

PyjamaHR’s AI-powered features—such as Resume Match Scoring and AI Interviewer—are central to streamlining candidate evaluation. However, many users have questions about how to interpret, trust, and provide feedback on these AI-driven scores and interview assessments. This article explains how PyjamaHR’s AI scoring works, how you can provide actionable feedback to improve model accuracy, and how to troubleshoot common issues, ensuring you get the most out of your AI-powered hiring process.

Detailed Overview

What Are AI Models and Scoring in PyjamaHR?

AI Models and Scoring in PyjamaHR refer to automated systems that analyze candidate resumes and interview responses to generate objective, data-driven scores. The two main areas where this applies are:

  • Resume Match Score: Automatically evaluates how well a candidate’s resume aligns with the job description, required skills, and experience.

  • AI Interviewer Scoring: Assesses candidate responses in automated interviews, providing a structured evaluation based on predefined criteria.

Why It Matters:
These features help recruiters quickly identify top candidates, reduce manual screening time, and ensure a fair, consistent evaluation process. However, the effectiveness of these models depends on both the quality of your job setup and your feedback on their performance.

Integration with Other PyjamaHR Features

  • Screening Questions: Enhance AI scoring by adding clear, relevant screening questions.

  • Job Descriptions: The more detailed and accurate your job description, the better the AI can match candidates.

  • Candidate Profiles: Scores are visible directly in candidate profiles and can be used to filter or shortlist applicants.

Step-by-Step Guide: How to Provide Feedback on AI Models and Scoring

1. Interpreting AI Scores

  • Resume Match Score:

  • Found in each candidate’s profile under the applied job.

  • Categorized as High, Medium, or Low match.

  • Based on alignment with job responsibilities, primary/secondary skills, and experience.

  • AI Interviewer Score:

  • Available in the candidate’s interview report.

  • Based on responses to structured questions and overall fit for the role.

Tip: Always review the detailed breakdown or rationale behind the score (if available), not just the numeric or categorical value.

2. Providing Feedback on AI Scoring

A. For Resume Match Scores

  1. Identify Mismatches or Inaccuracies:
    - If you notice a candidate scored “High” but lacks key skills, or a “Low” score for a seemingly strong fit, note the specifics.

  2. Contact Support with Context:
    - Use the in-app chat or support email.
    - Provide:

    • Job ID and candidate name

    • Screenshot of the issue (if possible)

    • Description of the mismatch (e.g., “Candidate scored High but does not have ‘Argo’ skill as required”)

  3. Request Manual Review or Adjustment:
    - Support can manually trigger a re-evaluation or escalate the issue to the product team.

  4. Suggest Improvements:
    - If you consistently see issues (e.g., certain skills not weighted properly), share this pattern with support for model retraining.

B. For AI Interviewer Scoring

  1. Review the Interview Report:
    - Look for areas where the AI may have misinterpreted answers or scored unfairly.

  2. Document Specific Examples:
    - Note the question, candidate’s answer, and why you believe the score is off.

  3. Submit Feedback:
    - Use the feedback/comment section in the interview report (if available).
    - Or, contact support with:

    • Candidate name and interview link

    • Details of the scoring issue

  4. Request Human Review:
    - For critical roles, ask for a manual review by a recruiter or admin.

C. General Feedback on AI Behavior

  • If you notice the AI interviewer is too fast, interrupts candidates, or the scoring feels unreliable, provide:

  • Specific candidate examples

  • Description of the behavior (e.g., “AI interrupted mid-sentence”)

  • Any changes in model (e.g., switch from “Suchal” to “Vierra”)

3. Advanced Usage & Best Practices

  • Refine Job Descriptions:
    The more detailed and accurate your job description and skill requirements, the better the AI scoring.

  • Use Screening Questions:
    Add clear, role-specific screening questions to improve AI match accuracy.

  • Regularly Review Scores:
    Don’t rely solely on AI scores—use them as a starting point, then review candidate profiles for context.

  • Provide Ongoing Feedback:
    If you see recurring issues, escalate them to support or your account manager. Your feedback helps improve the AI for everyone.

  • Leverage Activity Logs:
    Use the Activity Panel and Timeline in candidate/job profiles to track changes and feedback history.

Troubleshooting & Common Issues

Common Issues and Solutions

Issue

Solution

Resume Match Score not appearing

Wait 10–20 minutes after job/candidate creation. If still missing, contact support with Job ID.

Scores seem inaccurate (e.g., High match with missing skills)

Provide candidate/job details and examples to support for review.

AI Interviewer is too fast or interrupts

Report specific candidate examples to support; team can adjust AI behavior.

Feedback form not loading

Try a hard refresh or different browser; if unresolved, share a screenshot with support.

Unable to submit feedback via mobile app

Use the desktop browser version; mobile app feedback is not supported yet.

Can’t see AI Recruiter section as interviewer

Only Super Admins, Admins, Recruiters, and Hiring Managers have access. Request a role change if needed.

Feedback not visible to other interviewers

By design, to avoid bias. Only your own feedback is visible. Request admin access for broader visibility.

What If Scenarios

  • What if the AI scoring is consistently off for a specific role?

  • Share multiple examples with support; the product team may adjust the model for your use case.

  • What if I want to customize the AI interviewer’s questions or scoring?

  • You can specify screening criteria in the “Configure AI” section. For advanced customization, contact support.

  • What if I want to disable or remove AI scoring for a job?

  • Contact support with the job ID; they can remove the AI Recruiter pipeline if configured by mistake.

Comprehensive FAQ

1. Can I see why a candidate received a specific AI score?
Yes, review the candidate’s profile or interview report for a breakdown of the score. If unclear, contact support for details.

2. How do I provide feedback if the AI scoring seems wrong?
Document the issue (job ID, candidate, screenshot, explanation) and send it to support.

3. Can I adjust how the AI scores candidates?
You can influence scoring by refining job descriptions and screening questions. For deeper changes, provide feedback to support.

4. Why do I have to contact support to enable or fix resume match scores?
While resume match is automatic for new jobs, occasional glitches may require manual intervention. The team is working to make this fully self-serve.

5. Why are so many candidates marked as “High” match?
If too many candidates are marked “High,” review your job’s required skills and responsibilities. The AI scores relative to the job setup.

6. Can interviewers see each other’s feedback?
No, to avoid bias. Only admins and recruiters can see all feedback.

7. Can I submit feedback via the mobile app?
Not currently; use the desktop browser.

8. What if the AI interviewer is too fast or interrupts?
Report specific examples to support; the team can adjust the AI’s behavior.

9. How do I request a manual review of AI scores?
Contact support with candidate/job details and your concerns.

10. Can I customize the AI interviewer’s personality or questions?
Some customization is possible via support; advanced options may require backend changes.

11. What if the feedback form doesn’t load?
Try a hard refresh or different browser; if unresolved, contact support with a screenshot.

12. Can I remove AI scoring from a job?
Yes, contact support with the job ID.

Related Features & Next Steps

  • Screening Questions & Auto-Reject Flows:
    Use these to further automate and improve candidate filtering.

  • Job Description Optimization:
    Ensure your job postings are detailed and accurate for better AI matching.

  • Candidate Timeline & Activity Panel:
    Track all changes, feedback, and actions for transparency.

  • Role-Based Access:
    Understand who can view and provide feedback on AI scoring.

Recommended Articles:
- How Resume Match Works in PyjamaHR
- Configuring AI Interviews and Screening Criteria
- Troubleshooting Common Issues in PyjamaHR

Summary

Providing feedback on AI models and scoring in PyjamaHR is essential for maximizing the accuracy and value of your automated hiring process. By understanding how scores are generated, proactively reviewing and reporting issues, and following best practices, you ensure your team gets the most relevant, high-quality candidates—while helping PyjamaHR’s AI get smarter for everyone.

Still have questions?
Contact PyjamaHR support with your job ID, candidate details, and a clear description of your feedback or issue for prompt assistance.

Did this answer your question?