Get help with GitHub Classroom

Set up autograding

When you create a new assignment, you can choose what autograding tests to run.

Tests run in a virtual Linux environment that comes with many popular tools pre-installed. See actions/virtual-environments for a complete list.

Add tests to a new assignment

To create a new assignment with autograding tests:

  1. Sign in to GitHub Classroom.
  2. Choose a classroom or create a new classroom for the assignment.
  3. Create a new assignment. Click New assignment, then click Individual assignment or Group assignment.
  4. Fill in the required information for the assignment.
  5. In the Optional settings section, click Enable autograding.
  6. Click Add test, then click a test type. Choose a language-specific test for an example or learn more about test types.
  7. Complete the details for the test type.
  8. Click Save test case.
  9. Add more tests, if needed.
  10. After you’ve completed any other assignment settings, click Create Assignment.

When students accept the assignment, the autograding tests will run after every push.

Add tests to an existing assignment

You can add tests to an existing assignment, but tests will only be applied to student repositories created after the assignment was changed. Existing student repositories will not be changed.

To add autograding tests to an existing assignment, follow these steps:

  1. Sign in to GitHub Classroom.
  2. Choose a classroom.
  3. Click on an assignment,then click Edit assignment.
  4. Fill in the required information for the assignment.
  5. In the Optional settings section, click Enable autograding.
  6. Click Add test, then click a test type. Choose a language-specific test for an example or learn more about test types below.
  7. Complete the details for the test type.
  8. Click Save test case.
  9. Add more tests, if needed.
  10. Click Update Assignment.

When students accept the assignment, the autograding tests will run after every push.

Test types

Autograding tests work by running a setup and test command in a GitHub Actions virtual environment. You don’t need to know how GitHub Actions work to use tests, but see actions/virtual-environments for a complete list of preinstalled software.

Input/Output

Input/Output tests run a setup command, a test command with standard input, and evaluates the test command’s output against an expected result.

Screenshot: Input/Output test case settings

Input/Output settings include:

  • Test name: the name of the test as it appears in the logs
  • Setup command: a command to run before tests, such as compilation or installation (for example, javac *.java)
  • Run command: the command to run the test. All standard output from this command is used to compare against the expected output.
  • Inputs: Standard input for run command
  • Expected Output: output to be compared to the actual output of the run command
  • Comparison: The type of comparison that should be done between expected outputs and actual outputs. Comparison types are:
    • Included: passes whenever the expected output appears anywhere in the actual output
    • Exact: passes whenever the expected output is completely identical to the actual output
    • Regex: passes if the regular expression in expected output matches against the actual output
  • Timeout: how long a test should run before being marked as a failure
  • Points: how many points this test is worth towards a total score

Run command and language tests

Run command tests run a setup command, a test command, and checks the exit status of the test command. Test commands that exit with 0 pass, while other exit codes fail.

Language-specific tests are run command tests with prefilled settings that you can use or customize. For example, the Run node test prefills the setup command with npm install and the test command with npm test.

Screenshot: Run command settings

Run command settings include:

  • Test name: the name of the test as it appears in the logs
  • Setup command: a command to run before tests, such as compilation or installation (for example, javac *.java)
  • Run command: the command to run the test. The exit code of this command determines whether the test passes or fails
  • Timeout: how long a test should run before being marked as a failure
  • Points: how many points this test is worth towards a total score