Gradescope_Autograder

Here is the link to the GitHub repo.

This is an example Python autograder that uses the framework layout from the gradescope example autograder and exports the testing to a node on a cluster which has a compatible kernel with the test programs and packages you write for your class.

• Problem: Assignment test code breaks if there are incompatibilities between the packages they use and the gradescope kernel.

• Solution: Rather than building a whole new container with the gradescope framework (which may not even run if there are incompatibilities), simply use ssh to run the tests on the custom container or VM on which the tests work and send the results to the gradescope container for publishing.

This solution offers great flexibility for all gradescope users, and especially for those using the OPE framework since the runtime container is already designed and configured on a cluster for running these tests. We want to provide as much ease of adoption to the OPE framework so that professors and general educators can start using OPE all over the world!

Architecture

Note that the architecture diagram references the NERC (New England Research Cloud), but you can set this up with any cloud you want.

Configuration

1. id_ed25519 (private key) and config files are for ssh configuration to connect to your VM on which the container runs. Configure these as needed.

2. Put the contents of your corresponding public key (i.e. $( cat id_ed25519.pub)) inside ~/.ssh/authorized_keys on your VM.

3. tests/ contains all your tests

4. real_run_autograder and real_setup.sh are called once the gradescope container uses ssh to log into your VM.

5. inside real_run_autograder a282 in the line CONTAINER_ID=$(docker run -dv $(pwd):/home/autograder_${EMAIL} a282) should be replaced with your docker image ID which we assume is already pulled onto the VM

6. zip.sh is a one line script to zip up your autograder and ignore any unecessary file (e.g. .git directory). We want to minimize the number of files we have to scp to our cluster for the sake of efficiency.

Example run

This framwork is designed to handle concurrent requests from different students on gradescope because upon copying state with scp to the VM, a new directory is created based on the student’s unique email ID, ~/autograder_${student_ID}, then subsequently deleted upon pushing the results back to the gradescope container.

Due to the naming of submission directories on the NERC container ~/autograder_${student_ID} rather than /autograder, gradescope_utils’s check_submitted_files fails since it expects the latter path to locate the submission. This wasn’t a big issue though and this test can be removed (it’s tests/test_files.py). All other tests worked. Here’s a screenshot of the VM on NERC to which the gradescope VM scps the state to and then starts a docker container to run the tests. At the end both the docker container and the directory are cleaned up (deleted) on the NERC VM.