Always write code with the correct indentation. The problems are obvious...

dountil () {
    (( answer = $RANDOM % 1-10 ))
    until [[ $guess -eq answer ]]
    do
         read -p "Enter a number between 1 and 10:" guess
         if [[ $guess -ne answer ]]; then
             echo "Incorrect. Try again"
         elif [[ $guess -eq answer ]]; then
             echo "Correct! Want to play again?"
             read -p prompt
             if [[ $prompt = yes ]]
                 dountil
                 if [[ $prompt = no ]]
                 then
                     echo "Goodbye."
                 fi
}

At least two then, two fi and one done missing. And calling the function again from within, what is that?

On your level of coding you really should add only one or two commands at a time and then run the new version of the script. Just in order to easily catch syntax problems.

avoid recursion

there is no need for a function anyway

while true; do
    # ...
    if [[ $prompt != yes ]]; then
        break
    fi
done

make the funcion a loop

dountil () {
    while true; do
        # ...
        if [[ $prompt != yes ]]; then
            break
        fi
    done
}

put only the core code in the function

guess_number () {
    (( answer = $RANDOM % 1-10 ))
    read -p "Enter a number between 1 and 10: " guess
    if [[ $guess -ne answer ]]; then
        echo "Incorrect."
    else
        echo "Correct."
    fi
}

while true; do
    guess_number
    # ...
    if [[ $prompt != yes ]]; then
        break
    fi
done