I think your question is more nuanced than you realize because there are a couple of critical anachronisms at play:
- In the 1980s, even sophisticated computer hardware like the Amiga was simple enough for a competent assembly or C programmer to understand most of the hardware features available, how they worked, and how to program them directly for maximum benefit of their program. It was much like what we call "embedded programming" nowadays.
- For the Amiga, and its contemporaries like the Mac and Atari ST, there was a necessity to provide highly reusable routines, usually in ROM, that served both as a simple hardware abstraction for the programmer and as a means of assuring compatibility as underlying hardware details were iterated for the platform.
Because of #1, you should not under-estimate the potential that programmer's avoided using standard Amiga toolbox routines, present in the Kickstart ROM, just out of laziness. If they had studied the hardware enough, why bother to also study the "approved" way of accessing it. They could simply disable the Exec and device drivers, run their program single-tasking, and control the hardware directly. The only downside being their program might break in the presence of new or non-standard hardware.
However, some programmers would have embraced the effort that went into the ROM Kernel Routines, and would have learned them so they could confidently reuse them in many programs and avoid the breakage of hardware changes.
They might have still done their own custom booter (but using trackdisk.device), and they might have still disabled Exec (using the approved API for that) and run single-tasking. Then they might take advantage of the graphics.library to setup the display in a way that would safely work with the next Amiga hardware revision. And they might also use dos.library for saved games or for making their software HDD compatible. In that case, they are using the "OS", just not all of it. They are choosing the parts that suit their application.
I think this second approach was way more common than you indicate in the question. The ROM Kernel was actually pretty popular with Amiga programmers and pretty widely used for all programs, including games. It was also flexible enough that you could use only what made sense for your program. You did not have to bypass these sensible parts in order to get adequate performance. If you did, chances are high it was just out of laziness.