I'm currently reading "The Design of Everyday Things" (thanks to @k8nowak for the recommendation!) and it's interesting to think about what the theory of product design has to say about instructional design.
For example, the author discusses how "learned helplessness" can result from poor design. A product whose functionality isn't discoverable, and that doesn't provide good feedback, will be frustrating to use, so users will likely give up after trying and failing a few times. Just as I was making the connection to teaching math in my mind, the author himself brought up math instruction as a common example of "taught helplessness": When math is presented as unintuitive and poor or misguided feedback is given, students are likely to just give up.