Integrated Cognitive Systems, Computational Cognitive Modeling, Cognitive Engineering, and much more. Interested in basic and applied research that leads to understanding the interplay of cognition, perception, and action in routine interactive behavior.

The cognitive systems that we seek to understand are humans interacting with complex tasks in interactive task environments. Our approach includes traditional experimental paradigms as well as video games and other complex tasks, which we treat as experimental paradigms.

Our work is as methodologically rigorous as possible, given the range of behaviors and tasks we attempt to study. For example, in our studies of video game players http://homepages.rpi.edu/~grayw/pubs/papers/2014/141002-KogWis-v4-archival.pdf , after the first move, everyone (potentially) has a different task. However, borrowing and adapting from developmental psychology the tradition of microgenetic analysis (e.g., Siegler, 1991), we capture and analyze system data, eye data (fixations and saccades), and response time data with precision measured in milliseconds.

The goal of understanding complex interactive behavior, often requires us to turn to traditional experimental psychology tasks such as task switching, nBack, and AX-CPT in order to test and resolve theoretical issues raised by our more complex task environments.

Our approach is informed by cognitive neuroscience, cognitive theory, and by detailed analyses of the world in which these are embedded (Anderson, 1991; Gray, Neth, & Schoelles, 2005). Understanding of mind in world requires understanding task structure, task payoffs (what the human actor is trying to achieve), as well as understanding the processing architecture (e.g., Altmann & Gray, 2008; Gray, Sims, Fu, & Schoelles, 2006; Howes, Lewis, & Vera, 2009; and Lewis, Shvartsman, & Singh (2013).

[Overview last revised: 2014.10.28]

Altmann, E. M. and Gray, W. D. (2008). An integrated model of cognitive control in task switching. Psychological Review, 115(3), pp. 602–639.

Anderson, J. R. (1991). Is human cognition adaptive? Behavioral and Brain Sciences, 14(3), pp. 471–517.

Gray, W. D., Neth, H., and Schoelles, M. J. (2006). The functional task environment. In Kramer, A. F., Wiegman, D. A., and Kirlik, A., editors, Attention: From theory to practice, pp. 100–118. Oxford University Press, New York.

Gray, W. D., Sims, C. R., Fu, W.-T., and Schoelles, M. J. (2006). The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review, 113(3), pp. 461–482.

Howes, A., Lewis, R. L., and Vera, A. (2009). Rational Adaptation Under Task and Processing Constraints: Implications for Testing Theories of Cognition and Action. Psychological Review, 116(4), pp. 717–751.

Lewis, R. L., Shvartsman, M., and Singh, S. (2013). The adaptive nature of eye movements in linguistic tasks: How payoff and architecture shape speed-accuracy trade-offs. Topics in Cognitive Science, 5(3), 581–610.

Siegler, R. S. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46(6), 606–620.

In Search Of

(a few good grad students and post-docs)

2014 September Addendum

My style in arranging these pages has changed over the years. Although what is on the rest of this page is still relevant and (I hope) interesting, here I present two convenient (to me) summaries of my work that were written about parts of what the lab is doing now! Below this you will find my 2009 update and below that, more general things and a partial index-by-topic of work done in the lab from around 1998 to 2008.


We are studying the acquisition and deployment of extreme expertise during the real-time interaction of a single human with complex, dynamic decision environments. Our dilemma is that people who have the specific skills we wish to generalize to (such as helicopter piloting, laparoscopic surgery, and air traffic control) are very rare in the college population and too expensive to bring into our lab. Our solution has been to study expert and novice video game players. Our approach takes the position that Cognitive Science has been overly fixated on isolating small components of individual cognition. That approach runs the danger of \emph{overfitting theories to paradigms}. Our way out of this dilemma is to bring together (a) powerful computational models, (b) machine learning techniques, and (c) microanalysis techniques that integrate analyses of cognitive, perceptual, and action data collected from extreme performers to develop, test, and extend cognitive theory.

Since our January 2013 start, we have built our experimental paradigm, collected naturalistic and laboratory data, published journal and conference papers, won Rensselaer Undergraduate research prizes, developed ``single-piece optimizers'' (SPOs, i.e., machine learning systems), compared machine performers to human performers, and begun analyzing eye and behavioral data from two 6hr human studies. Our tasks have been the games of Tetris and Space Fortress. Future plan include (a) using our SPOs to tutor piece-by-piece placement, (b) developing integrated cognitive models that account for cognition, action, and perception, and (c) continued exploration of the differences between good players and extreme experts in Tetris and Space Fortress.

Games such as Tetris and Space Fortress are often dismissed as ``merely requiring reflex behavior.'' However, with an estimated total number of board configurations of 299 (approx. 8 followed by 59 zeros), Tetris cannot be ``merely reflect behavior.'' Our preliminary analyses show complex goal hierarchies, dynamic ``two-piece'' plans that are updated after every episode, sophisticated use of subgoaling, and the gradual adaptation of strategies and plans as the speed of play increases. These are very sophisticated, human strategies, beyond our current capability to model, and are challenging topic for the study of the Elements of Extreme Expertise.


A challenging problem for current cognitive science would be to take an innovative display, such in the following figure, and generate zero-parameter predictions as to how long it would take an average human to find and extract the answer to a given query.

The eye saccades many times per second. Is it saccading to and fixating on the important information or is it being distracted by visually salient nodes and edges that have low semantic relevance for the query being pursued? When the analyst finds a few anomalous nodes, she imposes semantic meaning on them that they did not have for her before she found them; for this scenario, she has identified them as potential indicators of the anomalous pattern of interaction that she is seeking to uncover. Does the visual saliency of those nodes facilitate or inhibit her search for others? Are the important nodes visually similar to each other and dissimilar to others? If not, can the operator somehow mark the nodes so that she can create subsets of nodes which are visually similar to each other but visually dissimilar from the rest; thereby, creating an onscreen record of her search and of her currently favored hypotheses? Once discovered are these nodes so similar to others that they get ``lost'' every time she glances away from them to search for others? Or do they get ``lost'' every time she clicks on another node because the pattern of saliency has changed (as per the left vs right versions of the figure)?

Views from the CWL

2009 November Addendum

Every website in the world is out of date, including this one. Rather than updating my research interest page today (2009-11-23) I am writing this note instead. The most recent picture of my research interests is contained in the papers (mine and my student's) which are online at the Online Publications tab to your left. I am a bit slow getting the 2009 ones up there and am slower still in updating all of the research interest pages. As of today, my group, the CogWorks Laboratory, has research funding from NASA, ONR, AFOSR, and the Army Research Laboratory.

The theme to our work is the Cognitive Science of Natural Interaction with a focus on the integration of perception, motor, and cognitive operations at the 1/3 to 3 s timescale. This work has focused on human-technology, human-information, and (most recently) human-human interactions. We see the human-human interactions as part of what my colleague Ron Sun refers to as Cognitive Social Science; namely, an approach to traditional social psychology type questions that is rooted in cognitive science theory, modeling approaches, and methodologies. Recent work in human-technology interaction includes the study of fast-paced action games. In our premier gaming project we have collected EEG, eye data, and behavioral data from players over 31 hrs of play (per player), and are building computational cognitive models of expert game play as a means to understanding the control problems posed by the interleaving of goal-directed, cognitive, perceptual, and motor processes in real-time interactive behavior. We are also building models of airline pilots who get lost or confused while taxingly on the ground from the runway to the gate. Oh, we are also intent on solving the cognitive control of multitasking, interruptions, errors, and other common human behaviors which turn out to be incredibly hard to understand well enough to model. We plan to be at this for a long, long time. & yes. I do plan on taking new graduate students next fall. Looking with people with good computer science and mathematics skills and an intense interest in cognitive science! –Wayne–

Research Statement (in drastic need of updating but still relevant)

Research Slogan

milliseconds matter

Favorite Research Related Quotes

"Psychology has arrived at the point of unified theories of cognition--theories that gain their power by positing a single system of mechanisms that operate together to produce the full range of human cognition. I do not say they are here. But they are within reach and we should strive to attain them." Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.

“There is nothing so useful as a good theory.” Lewin, K. (1951). Field theory in social science. New York: Harper Row.

“Nothing drives basic science better than a good applied problem.” Newell, A., & Card, S. K. (1985). The prospects for psychological science in human-computer interaction. Human-Computer Interaction, 1(3), 209-242.

Research Interests

My research interests can be classifed into topics, domains, or methods. Topics are basic cognitive science issues such as human error, serial attention, cognitive workload, and interactive behavior. In my work on these topics I distinguish between basic versus applied research. When the topic is studied in the context of a simple laboratory experiment the work is basic research. When it is studied in the context of a real-world domain, the work is applied research. Domains that I have studied include the submarine Approach Officers' attempts to locate enemy submarines hiding in deep water, telephone operators, Lisp programmers, usability, and human-computer interaction.Whether it is basic or applied, different topics require the use of different research methods. Methods that I have used include GOMS, ACT-R, ACT-R/PM, action protocol analysis, verbal protocol analysis, and simulated task environments.

Topics, domains, and methods are obviously intertwined. Below I have tried to put together some pointers to parts of my work that are relevant to each. (However, for something that approaches a coherent world-view see my Research Statement.)

In Search Of

(a few good grad students and post-docs)
Images and Design © Robyn Gray 2008

Last changed: 2014-10-28 wdg