With the growth of the robotics field, it is expected that robots will increasingly become part of our everyday lives. Consequently, there is an emerging need for humans and robots to work together. Human-robot collaboration has become an important topic in various domains, such as home assistance and manufacturing. For effective collaboration, robots must be able to recognize human activity and plan, while determining which functions would be helpful to humans. The problem of recognizing human activity and plan, known as human activity and plan recognition (HAPR), is considered to be the main bottleneck for successful collaboration. HAPR becomes even more complex in the analysis of visual inputs, such as RGB-D images. This thesis addresses this bottleneck by investigating how to perform an efficient and accurate vision-based HAPR for fluent collaboration in real-world applications. The following limitations of state-of-the-art HAPR studies are examined in this thesis. First, although learning-based, model-free approaches are gaining significant attention owing to recent advances in deep learning, they require significant amount of training data. This makes recognition inefficient. Second, previous studies recognized human activity and plan separately and sequentially. They recognized human activity first and subsequently plan. Separate and sequential recognition cannot consider the plan context while recognizing human activity because the plan context is not available during activity recognition. However, the plan context provides useful information for activity recognition. Thus, separate and sequential recognition is inaccurate. We pose a fundamental question: Do humans share the same limitations when recognizing others' activity and plan? To answer this question, we introduce a novel problem called cognitive HAPR. Cognitive HAPR attempts to improve the HAPR system by adopting three ideas motivated by how cognitive humans perform HAPR. The first idea is to apply symbolic reasoning based on the preconditions-and-effects structure of activities, which humans understand well. For example, let us assume that a person is getting a bowl. It is intuitive to understand or assume that the person's hand must be empty, as a precondition of this activity, and the person would be holding a bowl as an effect of this activity. We propose that such intuitive preconditions-and-effects structure of activities provides valuable domain knowledge for HAPR. The second idea is the application of commonsense spatial knowledge with qualitative representations. Several cognitive science studies have shown that humans efficiently and effectively perceive their surroundings by abstracting the scene using qualitative representations. Qualitative representations are more compact and effective than quantitative data such as 6-D poses (i.e., x, y, z, roll, pitch, and yaw) of objects. We propose qualitative spatial representation (QSR), a representation framework that describes the spatial information of objects in a qualitative manner, as a good qualitative representation tool for HAPR. We effectively model complex predicates relevant to activities through QSR statements using intuitive commonsense knowledge. This modeling of predicates also provides valuable domain knowledge for HAPR. The third idea is the application of context-aware human activity recognition using a plan context. Several cognitive science studies have proven that humans recognize activity and plan as a combined framework, instead of recognizing them separately and sequentially. Humans employ the plan context when recognizing activity using the combined framework. We proposed a combined model for HAPR that captures the Bayesian theory of mind (BToM) from cognitive science. This thesis presents a cognitive HAPR system called cognitively motivated plan and activity estimation system (COMPASS) that achieves the three ideas. We evaluate COMPASS in a home care scenario, called the activities of daily life (ADL). The ADL scenario takes place in a household environment where a human and robot collaborate to complete daily tasks. We use the ADL scenario to demonstrate that COMPASS resolves the two limitations of previous HAPR studies. First, by using a model-based approach, COMPASS requires significantly less training data compared to the case using a learning-based approach. This makes COMPASS more efficient. The two ideas of applying symbolic reasoning based on the preconditions-and-effects structure of activities and commonsense spatial knowledge with qualitative representations provide good domain knowledge for model-based recognition that requires minimal modeling effort. Second, by using a combined framework, COMPASS can perform context-aware human activity recognition using the plan context. This makes COMPASS more accurate compared to the case using the sequential model.Ph.D.