Ari Bader-Natal August 6, 2010
We were lucky to have Daniel Furr join us this summer for the Grockit Graduate Research Internship program. Dan is a Ph.D. student in the University of California Berkeley’s Graduate School of Education, with a focus on Quantitative Methods and Evaluation.
At Grockit, Dan has been working on a variety of assessment-oriented projects, such as: on understanding how much data is needed to make use of an item response theory model in a Grockit network, on longitudinal models that help us refine our understanding of student improvement over time, and on extending our current models to incorporate the effect of collaboration, discussion, and repetition on a student’s probability of response accuracy. Understanding the effect of factors like these allows us to better select appropriate challenges for students in Grockit games.
About this internship, Dan writes:
Over the summer I focused on comparing approaches to estimating IRT models and experimenting with longitudinal models to measure user improvement over time. I compared estimations conducted with different R packages, with varying subsets of data, and ways of incorporating information on social interactions that occur alongside item responses. I used random item models to assess learning over time–time as calendar time, as item presentation order, and as discrete “sittings”. The people of Grockit are welcoming and innovative, and I feel fortunate to have had the chance to work with them and grapple with many interesting complexities.
I enjoyed working with Dan over the past few months, and wish him well on his return to academia in the Fall. Keep an eye on this blog for announcements of new assessment-oriented features on Grockit, several of which have benefited from Dan’s hard work this summer.
Ari Bader-Natal June 22, 2010
As you probably know, Grockit offers learners the choice of three different modes of study: individual practice, peer study groups, and instructor-led lessons. What you might not know (unless you’ve read about Research at Grockit) is that these three modes of learning draws on ongoing research in three corresponding subfields of study. Last week, I got up-to-speed on the state of the art in one of these areas, individual study, at the Tenth International Conference on Intelligent Tutoring Systems. I wanted to share a few highlights from the conference here.
On Day 1, I started by co-organizing a workshop with Erin Walker and Carolyn Penstein Rosé (both of Carnegie Mellon University) on Opportunities for intelligent and adaptive behavior in collaborative learning systems. It was a great opportunity to bring together a group of people actively pursuing research at the intersection of intelligent systems and collaborative learning. Here is a snippet from the workshop website.
Intelligent tutoring systems are generally designed to tailor instruction to the individual student, but this does not mean that ITS-guided learning must necessarily be a solitary activity. A variety of recent systems have demonstrated ways in which an adaptive learning environment can incorporate and benefit from the presence of multiple learners. Similarly, students using computer-supported collaborative learning systems have been shown to benefit from the introduction of adaptive support that targets the collaboration. In this workshop, we invite discussion and seek to explore ways in which the combination of collaborative and intelligent aspects of a system can benefit the learner by creating a more productive learning environment.
If you’re interested in learning more about this workshop, you can find the presenter list, rapid-fire slides, and full proceedings on the workshop website.
As the week continued, I got a chance to hear presentations on a wide variety of research projects. A few of the talks that I found interesting this year touched on these topics: deciding if and when to provide hints, identifying the moment of learning from patterns in data, incorporating teachers into the process of designing a system, automatically generating hints from patterns in past data, understanding the ways in which human tutors are adaptive, incorporating dialogue agents in peer collaborations, and modeling learning gains over time. You can see the full spectrum of work presented in the conference proceedings index.
Grockit was proud to be one of the sponsors of this year’s Intelligent Tutoring Systems conference.
Ari Bader-Natal June 21, 2010
Two weeks ago, Grockit participated in the Games+Learning+Society conference. On the second day of the conference, I joined Jeramy Gatza, Curriculum Innovation Specialist on the Research and Discovery Team at Florida Virtual School, to discuss a recent project in which eight Algebra I classes at Florida Virtual School piloted the use of Grockit as a supplement to their standard course curriculum. The presentation, entitled Collaborative Learning Games in the Virtual Classroom: Piloting Grockit at Florida Virtual School, took an interactive form. Each participant had a laptop, which allowed us to ground our discussion of the social, motivational, and collaborative aspects of the platform in first-hand experience. Here’s a bit more about the discussion, taken from the presentation abstract:
A virtual school can offer a student the ability to complete a course on their own schedule, from any location. The challenge in providing a flexible, individualized learning environment is that students may feel disconnected from each other, and can miss the opportunity to learn from interactions with their peers. Multi-player online learning games may hold a solution. By providing a venue for learners to connect and interact, these games can extend the benefits of collaborative learning opportunities to the geographically-dispersed students in a virtual school.
…Our goal for this workshop is to share with participants both an intuitive sense and data-grounded evidence about how multi-player learning games, like those in Grockit, can help connect, motivate, and engage students who are geographically and socially isolated. The workshop will conclude with a group-wide discussion of other experiences with, and opportunities for, using game-based collaborations as a way to connect learners across the web.
Two days later, Jeramy and I again presented together, this time to discuss our work with a group of educators as part of the GLS Educator Symposium. This presentation was nominally grounded in work that I’ve published on how we go about deciding between synchronous and asynchronous interactions for the various components of Grockit’s collaborative learning activities (for more info, see the source paper and the asynchronous discussion thread on this work). The real heart of the session, however, was devoted to a series of interesting questions raised by participating educators about the real-world task of incorporating Grockit (or a system like it) into the classroom setting. Jeramy Gatza provided a very interesting perspective, responding to several questions based on his experiences using Grockit at Florida Virtual School.
While I was at the Games+Learning+Society conference, I saw/heard/participated in a number of excellent talks, tutorials, demos, and keynotes. It was exciting to see the incredible variety of ways in which researchers and educators have been incorporating games into learning and learning into games. I’m excited to bring these ideas back to Grockit, and use them as inspiration for our own internal brainstorming sessions on games and learning. Stay tuned for new announcements on this front…
Ari Bader-Natal May 18, 2010
When you play a solo game in Grockit, the questions that you see are selected based on your recent performance (Grockit gets more adaptive). Our goal in doing this is to offer you challenges that are neither too easy nor too hard for you, in order to provide better opportunities for learning.
Our concept of how difficult you’ll find a question is based on our estimate of the likelihood that you will answer the question correctly. If, for instance, we estimate that there is a 90% chance that you will answer the question correctly, we think that you’ll consider that question to be pretty easy. Others may find the same question difficult, but you’ll probably find it to be straightforward.
Today’s question: Does a subjective measure of difficulty have any relationship with the length of time it takes a student to solve the problem?
Think for a moment about your own problem-solving experiences. Do you think that you spend more time on a harder question than on an easier question? Less time?
We record a lot of interesting data when students answer questions on Grockit, on some of this data can help us answer the question at hand. Below are two plots of the average time taken by Grockit students to answer GMAT Quantitative and ACT Math questions, as a function of how likely we believe the student to correctly solve the question. When you look at the plots, the first thing to note is that the relationship between subjective difficulty and response time isn’t linear.
These plots indicates that, on average, students are spending less time answering each question that are subjectively very easy or very difficult than on the questions that are in between. Do you find this surprising? Interesting? Does this reflect your own experiences studying? How would you interpret these plots?
Ari Bader-Natal April 26, 2010
We’re really excited to share some of the research that we’ve incorporated into making Grockit the learning tool that it is. We just added a new Research at Grockit page to the site, where you can learn about how Grockit continues to participate in, draw on, and contribute to the research community.
Along these lines, I’ll be co-organizing a workshop on Opportunities for intelligent and adaptive behavior in collaborative learning systems at the Tenth International Conference on Intelligent Tutoring Systems in mid-June. The workshop website is now online, and accepted papers will be posted in June.
Keep an eye on the Research at Grockit page, as we’ll be adding more to it over the next few weeks…
Ari Bader-Natal February 3, 2010
I’m happy to announce that Grockit will be offering its paid summer research internship program for the summer of 2010. This is the second year that we’re doing this (thanks again, Angela!), and I think it’s a great opportunity for doctoral students to apply their own research experience to a system that a large (and growing) community of learners uses everyday. It’s worth mentioning that Grockit has a large and interesting set of educational data, a variety of research interests, a very talented team, and a fantastic work environment. I just posted details about this program (with an application form) on the 2010 Summer Research Internship, and I encourage you to check it out.
I wanted to share a few thoughts on why we’re offering this, what we have in mind for the program, and why you (or perhaps someone you know) should consider applying.
Grockit, as you may know, is a San Francisco-based web startup building a platform for — and a community around — synchronous collaborative learning games. We strive to provide our growing global network of learners with a smart platform informed by peer assistance and adaptive support. Towards this end, we’re constantly exploring new ways to support collaborative learning online, and we’re frequently examining and applying techniques for analyzing the learning data that we’ve been collecting. One reason that we’re offering this program is to expand on the ways in which we pursue these goals.
Two of the challenges in studying computational systems for peer learning — both of which I faced in completing my own graduate work — is that these systems can take quite some time to build, and it can often take even longer to cultivate a sufficiently large community of participating learners. As a result, the time required to get from hypothesis to data analysis can be (or at least can feel) quite long. At Grockit, we’ve been making good progress with regards to both challenges, and hope that this internship will provide an enterprising graduate student with the opportunity to speed up this process for their own research questions.
In addition to the research opportunity, we’re offering a program stipend, an accommodation stipend, and a travel stipend. You’ll also get a healthy breakfast and lunch cooked in the office every weekday and the chance to spend your summer in vibrant San Francisco. So if you are a doctoral student studying in a university in the United States and interested in applying for a summer research position with us, I’d encourage you to submit an application.
The deadline is March 1, 2010, and you can apply today.
Ari Bader-Natal December 19, 2009
Many of the approaches that we draw on at Grockit — adaptive web-based learning, study group formation, embedded assessment, computer support for collaborative learning, educational applications of data mining and social network analysis, learning in games, and the benefits of practice, feedback, spacing, teaching, discussing — are no longer exclusively discussed in academic journals and conference proceedings. Several of the researchers and practitioners in these areas have hit the blogosphere, where ideas are informally sketched, quickly shared, and freely accessed. Here are 25 such blogs that are currently in my newsreader:
OLDaily
Learning Games Network
Digital Media and Learning
Epistemic Games
Clark Aldrich On Simulations and Serious Games
Confessions of an Aca-Fan
Raph Koster’s website
Learnlets
Will at Work Learning
Informal Learning Blog
CSCL Community
Connectivism: networked and social learning
P2P Foundation
Iterating Towards Openness
Sharing Nicely
TravelinEdMan
Innovate journal
Learning Sciences and Educational Technology
International Review of Research in Open and Distance Learning
Infinite Thinking Machine
Online Sapiens
Elearnspace
Educational Technology News
Fortnightly Mailing
Virtual Canuck
Have some that we’re missing? Leave a link, and we’ll check it out.
Ari Bader-Natal November 24, 2009
As promised in my previous post about the E-Learn conference, I wanted to share a few thoughts sparked by Terry Anderson’s keynote. The slide that caught my eye — and the one that I’ve been thinking about the most since the conference — mapped out three different types of “many” in social learning environments: the group, the network, and the collective. Based on a set of papers and blog posts by Jon Dron and Terry Anderson over the past few years, the model describes the characteristics of each of these social learning contexts. Here’s a composite reproduction of their illustrations, describing some of the key characteristics of each type of social software:
Dron & Anderson: Groups, Networks, and Collectives
While I’m several years late to the discussion, I think that it’s still worth mentioning why I find it so interesting. At Grockit, we’ve also been thinking about contexts for learning, albeit in different terms: we’ve been thinking in terms of supporting learning from experts, learning with peers, and learning alone. I can’t help but see parallels between these three learning contexts and that of Dron and Anderson’s groups, networks, and collectives.
Grockit: Learning from experts, with peers, and alone
Notice the similarities:
- Expert-led classes in Grockit resemble traditional classroom groups.
- Peer-driven sessions are based on an ever-changing network of participants.
- Self-directed solo learners benefit from the collective behavior of all past Grockit interactions.
Each of these contexts has advantages and disadvantages, so we’ve opted to support all three and leave it up to the student to choose the way in which they wish to interact with others. We’ve noticed that for many, this choice changes from day to day, or even over the course of a single sitting. The choice is yours: How do you want to learn from others today?
thomas lotze November 20, 2009
That’s the question FlowingData posed last week. Nathan Yau examined the average SAT scores and high school class sizes for every state, using a parallel coordinates plot. From his analysis, it looked like having smaller classes (fewer students per teacher) resulted in higher SAT scores. Makes sense, right? Fewer students means more attention from the teacher, which should result in better scores.
But this wasn’t the whole story. A lot of people commented on the original post, suggesting some other factors that weren’t being taken into account. The main effect was that of the ACT — because many states have students take the ACT rather than the SAT, these states have only a few students taking the SAT. In this case, the only students taking the SAT might be the ones who expect to do well (or those students applying to colleges which require it).
I’ve put up an interactive visualization of the education data by state, which you can use to explore the different relationships. One clear effect is that having a low percent of students taking the SAT does result in higher average SAT scores for that state. In the graph below, the y-axis shows the average composite SAT score for the state, while the color shows the percent of students who take the SAT. The highest scoring states all tend to have low percentages of students taking the SAT (the dots are more blue), while the states where more students take the SAT (the dots are more yellow) are on the lower part of the plot.
The question of class size and its effect on education is still open. The investigation is continuing on the FlowingData forums, where anyone can contribute their own visualization. I’ve put mine up there, but I’m looking forward to seeing what everyone else comes up with.
Here at Grockit, we’re interested in this topic because using study groups is one way students learn in Grockit. Understanding the impact the size of a group has on learning helps us improve our learning environment.
Ari Bader-Natal November 9, 2009
Last week, I had the opportunity to attend the E-Learn 2009 conference, both to present some of our ongoing work at Grockit and to learn from others in the field. In this post, I’ll describe a bit about our contribution, and in a companion post, I’ll share some highlights from the talks that I heard.
The slides above were prepared to accompany my paper, Interaction synchronicity in web-based collaborative learning systems. Here’s a summary:
In building a web-based platform to support live collaborative learning online, we’ve faced (and continue to face) a variety of technical challenges. One significant challenge has been the mismatch between what the web was designed for — namely, a network of linked documents — and what we’re using it for — a network of live collaborations. Different learning systems deal with this mismatch in different ways. Some systems stick with the web-native document-oriented model, and support asynchronous (i.e. different time) collaborative interactions around “content”, “educational resources”, or “learning objects”. Other systems support live, synchronous (i.e. same time) collaborative interactions by relying on better-suited network protocols, but do not run within a browser. Finally, a growing class of systems, including Grockit, have chosen to engineer a way to support live user interactions on the Web. For us, the motivation for engineering a solution is to enable you to use Grockit from the comfort of most any web browser on most any computer. If you’ve participated in one of our learning games / study groups, you know how convenient this can be.
But just because the “hard” work is done (read: supporting synchronous interactions on the web) doesn’t mean that we necessarily have to pass on doing the “easier” work (read: supporting asynchronous interactions.) If you’ve spent time reviewing your Grockit study sessions and reading through the explanations and discussion threads, you’d probably agree that this, too, can be quite valuable. Here’s where I think that it gets interesting: If we can support both synchronous and asynchronous interactions among learners, how do we know which to use? Which interactions should to be synchronous? Which should be asynchronous? Which should be a mix? What should that mix be?
There are no clear correct (or incorrect) answers here, so it’s been an interesting challenge trying to figure out what seems to work best. In the paper, I share a few lessons that we’ve learned in the course of grappling with this, including:
- implications of question complexity
- implications of activity visibility
- implications of continuous communication
- implications for discussion repurposing
- implications of group size on discussion dynamics
- implications of community size on group formation
If you’re interested in learning more, you can read the full paper.
