research | Grockit

Machine Learning for Student Achievement

By thomas lotze posted November 29, 2011

We at Grockit have just posted a new educational machine learning competition focused on analyzing student performance: http://www.kaggle.com/c/WhatDoYouKnow. The competition is hosted on Kaggle, which has rapidly become an excellent site for hosting data mining and forecasting competitions. Our sponsored competition revolves around analyzing students’ past study habits and predicting whether they can correctly answer future practice questions. Like Netflix did for movies, we’re trying to improve on education industry standards for adaptive learning algorithms.

I’m very excited that we’re reaching out to the machine learning community, because I’m tremendously excited about the potential for technology to change education. I think leveraging the data we can collect is a huge part of that. I honestly believe that by improving assessment, we can improve the way people learn. By knowing what areas you’re strong and weak in, you can better know when you’re at a suitable level and how to direct your future studying. We want to be able to help students direct their studying — and eventually, improve student testing itself. There’s a lot more to knowing your level than simply what questions you got right or wrong, and we want to take advantage of that as much as possible.

This is a manifestation of the Grockit value of putting the learner first. We’re doing this purely to try and improve the way we help students study. We’re putting our money where are mouth is, too, with cash prizes will also be awarded to the top three finishers. The competition runs through February. Details on the data set, the prizes, and the competition are all available at:

http://www.kaggle.com/c/WhatDoYouKnow

Thanks for reading, and we hope you check it out!

No Comments »

Answering the question that you’re about to ask

By Ari Bader-Natal posted October 17, 2011

We’re happy to announce the newest addition to the Grockit social learning platform: Grockit Answers, a tool for hosting timely Q&A discussions around web videos. We started by designing this for our own videos, but quickly decided that it could have more impact if we expanded our scope a bit. So as of today, Grockit Answers works with any video hosted on YouTube or Vimeo.

There’s a wealth of educational videos available online, but the primary mode of interaction among viewers — short-form commenting — doesn’t do much to enhance or facilitate learning. When you’re trying to grok a challenging lecture from an AI course at Stanford or an open talk at MIT, you need more than comments. You need a way to ask a question about a particular point that confused you, or ask for a clarification or maybe a useful reference on something mentioned in the video. You really need a Q&A site, and you need to be able to attach each of your questions to a particular time in the video. You need Grockit Answers. Here’s a quick video clip (live Q&A discussion here). Feel free to ask questions about this video on Grockit Answers.

Built on Popcorn.js, a fantastic new HTML5 media framework for better connecting web videos to web pages, Grockit Answers displays each question and its answers at the exact point in the video that they are most relevant. As a viewer, some of the things that confuse you are are likely to be the same as what confused others before you. In a time-anchored Q&A on Grockit Answers, just as the question begins to form in your mind, it may magically pop up on the screen, along with a few great answers.

For the educators out there, I’m happy to say that we developed a number of additional features with you in mind. You can create a Q&A page on a video that only you and your students have access to, and you can moderate the conversation to get additional controls and deeper insight into what’s happening. Finally, and most exciting, you can add these Q&A pages to your own course websites (example), easily replacing links to YouTube or Vimeo videos with your moderated discussions. Feel free to contact us at answers@grockit.com with any questions about this.

Start a Question & Answer page for any video on YouTube. Share it with your classroom or share it with your friends. And please, let us know what you think.

3 Comments »

Thoughts from the KIPP Charter School Annual Summit

By david marino posted August 11, 2011

Rusty Greiff is Grockit’s Chief Strategy and Corporate Development Officer and a resident of the greater Washington DC area. Rusty has an extensive background in politics and education.

My 8-year old son is a LEGO junky. He goes through three basic steps to LEGO greatness:

1) he shuts the door to his room

2) he pours the 500 pieces of LEGOs on the floor

3) he lets it rip.

Skyscrapers, spaceships, laser-rovers; massive and intricate structures are built – most of the time with his brother and best buddy next door. Often, I will peer through the crack in the door to assess their progress. Rarely do they know where their LEGO journey will take them, but they are comfortable in leaping ahead together, adjusting the pieces, and brainstorming until the creation is complete, often ending in a multi-player “high five”.

Last week, I played and learned with my own set of LEGOs along with some of the most talented educators in the country at the KIPP Charter School’s Annual Summit in Nashville, TN. Every year, KIPP brings its teachers, directors, funders, experts in the field and students together to discuss best practices. Alongside KIPP’s finest teachers, administrators and students, I led a 90-minute session, LEGOs in hand, on collaborative and social learning. The LEGO exercise and talk was meant to help participants better understand the impact of Grockit’s adaptive and collaborative platform as it is applied to KIPP schools. Currently, Grockit and KIPP are partnering to connect California KIPP students in peer-peer SAT/ACT test prep, with the goal of mastering test prep content and improving scores through social learning.

Not surprisingly, bringing people together (with a massive box of LEGOs) with different learning styles, specific skill sets and expertise, can create a dynamic experience that translates into some beautiful results.

By way of examples, Grace, a KIPP teacher from Washington DC preferred working in a team of 2 to build the foundation while her colleague Jennifer created a three-story roof garden to attach to Grace’s structure. A KIPP teacher from Newark encouraged her team to strategize first, drawing elaborate plans before finding color-coordinated LEGOs to complete the building. One AP teacher demanded of his team, “Just have fun” and attacked the LEGOs, building a killer “car-house” that of course could fly.

The LEGO exercise and session was an easy demonstration of existing academic research suggesting that by engaging learners through social games, collaborative problem solving and peer-to-peer studying, companies like Grockit are helping learners master subjects and achieve higher test scores. Literally, this is what we do at Grockit every day for our hundreds-of-thousands of students.

No Comments »

An Experiment in Group Study

By thomas lotze posted May 9, 2011

Group study is very important to us at Grockit. In fact, it’s something we’re staking our name and our business on: the notion that studying together is better than studying alone. While we’re not alone in believing that group study is important and have heard plenty of anecdotal stories about how people like studying with others, we needed some solid experimental evidence. We believe that it’s critical to test and validate our assumptions, especially assumptions about the core learning experiences in Grockit. We wanted to see whether group study actually impacted people’s studying: did they stay on task longer, study more, or study more effectively? Looking at how people study on Grockit, the answer to all of these seems to be yes. However, because we let learners direct their own studying on Grockit, we can’t separate the effect of studying in a group from a person’s choosing to study in a group, which might show biases according to the learner’s ability or intent. So when we couldn’t find any studies which looked at the sort of group study we’re providing, we decided to create our own.

The experiment is fairly straightforward. We randomly selected a portion of newly arriving Grockit learners and put them into the testing group, giving them a slightly modified Grockit experience. Instead of letting people select whether to study in a group or alone, we gave them a single button: “Study Now”. Each time they chose to study, we’d look and see whether there were other people studying the same thing. If there were, we’d treat this as an experimental case that we could test and flip a coin to see what study condition this would go in. Half the time, we put the learner into a solo game, studying on their own; the other half, we put the learner into a group game, studying with other people.

We found three key results supporting the hypothesis that group study is more engaging and effective for learners:

We found that learners studied longer when they were in a group study session than when they were alone. Studies have shown that time on task is a key predictor of study effectiveness. It’s a simple correlation: studying longer is more effective. By providing a more engaging study experience, learners have more fun and spend longer studying. On average, learners spent 3x as long studying when they were in a group study session as when they played alone. The boxplot is even more striking: you can see that learners spend more time studying in group games across the board. (For my fellow stats geeks out there, the Mann-Whitney-Wilcoxon Rank-sum test results in Z = 8.3883, with a p-value < 2.2e-16. That's pretty irrefutable, as statistics go.)
We found that learners studied over 2x as many questions when they were in a group study session as when they were alone. This was important because we wanted to make sure that studying with others wasn’t just making people slower to answer questions. But learners study more than twice as many questions, on average, when in group study than when they study alone.
We found that learners were more likely to answer questions correctly when they were in a group study session than when they were alone. It could have been that learners were simply goofing off in group study, and that they were socializing rather than seriously paying attention to the material. But in fact, the opposite was the case with Grockit learners, where the social pressure is to focus on the material. Learners in group games actually answered correctly more often than when they were playing solo.

We want to find the most effective ways to help people learn, and that requires making sure the results are valid and verifiable. We’re working on a peer-reviewed paper to provide full results and data from our experiment; if you want access in the meantime, please write to research@grockit.com. Grockit was founded on the belief that we could come up with a better way to help people study, and that a key part of doing that was helping learners teach and learn from each other. We’re glad to have some strong evidence to support this belief, and will continue to test our theories to build an even better system.

4 Comments »

Grockit is a finalist in the Next Generation Learning Challenges

By Ari Bader-Natal posted April 14, 2011

I’m happy to announce that Grockit is a finalist in the second wave of Next Generation Learning Challenges. I’d like to share a bit about this project and why we think it’s important. Here’s the project summary:

The Virtual Study Hall: Making Personalized Learning Collaborative

Virtual schools let students learn from any place at their own pace. The challenge in this individualized learning is that without a synchronized curriculum and shared venue, students can feel disconnected from their peers and alone in their studies. As online learning options are increasingly available at the secondary and post-secondary level, the issues that virtual high schools face today are a preview of challenges to come with this new wave in technology-enabled learning. While the move towards personalized learning appears to necessitate a move away from social/collaborative learning, this need not be the case. Grockit has developed a web-based learning platform that bridges this gap, with a collaborative learning platform that creates networks for self-paced learners. Building on an existing relationship with Florida Virtual School, we will demonstrate and measure how this platform can increase learning gains by connecting and engaging thousands of students enrolled in 7th-9th grade level math courses online.

This project addresses a stumbling block that many technology-based efforts for individualized learning either have faced or will face soon: a large-scale shift towards truly personalized learning in a school fragments a single cohort of learners into single-learner cohorts. For online programs and courses, this fragmentation can negatively impact student engagement and completion rates. In building a system to be both collaborative and adaptive, Grockit has grappled with this issue for several years, and has now identified several solutions to enable both approaches to be supported simultaneously. For courses offered entirely online, restoring the opportunity for peer-to-peer collaborative learning can be quite powerful. By building a technology-based learning environment as a social construct from the bottom-up, students can reap the benefits of studying within a community of learners — a sense of connection to other learners, informal peer-to-peer assistance and motivation, and a social context for studying.

We’re looking forward to the opportunity to implement and evaluate this approach on a large scale at Florida Virtual School, which is the largest state-led public virtual school in the country. For this project, we’ll track and quantify the impact of incorporating Grockit’s core platform into 7th-9th grade level math courses on student engagement and learning outcomes, and will share and publish our findings with the larger community of researchers and educators interested in improving online learning.

I’m happy to say that Grockit is quite open about our ongoing research efforts, including our peer-reviewed publications, assorted research collaborations, current domain applications, and underlying pedagogical foundations. If you’re interested in learning more, check out our overview on the Methodologies and Technologies of the Grockit Learning Platform. If you work with a school (traditional or virtual) that may be interested in partnering with Grockit, or if you are an education researcher or learning systems designer interested in discussing potential research collaborations, please let me know and I’ll be in touch.

I’d encourage you to check out the full list of NGLC Wave II finalists. Lots of exciting work happening here!

No Comments »

Thank you, Dan!

By Ari Bader-Natal posted August 6, 2010

We were lucky to have Daniel Furr join us this summer for the Grockit Graduate Research Internship program. Dan is a Ph.D. student in the University of California Berkeley’s Graduate School of Education, with a focus on Quantitative Methods and Evaluation.

At Grockit, Dan has been working on a variety of assessment-oriented projects, such as: on understanding how much data is needed to make use of an item response theory model in a Grockit network, on longitudinal models that help us refine our understanding of student improvement over time, and on extending our current models to incorporate the effect of collaboration, discussion, and repetition on a student’s probability of response accuracy. Understanding the effect of factors like these allows us to better select appropriate challenges for students in Grockit games.

About this internship, Dan writes:

Over the summer I focused on comparing approaches to estimating IRT models and experimenting with longitudinal models to measure user improvement over time. I compared estimations conducted with different R packages, with varying subsets of data, and ways of incorporating information on social interactions that occur alongside item responses. I used random item models to assess learning over time–time as calendar time, as item presentation order, and as discrete “sittings”. The people of Grockit are welcoming and innovative, and I feel fortunate to have had the chance to work with them and grapple with many interesting complexities.

I enjoyed working with Dan over the past few months, and wish him well on his return to academia in the Fall. Keep an eye on this blog for announcements of new assessment-oriented features on Grockit, several of which have benefited from Dan’s hard work this summer.

No Comments »

“How would you describe Grockit to a friend?”

By Ari Bader-Natal posted July 14, 2010

We asked a group of our users to tell us how they would describe Grockit to a friend. We read each comment in detail and we thought it might be interesting to enter all the student responses into Wordle (typos included, but that’s ok, these were Algebra students). Wordle is helpful because ‘The clouds give greater prominence to words that appear more frequently in the source text’. It was cool to see ‘Fun’, ‘Math’ and ‘Helps’ as some of the biggest words.

1 Comment »

Being intelligent about supporting learning

By Ari Bader-Natal posted June 22, 2010

As you probably know, Grockit offers learners the choice of three different modes of study: individual practice, peer study groups, and instructor-led lessons. What you might not know (unless you’ve read about Research at Grockit) is that these three modes of learning draws on ongoing research in three corresponding subfields of study. Last week, I got up-to-speed on the state of the art in one of these areas, individual study, at the Tenth International Conference on Intelligent Tutoring Systems. I wanted to share a few highlights from the conference here.

On Day 1, I started by co-organizing a workshop with Erin Walker and Carolyn Penstein Rosé (both of Carnegie Mellon University) on Opportunities for intelligent and adaptive behavior in collaborative learning systems. It was a great opportunity to bring together a group of people actively pursuing research at the intersection of intelligent systems and collaborative learning. Here is a snippet from the workshop website.

Intelligent tutoring systems are generally designed to tailor instruction to the individual student, but this does not mean that ITS-guided learning must necessarily be a solitary activity. A variety of recent systems have demonstrated ways in which an adaptive learning environment can incorporate and benefit from the presence of multiple learners. Similarly, students using computer-supported collaborative learning systems have been shown to benefit from the introduction of adaptive support that targets the collaboration. In this workshop, we invite discussion and seek to explore ways in which the combination of collaborative and intelligent aspects of a system can benefit the learner by creating a more productive learning environment.

If you’re interested in learning more about this workshop, you can find the presenter list, rapid-fire slides, and full proceedings on the workshop website.

As the week continued, I got a chance to hear presentations on a wide variety of research projects. A few of the talks that I found interesting this year touched on these topics: deciding if and when to provide hints, identifying the moment of learning from patterns in data, incorporating teachers into the process of designing a system, automatically generating hints from patterns in past data, understanding the ways in which human tutors are adaptive, incorporating dialogue agents in peer collaborations, and modeling learning gains over time. You can see the full spectrum of work presented in the conference proceedings index.

Grockit was proud to be one of the sponsors of this year’s Intelligent Tutoring Systems conference.

No Comments »

Games+Learning+Society+Grockit

By Ari Bader-Natal posted June 21, 2010

Two weeks ago, Grockit participated in the Games+Learning+Society conference. On the second day of the conference, I joined Jeramy Gatza, Curriculum Innovation Specialist on the Research and Discovery Team at Florida Virtual School, to discuss a recent project in which eight Algebra I classes at Florida Virtual School piloted the use of Grockit as a supplement to their standard course curriculum. The presentation, entitled Collaborative Learning Games in the Virtual Classroom: Piloting Grockit at Florida Virtual School, took an interactive form. Each participant had a laptop, which allowed us to ground our discussion of the social, motivational, and collaborative aspects of the platform in first-hand experience. Here’s a bit more about the discussion, taken from the presentation abstract:

A virtual school can offer a student the ability to complete a course on their own schedule, from any location. The challenge in providing a flexible, individualized learning environment is that students may feel disconnected from each other, and can miss the opportunity to learn from interactions with their peers. Multi-player online learning games may hold a solution. By providing a venue for learners to connect and interact, these games can extend the benefits of collaborative learning opportunities to the geographically-dispersed students in a virtual school.

…Our goal for this workshop is to share with participants both an intuitive sense and data-grounded evidence about how multi-player learning games, like those in Grockit, can help connect, motivate, and engage students who are geographically and socially isolated. The workshop will conclude with a group-wide discussion of other experiences with, and opportunities for, using game-based collaborations as a way to connect learners across the web.

Two days later, Jeramy and I again presented together, this time to discuss our work with a group of educators as part of the GLS Educator Symposium. This presentation was nominally grounded in work that I’ve published on how we go about deciding between synchronous and asynchronous interactions for the various components of Grockit’s collaborative learning activities (for more info, see the source paper and the asynchronous discussion thread on this work). The real heart of the session, however, was devoted to a series of interesting questions raised by participating educators about the real-world task of incorporating Grockit (or a system like it) into the classroom setting. Jeramy Gatza provided a very interesting perspective, responding to several questions based on his experiences using Grockit at Florida Virtual School.

While I was at the Games+Learning+Society conference, I saw/heard/participated in a number of excellent talks, tutorials, demos, and keynotes. It was exciting to see the incredible variety of ways in which researchers and educators have been incorporating games into learning and learning into games. I’m excited to bring these ideas back to Grockit, and use them as inspiration for our own internal brainstorming sessions on games and learning. Stay tuned for new announcements on this front…

No Comments »

Do students spend more time solving appropriate challenges?

By Ari Bader-Natal posted May 18, 2010

When you play a solo game in Grockit, the questions that you see are selected based on your recent performance (Grockit gets more adaptive). Our goal in doing this is to offer you challenges that are neither too easy nor too hard for you, in order to provide better opportunities for learning.

Our concept of how difficult you’ll find a question is based on our estimate of the likelihood that you will answer the question correctly. If, for instance, we estimate that there is a 90% chance that you will answer the question correctly, we think that you’ll consider that question to be pretty easy. Others may find the same question difficult, but you’ll probably find it to be straightforward.

Today’s question: Does a subjective measure of difficulty have any relationship with the length of time it takes a student to solve the problem?

Think for a moment about your own problem-solving experiences. Do you think that you spend more time on a harder question than on an easier question? Less time?

We record a lot of interesting data when students answer questions on Grockit, on some of this data can help us answer the question at hand. Below are two plots of the average time taken by Grockit students to answer GMAT Quantitative and ACT Math questions, as a function of how likely we believe the student to correctly solve the question. When you look at the plots, the first thing to note is that the relationship between subjective difficulty and response time isn’t linear.

These plots indicates that, on average, students are spending less time answering each question that are subjectively very easy or very difficult than on the questions that are in between. Do you find this surprising? Interesting? Does this reflect your own experiences studying? How would you interpret these plots?