1. Steffen Rendle, who identified useful features for users and questions, then used factorization machines to automatically develop a factorization model from those features.
2. Alexander d’Yakonov, who used traditional machine learning approaches to solve the challenge as a classification problem.
3. Pankaj Mishra, who blended an ensemble of results from collaborative filtering and a variety of IRT models, based on models used in the Netflix prize and the 2010 KDD cup.

We learned a number of interesting things by hosting this competition:

1. A sense of how good our existing approach was.  We posted benchmark results from the IRT algorithm at the heart of our internal prediction model.  While it didn’t win the competition (and we’d have been disappointed if it had), it was a difficult benchmark to beat — improving on it was a serious challenge for the competitors.  Before the competition, though, we didn’t really know whether we could feel confident in the strength of our model, or is there was something much better we could be doing.  Knowing that the problem has been looked at by hundreds of world-class data scientists gives us a lot of confidence that there isn’t a lot of room for improvement on the results.
2. Collaborative filtering/factorization methods for organizing and clustering our questions.  While we’ve relied on manual categorization, this is a promising way of identifying the actual knowledge areas used by each questions, so that we can get a sense for what questions actually use a common set of skills.  This should help us give students a more accurate picture of the areas they really understand and which questions they’ll actually get right.
3. Stefan’s libFM tool.  This proved to be a very powerful way of developing those factorization models.

We’re glad to have some new tools in our search for better ways to understand what people are learning and to help them learn better.  All the data (training and test) from the competition is now publicly available in the data section of the kaggle page.  We hope that it will continue to be useful for understanding student knowledge and predicting performance.

I’m very excited that we’re reaching out to the machine learning community, because I’m tremendously excited about the potential for technology to change education. I think leveraging the data we can collect is a huge part of that. I honestly believe that by improving assessment, we can improve the way people learn. By knowing what areas you’re strong and weak in, you can better know when you’re at a suitable level and how to direct your future studying. We want to be able to help students direct their studying — and eventually, improve student testing itself. There’s a lot more to knowing your level than simply what questions you got right or wrong, and we want to take advantage of that as much as possible.

This is a manifestation of the Grockit value of putting the learner first. We’re doing this purely to try and improve the way we help students study. We’re putting our money where are mouth is, too, with cash prizes will also be awarded to the top three finishers. The competition runs through February. Details on the data set, the prizes, and the competition are all available at:

http://www.kaggle.com/c/WhatDoYouKnow

Thanks for reading, and we hope you check it out!

There’s a wealth of educational videos available online, but the primary mode of interaction among viewers — short-form commenting — doesn’t do much to enhance or facilitate learning. When you’re trying to grok a challenging lecture from an AI course at Stanford or an open talk at MIT, you need more than comments. You need a way to ask a question about a particular point that confused you, or ask for a clarification or maybe a useful reference on something mentioned in the video. You really need a Q&A site, and you need to be able to attach each of your questions to a particular time in the video. You need Grockit Answers. Here’s a quick video clip (live Q&A discussion here). Feel free to ask questions about this video on Grockit Answers.

Built on Popcorn.js, a fantastic new HTML5 media framework for better connecting web videos to web pages, Grockit Answers displays each question and its answers at the exact point in the video that they are most relevant. As a viewer, some of the things that confuse you are are likely to be the same as what confused others before you. In a time-anchored Q&A on Grockit Answers, just as the question begins to form in your mind, it may magically pop up on the screen, along with a few great answers.

For the educators out there, I’m happy to say that we developed a number of additional features with you in mind. You can create a Q&A page on a video that only you and your students have access to, and you can moderate the conversation to get additional controls and deeper insight into what’s happening. Finally, and most exciting, you can add these Q&A pages to your own course websites (example), easily replacing links to YouTube or Vimeo videos with your moderated discussions. Feel free to contact us at answers@grockit.com with any questions about this.

Start a Question & Answer page for any video on YouTube. Share it with your classroom or share it with your friends. And please, let us know what you think.

Rusty Greiff is Grockit’s Chief Strategy and Corporate Development Officer and a resident of the greater Washington DC area.  Rusty has an extensive background in politics and education.

My 8-year old son is a LEGO junky.  He goes through three basic steps to LEGO greatness:

1) he shuts the door to his room

2) he pours the 500 pieces of LEGOs on the floor

3) he lets it rip.

Skyscrapers, spaceships, laser-rovers; massive and intricate structures are built – most of the time with his brother and best buddy next door.  Often, I will peer through the crack in the door to assess their progress.  Rarely do they know where their LEGO journey will take them, but they are comfortable in leaping ahead together, adjusting the pieces, and brainstorming until the creation is complete, often ending in a multi-player “high five”.

Last week, I played and learned with my own set of LEGOs along with some of the most talented educators in the country at the KIPP Charter School’s Annual Summit in Nashville, TN.  Every year, KIPP brings its teachers, directors, funders, experts in the field and students together to discuss best practices.  Alongside KIPP’s finest teachers, administrators and students, I led a 90-minute session, LEGOs in hand, on collaborative and social learning.  The LEGO exercise and talk was meant to help participants better understand the impact of Grockit’s adaptive and collaborative platform as it is applied to KIPP schools.  Currently, Grockit and KIPP are partnering to connect California KIPP students in peer-peer SAT/ACT test prep, with the goal of mastering test prep content and improving scores through social learning.

Not surprisingly, bringing people together (with a massive box of LEGOs) with different learning styles, specific skill sets and expertise, can create a dynamic experience that translates into some beautiful results.

By way of examples, Grace, a KIPP teacher from Washington DC preferred working in a team of 2 to build the foundation while her colleague Jennifer created a three-story roof garden to attach to Grace’s structure.  A KIPP teacher from Newark encouraged her team to strategize first, drawing elaborate plans before finding color-coordinated LEGOs to complete the building.  One AP teacher demanded of his team, “Just have fun” and attacked the LEGOs, building a killer “car-house” that of course could fly.

The LEGO exercise and session was an easy demonstration of existing academic research suggesting that by engaging learners through social games, collaborative problem solving and peer-to-peer studying, companies like Grockit are helping learners master subjects and achieve higher test scores.  Literally, this is what we do at Grockit every day for our hundreds-of-thousands of students.

The experiment is fairly straightforward. We randomly selected a portion of newly arriving Grockit learners and put them into the testing group, giving them a slightly modified Grockit experience. Instead of letting people select whether to study in a group or alone, we gave them a single button: “Study Now”. Each time they chose to study, we’d look and see whether there were other people studying the same thing. If there were, we’d treat this as an experimental case that we could test and flip a coin to see what study condition this would go in. Half the time, we put the learner into a solo game, studying on their own; the other half, we put the learner into a group game, studying with other people.

We found three key results supporting the hypothesis that group study is more engaging and effective for learners:

• We found that learners studied longer when they were in a group study session than when they were alone. Studies have shown that time on task is a key predictor of study effectiveness. It’s a simple correlation: studying longer is more effective. By providing a more engaging study experience, learners have more fun and spend longer studying. On average, learners spent 3x as long studying when they were in a group study session as when they played alone. The boxplot is even more striking: you can see that learners spend more time studying in group games across the board. (For my fellow stats geeks out there, the Mann-Whitney-Wilcoxon Rank-sum test results in Z = 8.3883, with a p-value < 2.2e-16. That's pretty irrefutable, as statistics go.)
• We found that learners studied over 2x as many questions when they were in a group study session as when they were alone. This was important because we wanted to make sure that studying with others wasn’t just making people slower to answer questions. But learners study more than twice as many questions, on average, when in group study than when they study alone.
• We found that learners were more likely to answer questions correctly when they were in a group study session than when they were alone. It could have been that learners were simply goofing off in group study, and that they were socializing rather than seriously paying attention to the material. But in fact, the opposite was the case with Grockit learners, where the social pressure is to focus on the material. Learners in group games actually answered correctly more often than when they were playing solo.

We want to find the most effective ways to help people learn, and that requires making sure the results are valid and verifiable. We’re working on a peer-reviewed paper to provide full results and data from our experiment; if you want access in the meantime, please write to research@grockit.com. Grockit was founded on the belief that we could come up with a better way to help people study, and that a key part of doing that was helping learners teach and learn from each other. We’re glad to have some strong evidence to support this belief, and will continue to test our theories to build an even better system.

The Virtual Study Hall: Making Personalized Learning Collaborative

Virtual schools let students learn from any place at their own pace. The challenge in this individualized learning is that without a synchronized curriculum and shared venue, students can feel disconnected from their peers and alone in their studies. As online learning options are increasingly available at the secondary and post-secondary level, the issues that virtual high schools face today are a preview of challenges to come with this new wave in technology-enabled learning. While the move towards personalized learning appears to necessitate a move away from social/collaborative learning, this need not be the case. Grockit has developed a web-based learning platform that bridges this gap, with a collaborative learning platform that creates networks for self-paced learners. Building on an existing relationship with Florida Virtual School, we will demonstrate and measure how this platform can increase learning gains by connecting and engaging thousands of students enrolled in 7th-9th grade level math courses online.

This project addresses a stumbling block that many technology-based efforts for individualized learning either have faced or will face soon: a large-scale shift towards truly personalized learning in a school fragments a single cohort of learners into single-learner cohorts. For online programs and courses, this fragmentation can negatively impact student engagement and completion rates. In building a system to be both collaborative and adaptive, Grockit has grappled with this issue for several years, and has now identified several solutions to enable both approaches to be supported simultaneously. For courses offered entirely online, restoring the opportunity for peer-to-peer collaborative learning can be quite powerful. By building a technology-based learning environment as a social construct from the bottom-up, students can reap the benefits of studying within a community of learners — a sense of connection to other learners, informal peer-to-peer assistance and motivation, and a social context for studying.

We’re looking forward to the opportunity to implement and evaluate this approach on a large scale at Florida Virtual School, which is the largest state-led public virtual school in the country. For this project, we’ll track and quantify the impact of incorporating Grockit’s core platform into 7th-9th grade level math courses on student engagement and learning outcomes, and will share and publish our findings with the larger community of researchers and educators interested in improving online learning.

I’m happy to say that Grockit is quite open about our ongoing research efforts, including our peer-reviewed publications, assorted research collaborations, current domain applications, and underlying pedagogical foundations. If you’re interested in learning more, check out our overview on the Methodologies and Technologies of the Grockit Learning Platform. If you work with a school (traditional or virtual) that may be interested in partnering with Grockit, or if you are an education researcher or learning systems designer interested in discussing potential research collaborations, please let me know and I’ll be in touch.

I’d encourage you to check out the full list of NGLC Wave II finalists. Lots of exciting work happening here!

We were lucky to have Daniel Furr join us this summer for the Grockit Graduate Research Internship program. Dan is a Ph.D. student in the University of California Berkeley’s Graduate School of Education, with a focus on Quantitative Methods and Evaluation.

At Grockit, Dan has been working on a variety of assessment-oriented projects, such as: on understanding how much data is needed to make use of an item response theory model in a Grockit network, on longitudinal models that help us refine our understanding of student improvement over time, and on extending our current models to incorporate the effect of collaboration, discussion, and repetition on a student’s probability of response accuracy. Understanding the effect of factors like these allows us to better select appropriate challenges for students in Grockit games.

About this internship, Dan writes:

Over the summer I focused on comparing approaches to estimating IRT models and experimenting with longitudinal models to measure user improvement over time. I compared estimations conducted with different R packages, with varying subsets of data, and ways of incorporating information on social interactions that occur alongside item responses. I used random item models to assess learning over time–time as calendar time, as item presentation order, and as discrete “sittings”. The people of Grockit are welcoming and innovative, and I feel fortunate to have had the chance to work with them and grapple with many interesting complexities.

I enjoyed working with Dan over the past few months, and wish him well on his return to academia in the Fall. Keep an eye on this blog for announcements of new assessment-oriented features on Grockit, several of which have benefited from Dan’s hard work this summer.

We asked a group of our users to tell us how they would describe Grockit to a friend. We read each comment in detail and we thought it might be interesting to enter all the student responses into Wordle (typos included, but that’s ok, these were Algebra students). Wordle is helpful because ‘The clouds give greater prominence to words that appear more frequently in the source text’. It was cool to see ‘Fun’, ‘Math’ and ‘Helps’ as some of the biggest words.

As you probably know, Grockit offers learners the choice of three different modes of study: individual practice, peer study groups, and instructor-led lessons. What you might not know (unless you’ve read about Research at Grockit) is that these three modes of learning draws on ongoing research in three corresponding subfields of study. Last week, I got up-to-speed on the state of the art in one of these areas, individual study, at the Tenth International Conference on Intelligent Tutoring Systems. I wanted to share a few highlights from the conference here.

On Day 1, I started by co-organizing a workshop with Erin Walker and Carolyn Penstein Rosé (both of Carnegie Mellon University) on Opportunities for intelligent and adaptive behavior in collaborative learning systems. It was a great opportunity to bring together a group of people actively pursuing research at the intersection of intelligent systems and collaborative learning. Here is a snippet from the workshop website.

Intelligent tutoring systems are generally designed to tailor instruction to the individual student, but this does not mean that ITS-guided learning must necessarily be a solitary activity. A variety of recent systems have demonstrated ways in which an adaptive learning environment can incorporate and benefit from the presence of multiple learners. Similarly, students using computer-supported collaborative learning systems have been shown to benefit from the introduction of adaptive support that targets the collaboration. In this workshop, we invite discussion and seek to explore ways in which the combination of collaborative and intelligent aspects of a system can benefit the learner by creating a more productive learning environment.

If you’re interested in learning more about this workshop, you can find the presenter list, rapid-fire slides, and full proceedings on the workshop website.

As the week continued, I got a chance to hear presentations on a wide variety of research projects. A few of the talks that I found interesting this year touched on these topics: deciding if and when to provide hints, identifying the moment of learning from patterns in data, incorporating teachers into the process of designing a system, automatically generating hints from patterns in past data, understanding the ways in which human tutors are adaptive, incorporating dialogue agents in peer collaborations, and modeling learning gains over time. You can see the full spectrum of work presented in the conference proceedings index.

Grockit was proud to be one of the sponsors of this year’s Intelligent Tutoring Systems conference.

Two weeks ago, Grockit participated in the Games+Learning+Society conference. On the second day of the conference, I joined Jeramy Gatza, Curriculum Innovation Specialist on the Research and Discovery Team at Florida Virtual School, to discuss a recent project in which eight Algebra I classes at Florida Virtual School piloted the use of Grockit as a supplement to their standard course curriculum. The presentation, entitled Collaborative Learning Games in the Virtual Classroom: Piloting Grockit at Florida Virtual School, took an interactive form. Each participant had a laptop, which allowed us to ground our discussion of the social, motivational, and collaborative aspects of the platform in first-hand experience. Here’s a bit more about the discussion, taken from the presentation abstract:

A virtual school can offer a student the ability to complete a course on their own schedule, from any location. The challenge in providing a flexible, individualized learning environment is that students may feel disconnected from each other, and can miss the opportunity to learn from interactions with their peers. Multi-player online learning games may hold a solution. By providing a venue for learners to connect and interact, these games can extend the benefits of collaborative learning opportunities to the geographically-dispersed students in a virtual school.

…Our goal for this workshop is to share with participants both an intuitive sense and data-grounded evidence about how multi-player learning games, like those in Grockit, can help connect, motivate, and engage students who are geographically and socially isolated. The workshop will conclude with a group-wide discussion of other experiences with, and opportunities for, using game-based collaborations as a way to connect learners across the web.

Two days later, Jeramy and I again presented together, this time to discuss our work with a group of educators as part of the GLS Educator Symposium. This presentation was nominally grounded in work that I’ve published on how we go about deciding between synchronous and asynchronous interactions for the various components of Grockit’s collaborative learning activities (for more info, see the source paper and the asynchronous discussion thread on this work). The real heart of the session, however, was devoted to a series of interesting questions raised by participating educators about the real-world task of incorporating Grockit (or a system like it) into the classroom setting. Jeramy Gatza provided a very interesting perspective, responding to several questions based on his experiences using Grockit at Florida Virtual School.

While I was at the Games+Learning+Society conference, I saw/heard/participated in a number of excellent talks, tutorials, demos, and keynotes. It was exciting to see the incredible variety of ways in which researchers and educators have been incorporating games into learning and learning into games. I’m excited to bring these ideas back to Grockit, and use them as inspiration for our own internal brainstorming sessions on games and learning. Stay tuned for new announcements on this front…