Author: Frank Kane, Platform: Facebook
Allow me to join the club of instructors pontificating on the nature of reviews.
It really seems to me that there is very little correlation between the actual quality of my seven courses, and the average review scores they have. In fact, the course I'm most proud of has the lowest score - and my earliest courses that make me cringe when I look at them are just as highly rated as my newer, professionally produced ones.
Here's what I think reviews are really measuring, assuming your courses pass some reasonable quality bar (based on my over 100,000 students):
1. How difficult is the course material? Topics that require a mathematical or computer science background - or worse, both - are going to be challenging for a lot of people who are trying to learn a valuable skill but lack the necessary educational background. If they simply can't "get it" no matter how good of an instructor you are, they're going to blame you as the instructor, not themselves. And they'll take it out in their review scores.
2. How much did students pay for your course? We know that the less someone pays for something, the less they value it. I've seen this clearly with a free course that I ultimately unpublished, and the decline in rating scores I've seen as Udemy has dropped its price points over time. I think there's also a valid concept of "quality of student," and lower price points attract people who are more likely to give you a bad rating and get a refund just for the fun of it.
3. What is the international breakdown of your students? One of my courses has been localized to Japanese and German. The same exact course gets a 4.5 rating in the US, but a 3.8 in Japan and a 4.1 in Germany. Different cultures interpret star ratings in different ways, and language barrier issues can also affect student satisfaction. I've noticed that almost all of the negative reviews on my English-language courses are from students from India. My most recent negative review came from a student who wrote the review in Spanish, complaining that I didn't use a "pencil" on my slides to write down notes on them while I was talking. Many International students have an easier time reading English than listening to it.
So, what's the positive, constructive takeaway on this? Well, for one thing, maybe it doesn't really matter - I don't see reviews influencing search placement all that much. It also means you need to think about your international students. Speak clearly, and make sure your captions don't suck. And as best you can, set clear expectations of the skills your students need as a pre-requisite. Listing these in the course requirements isn't enough - say it again in your promo video.
At Udemy Live there was a lot of talk of matching the right students to the right courses. There is a recognition from Udemy's end that many low reviews are really just a mismatch between students and the right courses for them and their skill level, and they are taking steps to correct this.
We also know Udemy is focusing a lot of energy on international expansion and creating a good experience for international students.
So, some patience is warranted. Udemy has the data and the people needed to work this all out, and they're on it from what I can tell.
Gregory Caremans: You forgot to mention: 4. whether they are having a good or a bad day. I'm not even kidding. If they just had a fight, or they're stressed, or even if they are just hungry (seriously!), they are much more likely to give us a lower rating.
Frank Kane Maybe an equally valid hypothesis is that students are rating you, the instructor, much more so than the course itself. That would explain why all of my courses are rated more or less the same.
Chris Parker I've taken courses where I couldn't stand the instructors accent and was distracting from learning. I can imagine I've had students that didn't like my voice either. Nothing personal. And nothing you can do about it.
John Bura I try to mix game programming and game art tutorials and I get the same thing: 3 stars - Can't program - Can't do art.
Eventually, I think Udemy will have to price courses based on weight (hours). A 50-hour course should cost marginally more than a 5-hour course.
Udemy is a new business they are figuring it out all of the time. I look forward to what they do next.
Bo Andersen I agree with #1. I think it's partially caused by Udemy's extremely aggressive pricing strategy. Some students are just not meant to join a given course, but join it when saving 95%. For example, I've tried clearly stating that a course is for beginners, and yet people complain that the course is too basic. When adding loads of new content to the course, I get people saying that the course is too comprehensive. Basically I've noticed that many students don't read the course description, learning goals, and requirements at all. A simple example is that one of my courses clearly mentions as the single prerequisite that students must know JSON before enrolling. It's been at the top of the landing page for years, and I still get bad reviews from people because they don't know JSON. A lot of it just comes down to overselling stuff, which is pretty much Udemy's business model. The latest feature of being able to add courses to the cart directly from the search results is not really going to help, but that's another story.
My point is that as an instructor, you just cannot please everyone, especially if your course is not aimed at absolute beginners, because you are just going to see people enrol in the course who really shouldn't. There are many other factors as well, but this is one of them. The more complicated the subject, the harder it is to get good ratings, even if you do a great job at explaining. You are just going to see people enrol in an intermediate programming course who don't know how to open a file in a text editor, etc. (slightly exaggerated, but you get the point) :)
P.S. Forgot to mention that there is no apparent connection between how much value your course gives. You can have a 15 hour course packed with tons of valuable information, but a 3 hour course covering just the basics will often get better ratings. It's more about instructor performance than the actual content.
Chris Parker In the end, you can't please all the people 100% of the time. It doesn't matter if you cover every possible scenario, someone will find something they don't like and give you a 1 star. I too had a free course. Got tired of getting low reviews and put it up as a paid course. It's now making money and getting better reviews.
Stéphane Maarek General advice: I create a lecture named: Course-requisite (alongside goals, expectations, etc). It's boring as hell, takes up 5 minutes of my courses, but at least, sets expectations. Students can't avoid seeing it. I encourage them to get a reimbursement if they feel this course isn't for them. I think it has worked on my favour
Additionally, make a "learning path". I have a beginners course, and I go more advanced in other courses. One course is a pre-requisite to another. Frank Kane, if you feel some students lack some background, make a course for it, and suggest in an intro video that they should check it out if they feel fuzzy about some concepts
Jon Avis I have also noticed poor review scores from Indian students. Every review I have on my course below 4 stars is from an Indian student. And none have left a comment so I dont know how to improve it. Everyone else has scored it between 4 and 5.
Luke Burkina Excellent post. I think they should allow the creation of a Udemy faculty body to help address the quality of courses and reviews. In the past, I wrote an algorithm that will remove outliers before computing the mean. I even submitted the idea, but it did not get traction. The mean is not a good measure of the center of the data when it contains outliers.
Why this important? If a course takes a half-star or two stars ratings from a student, for no apparent reason after just less than 20% of completion, the average rating will be depleted and the course in jeopardy of being removed from promotions. Over time, low ratings affect the overall brand.
Another quick alternative is to compute and display the median rating (which is robust).
