Basic principles of jugding kata in IJF competition

Like most pople know, the evaluation und judging of kata in IJF-kata competition is based on the official Kodokan instruction matierials, which is the DVDs and the textbooks. This lead to the opinion and conclusion, that the highest scores are given to those, who "copy" the movements of the DVDs. This is of course nonsense, but here and there we find this idea in the circles of competitors as well as in the circles of those, who critique this in public.

With this post I will try to explain the philosophy behind the system of scoring and doing this, I want to give some background information usually not explained in public. To say the truth: I do not believe, that all international kata judges are familiar with the points I will explain below. The published results f.ex. of the WC in Kyoto proof that.

1.) Outline of scoring

There 5 judges, who give scores for the kata. The highest and lowest score is dropped, so there are the scores of three judges remaining in the final result.

There are points from 0 to 10 given for:
- opening and closing
- each of the 15 (resp. 20/21) actions
- overal flow

In case of Nage-, Katame- and Ju-no-Kata there are 18 marks by each judge, what is a maximum score of 180 points, which is max. 540 points in total of three judges. Kime-no-Kata and Kodokan Goshinjutsu include more techniques, so the max. score is higher.

2.) Criteria for scoring the techniques

If the judges give marks from 0 to 10, then they need a clear guideline about the difference between f.ex. a 7 and 8 or 4 and 5 points. If there is no such guidelines, scoring becomes a lottery of personal "opinions" and the system is getting completely corrupt. Obiously the Kodokan DVDs do not offer such a guideline, so the DVDs cannot be a guideline for scoring.

There is a very clear philosophy behind the scoring, although of course there are many problems to put it in practise, also because the same ideas should be applied to completely different kata. I will adress some of these issues later.

What or the basic ideas:

A) No matter how "accidetal" of wrong an attempt to execute something is - it must always be scored higher then a forgotten technique.

--> so the rules say, that the score for a forgotten technique is 0 and the score for any attempt is minimum 1 point. I think this is understandable and cannot be questioned.

B) A technique, which is completely wrong must always be scored lower, then a technique which is basically correct, but "only" lacks realism and effectivity.

--> the score for a completely wrong or failed technique (f.ex. harai-goshi instead of uchi-mata, dropping a weapon, being "killed" in KDG or Kime-no-Kata  etc.) is something between 1 and 5 and additionally the max. score for overall flow is 5.

In other words. If a technique is basically correct, but lacks in realism and/or effectivity the minimum score is 5 without limiting the score for the overall flow.

I think this general rule can also not be questioned. Totally wrong is worse then ineffective....

C) An execution of an action, which is basically correct, but lacks in realism and/or effectivity must always be scored lower then a technique, which has some room for improvements, but which do not really effect the functionality of the technique.

---> so techniques which are bascially correct, but flawed in realism have to be scored between 5 and 7 points. The range from 8 to 10 points is reserved to executions without any real problems in realism and effectivity.

Typical flaws of realism/effectivity are wrong distances, not enough kuzushi, "reaction" before an attack, Uke moves voluntarily, to little dynamic of the action etc.

So we have clear guidelines:

0 --> forgotten technique
1-5 --> completely wrong or failed
5-7 --> basically correct, but flawed in realism
8-10 --> correct and effective technique

The Kodokan DVDs and the textbook only can be used as guidelines about what is basically correct - not less, not more.

The scoring sheet of IJF kata-competitions

Maybe it was not the best idea, but when the basic guidelines were ready, the task of making scoring sheets had to be done. The idea was to make crosses instead of writing marks. So the idea of substracing mistakes of different levels from the ideal of 10 was born.

Big mistake: a big mistake is defined as an action, which is completely failed of basically wrong. A big mistake is a substraction of 5 points from the max of 10. A big mistake can be combined with additional small and medium mistakes, so the max. score for a technique, that includes a big mistake is according to the above mentioned general rule 5 points or less.

Medium mistake: If there is a medium mistake in an action, the max. score is 7 (see above). So medium mistakes lead to a substraction of 3 points. Attention: there can only one medium mistake scored in a technique, because two medium mistakes would be a substracion of 6 points and the score lower then 5. This is not possible according to the general guideline. So the question is: was there a lack of realism or not. This is a problem in evaluating nage-no-kata (where right and left techniques are scored comprehensively) and ju-no-kata with its longer sequences of attack and defence, where we have a lot of "chances" to make medium mistakes....

Small mistake: a small mistake is not really a mistake, but an "imperfection". It´s something not really perfect, but without effect on the realism of e technique. A small mistake is a substraction of one point and each judge can score a maximum of two small mistakes for each technique (a third one would equal a medium mistake and therefore something that lacks realism - this would be against the general guideline, so it is not possible).

Small mistakes without a medium mistake result in 8-10 points, in combination with a medium mistake the score is between 5 and seven.

There are of course some finer points:

Follow-up mistakes: very often, we find that one mistake results in a second one and so on. The general rule is, that only the biggest of these should be scored. For example, if the starting distance is not really wrong, but a little bit to short (little mistake) and that results in lack of kuzushi (medium mistake), which in turn results in (very little) problems with the final balance, then only the medium mistake should be marked and the score is 7. In contrast, if the distance is a bit short, but kuzushi is ok and there are minor problems with final balance, the score is 8 (to small mistakes). If the final balance is also perfect, then it´s a 9 (only one small mistake).

Typically in ju-no-kata the follow-up mistakes are a problem, but it only occurs, when there is already a medium mistake - resp. a flaw in realism.

Positioning: wrong positioning on the tatami is regarded as medium mistake, hat should be changed IMHO. Doing a correct thing from a wrong position should be scored higher the doing a flawed thing from the "correct" position. But these problems can be avoided, because those who enter kata-competition should be able to prepare themselves and find out, what the correct starting positions for each kata are.

What I tried to explain is the main reasoning behind the scores. BTW: the demonstrations seen on the Kodokan DVDs are not perfect in the sense, that they would be scored with 10 points for each technique. The average score would likely be around 8 points...

If one looks at the scores of international competitions, one finds out, that the cut for the medal-ranks is around 80% which is an average score of 8 points. So there are only very few people in the world at the moment, who can demonstrate a full kata without any "medium" mistakes. Of course there are a lot of other people believing, that they can do it, but a reality check usually opens their eyes.

Of course - effective technique and realism is not all about kata, but there cannot be "a better" kata, when technical flaws are repeatedly done and people do not work on them.

wdax wrote:Like most pople know, the evaluation und judging of kata in IJF-kata competition is based on the official Kodokan instruction matierials, which is the DVDs and the textbooks. This lead to the opinion and conclusion, that the highest scores are given to those, who "copy" the movements of the DVDs. This is of course nonsense, but here and there we find this idea in the circles of competitors as well as in the circles of those, who critique this in public.

With this post I will try to explain the philosophy behind the system of scoring and doing this, I want to give some background information usually not explained in public. To say the truth: I do not believe, that all international kata judges are familiar with the points I will explain below. The published results f.ex. of the WC in Kyoto proof that.

1.) Outline of scoring

There 5 judges, who give scores for the kata. The highest and lowest score is dropped, so there are the scores of three judges remaining in the final result.

There are points from 0 to 10 given for:
- opening and closing
- each of the 15 (resp. 20/21) actions
- overal flow

In case of Nage-, Katame- and Ju-no-Kata there are 18 marks by each judge, what is a maximum score of 180 points, which is max. 540 points in total of three judges. Kime-no-Kata and Kodokan Goshinjutsu include more techniques, so the max. score is higher.

2.) Criteria for scoring the techniques

If the judges give marks from 0 to 10, then they need a clear guideline about the difference between f.ex. a 7 and 8 or 4 and 5 points. If there is no such guidelines, scoring becomes a lottery of personal "opinions" and the system is getting completely corrupt. Obiously the Kodokan DVDs do not offer such a guideline, so the DVDs cannot be a guideline for scoring.

There is a very clear philosophy behind the scoring, although of course there are many problems to put it in practise, also because the same ideas should be applied to completely different kata. I will adress some of these issues later.

What or the basic ideas:

A) No matter how "accidetal" of wrong an attempt to execute something is - it must always be scored higher then a forgotten technique.

--> so the rules say, that the score for a forgotten technique is 0 and the score for any attempt is minimum 1 point. I think this is understandable and cannot be questioned.

B) A technique, which is completely wrong must always be scored lower, then a technique which is basically correct, but "only" lacks realism and effectivity.

--> the score for a completely wrong or failed technique (f.ex. harai-goshi instead of uchi-mata, dropping a weapon, being "killed" in KDG or Kime-no-Kata  etc.) is something between 1 and 5 and additionally the max. score for overall flow is 5.

In other words. If a technique is basically correct, but lacks in realism and/or effectivity the minimum score is 5 without limiting the score for the overall flow.

I think this general rule can also not be questioned. Totally wrong is worse then ineffective....

C) An execution of an action, which is basically correct, but lacks in realism and/or effectivity must always be scored lower then a technique, which has some room for improvements, but which do not really effect the functionality of the technique.

---> so techniques which are bascially correct, but flawed in realism have to be scored between 5 and 7 points. The range from 8 to 10 points is reserved to executions without any real problems in realism and effectivity.

Typical flaws of realism/effectivity are wrong distances, not enough kuzushi, "reaction" before an attack, Uke moves voluntarily, to little dynamic of the action etc.

So we have clear guidelines:

0 --> forgotten technique
1-5 --> completely wrong or failed
5-7 --> basically correct, but flawed in realism
8-10 --> correct and effective technique

The Kodokan DVDs and the textbook only can be used as guidelines about what is basically correct - not less, not more.

The scoring sheet of IJF kata-competitions

Maybe it was not the best idea, but when the basic guidelines were ready, the task of making scoring sheets had to be done. The idea was to make crosses instead of writing marks. So the idea of substracing mistakes of different levels from the ideal of 10 was born.

Big mistake: a big mistake is defined as an action, which is completely failed of basically wrong. A big mistake is a substraction of 5 points from the max of 10. A big mistake can be combined with additional small and medium mistakes, so the max. score for a technique, that includes a big mistake is according to the above mentioned general rule 5 points or less.

Medium mistake: If there is a medium mistake in an action, the max. score is 7 (see above). So medium mistakes lead to a substraction of 3 points. Attention: there can only one medium mistake scored in a technique, because two medium mistakes would be a substracion of 6 points and the score lower then 5. This is not possible according to the general guideline. So the question is: was there a lack of realism or not. This is a problem in evaluating nage-no-kata (where right and left techniques are scored comprehensively) and ju-no-kata with its longer sequences of attack and defence, where we have a lot of "chances" to make medium mistakes....

Small mistake: a small mistake is not really a mistake, but an "imperfection". It´s something not really perfect, but without effect on the realism of e technique. A small mistake is a substraction of one point and each judge can score a maximum of two small mistakes for each technique (a third one would equal a medium mistake and therefore something that lacks realism - this would be against the general guideline, so it is not possible).

Small mistakes without a medium mistake result in 8-10 points, in combination with a medium mistake the score is between 5 and seven.

There are of course some finer points:

Follow-up mistakes: very often, we find that one mistake results in a second one and so on. The general rule is, that only the biggest of these should be scored. For example, if the starting distance is not really wrong, but a little bit to short (little mistake) and that results in lack of kuzushi (medium mistake), which in turn results in (very little) problems with the final balance, then only the medium mistake should be marked and the score is 7. In contrast, if the distance is a bit short, but kuzushi is ok and there are minor problems with final balance, the score is 8 (to small mistakes). If the final balance is also perfect, then it´s a 9 (only one small mistake).

Typically in ju-no-kata the follow-up mistakes are a problem, but it only occurs, when there is already a medium mistake - resp. a flaw in realism.

Positioning: wrong positioning on the tatami is regarded as medium mistake, hat should be changed IMHO. Doing a correct thing from a wrong position should be scored higher the doing a flawed thing from the "correct" position. But these problems can be avoided, because those who enter kata-competition should be able to prepare themselves and find out, what the correct starting positions for each kata are.

What I tried to explain is the main reasoning behind the scores. BTW: the demonstrations seen on the Kodokan DVDs are not perfect in the sense, that they would be scored with 10 points for each technique. The average score would likely be around 8 points...

If one looks at the scores of international competitions, one finds out, that the cut for the medal-ranks is around 80% which is an average score of 8 points. So there are only very few people in the world at the moment, who can demonstrate a full kata without any "medium" mistakes. Of course there are a lot of other people believing, that they can do it, but a reality check usually opens their eyes.

Of course - effective technique and realism is not all about kata, but there cannot be "a better" kata, when technical flaws are repeatedly done and people do not work on them.

Wdax,

Thank you for doing this. Like you say, it is useful for both readers and kata judges.

It is a great experience for all us to hear you write about this with passion. As a successful kata competitor and a person who has submitted himself to judgment on his kata for many years has earned our respect. You are not just a talker but also a doer and someone who also teaches kata allowing others to benefit from your experiences.

As you will see I agree with almost everything you say, when you enter personal notes. So, I have no problem with you, but I do have a problem with the rules and system of rules.

As rational human beings we like to understand things, and thus written rules, especially if concise, may help that understanding. Only the judging experience has the same characteristics as a chain and is only as solid as its weakest link.

Let's take the IJF shiai refereeing rules. The description of what is an ippon score is pretty clear, and also fundamental to judo. It's one of the first words new judoka learn. If ippon is so clear so obvious, then how come it is constantly contested ?  It is because situations deviated from textbook exaples, it is because referees have limits in understanding, it is because of nerves, it is because of prejudice, it is because numerous factors.

Few kata judges have studied kata like you have, yet all of them are supposedly certified to judge you. How is that possible ?  Sure, there is no requirement that one has to be able to perform things as well as the performer in order to be able to judge them. I can quite in depth judge and explain why Wilhelm Kempff's Beethoven is so special, and I can also say how it differs from that of Wilhelm Backhaus whose Beethoven is equally brilliant, yet different from that of both Kempff and Edwin Fischer. At the same time, it also illustrates that it is possible for three different performances to be near perfection. Yet, there is only one and the same music score for each piece. How is this possible ?

While I know I can do that, there are also occasions where friends of mine are exposed to music and where conversations arise during which it is quickly obvious they have no clue and can't distinguish. Why ?  Obviously a lot of training and study.

Back to kata. There are going to be judges who despite not having your level who may be able to judge you properly, and there are going to be others who don't.

But the question is "how do we know?" And how can we use that knowledge to make differential decisions in who can and should be a kata judge.

The system of selection is problematic too, since it is not open. I am not an IJF kata judge, yet I have studied and researched kata very extensively. I have people in Japan whom I call my "friends" people whom I trust and I know from extensive interaction that their knowledge outweighs that of most teachers at the Kôdôkan and in the IJF. They are not kata judges either, are not even Kôdôkan instructors or much associated with the Kôdôkan. Under the current system it is not possible to get in the IJF or EJU committees or be judges unless you are first nominated by your own national federation, which practically means that you have to submit to all the politics and all the brownnosing. And some people excel in doing that even though their skills are very limited, whereas others refuse to submit to that. The consequence is that your end product does not consist of the most capable people but of the people who are willing to pay lip service to the greatest extent. This is problematic and also a known problem in economics and business management.

If one fear people who are critical, the consequence is that you are going to end up with a deficient product because even if in your organization their are people who know about the flies, they are going to keep their mouth shut because primarily they want to be your friend and not lose their job. In consequence, the same problems still exist, but will only emerge when the product is public, and the criticism will come from the anonymous public who does not have and never received any saying.

When I was in the army and at one point working in the kitchen, I was disgusted by the cockroaches and substandards of hygiene. At one point we were told that there would be an inspection from some high-up bobo, like a general or something. So our colonel forced everyone to work day and night to make the kitsch so clean that you could eat off the ground. We were threatened that if they would be able to make any remark that the colonel would make our life into a misery and find a reason to lock us up. when the inspection came, it was judged that our facility was exemplary. As soon as the general was gone, the cockroaches came out again and it was back to the same. Now, why did no one step up to the general and say "general, this is all one big sham, our kitchen is disgusting and everday there are cockroaches everywhere, but the colonel threatened us to set up this piece of theater and no one dares to open his mouth out of fear for represailles" ?  Why did this not happen ?  Because people do not want to risk their own perks. You know as well as I do that if anyone I would probably have done precisely that even if I knew that I would be screwed. In the end also that did not transpire because I had no kitchen service on the day the general came and was not even there when he visited.

The IJF and EJU are the same. Unless you kiss ass, it is not possible to be in there. There is no one there who refuses to kiss ass and goes just for quality. I write this not as criticism to the organization but to show that even if your rules would be perfect, the enforcement can't be because of considerable people's flaws.

I will now stepwise address your points:

1) Scoring

"There are points from 0 to 10". We know from other aesthetic sports that this is not really true and that scoring is skewed. In reality there are points from 6-8, unless something is so obvious, like skipping a technique. Now one could argue that it is only logical that scores are mostly 6-8 because likely people are mostly going to score in the middle with here and there a couple that is exceptionally poor and one that is exceptionally good. However, that is not necessarily true. You do not know beforehand what the statistical distribution is going to be and if quality indeed will have Gaussian distribution. It may or may not. The problem is that judges intentionally give marks that are skewed. The reason is that if marks are extreme chances that they will be identified as being off are substantial with consequences for their future. Besides, extreme scores are dropped based on the entirely speculative view that the more they deviate from the median the more wrong they are. No judge intentionally gives scores to achieve his scores being dropped. The rationale behind the skewing of scores is that the average score will be the likely most correct one. This is, however, wrong. If there are 5 judges and three of them give a 7, while one of them gives a 2 and one of them gives a 9, this rational dictates that the 7 is likely the most correct score because it is in the middle and because the frequency at which this score was given is three times as high as the frequency that a score of 2 or of 9 was given. Unfortunately that rational is simplistic and fundamentally wrong. In fact, it may well be that the score of 9 or the score of 2 was the most correct one, and that in fact that correctness would have been achieved by dismissing all other scores including the most frequently given one. That is because knowledge is logarithmic and not accumulative or serial. To put it simple, if you take a group of 50 physicists and you take Einstein and you ask a question about relativity, who is likely going to give the correct answer ?  Einstein, period. Or to put it slightly different, if you take a group of 100 people with an IQ of 130 they can never intellectually complete with a single individual with an IQ of 180. You simply cannot serially link ability. Five judges are not more accurate than one. It is unpredictable. They may be better, they may be the same or they may be worse. It all boils down to skill and knowledge.

2) Criteria for scoring techniques

I agree with everything you say.

In subsection 'B' it is said that a completely wrong technique must always be scores lower than a technique which only lacks realism of effectiveness. That sounds OK, but it isn't. Two reasons.

Suppose we do nage-no-kata, I am tori, you are uke, and we are about to start uki-goshi. You immediately attack me and hit me with your left fist. I notice this and appropriately respond throwing you with right uki-goshi. As you get up you realize your mistake and know attack with your right fist, to which I respond with lef uki-goshi. Thus this really deserve to be marked down ?  I could argue that I showed a higher understanding than most others because you deviated from the prescribed pattern by switching sides and I still reacted properly. We also did not forget a throw or attack, nor did we mix the position of uki-goshi with that of harai-goshi. The term "mistake" is not even proper here since at no point was the principle and objectives of this kata violated, and it adhered to the prime objective of kata: to improve my judo.

In subsection 'C' you write about techniques lacking in realism that should be scored between 5-7 points. This may sound logical, but too is open to problems, in fact two problems.

Firstly, what every judge sees behind 'realism' greatly differs. We see the same here on the forum. Fair enough, the spread is probably large, with some people being relatively novice while others aren't and yes, we have the ubiquitous critics of us all cowardly hiding behind our screen names, no doubt. However, that is not the point, the point is what is this realism ?  I argue that most jûdôka do not know what 'realism' in kata is. If they knew what it was they wouldn't be doing what they are doing. 'Realism' to them is what they know the people who are in a jury suggest they want. You can't seriously argue that any, but literally any koshiki-no-kata as it has ever been performed during the EJU continental kata championships or even today during the All Japan Kata Championships contains anything remotely connected to 'realism'. So, how then can they be assigning scores that go above 7 or even 5 because they completely lack realism and people's knowledg about what precisely realism in there is lacking. At least in nage-no-kata, yes, if one performs a 3rd kyû level nage-no-kata and strikes the opponent with a "weak hand" somewhere hitting a hole in the sky, sure, many will see that, but otherwise, it simply is not there.

There are a couple of Kôdôkan goshinjutsu videos from Kyôto online. It's outrageous, I was watching one and discussing privately with a couple of JudoForum members, and it struck me that uke was never out of balance. So, something as fundamental as kuzushi, and that at world level, and in advance kata, was totally absent. How can that be ?  How can someone like that gain a medal, and why do scores exceed 5 ?

In the jû-no-kata this is very problematic too. I can count the couple of people on the fingers of one hand who seem to know what realism is in jû-no-kata. Jû-no-kata today is mostly, and you know this too, performed devoid of realims. It is performed as an aesthetic gymnastics exercise. I was at the Kôdôkan international kata summer school, I think in 2010 and there were two Romanian girls who received the highest score and a special certificate. There performance was absolutely awful. What they did was gymnastics, and actually they turned out to have both a gymnastics past. It was synchronized acrobatics, but with jûdô it had nothing to do. There were no attacks, no realism, it was a dance, a display to impress what they could do with their bodies, how flexible they were. Why were these people not simply failed ?  I would have given higher marks to an 80 years olds with a hernia who shows how the exercise still contributes to his health, where there is true action/reaction, where you can see the atemi, the resistance, the attack, the defense, the response gô/jû.

So, one can write all one wants and say "5-7 basically correct, but flawed in realism", as long as this can't be filled in properly by people with the proper skills it is an empty carcass.

I also don't think that the lack of realism should imply that one just for that can't go lower than 5-7. When I watched the European Championships, or European 'Cup' Koshiki-no-kata, whatever I cannot for the life of me get those scores. When I score what I see the scores vary between 0-2. That is not a lack of realism where a score of 5 would still be justified. Impossible. It's a complete absence of comprehension


The scoring sheet of IJF competitions

I notice you write "the task of making scoring sheets had to be done". Why ?  I can see that to some people it would sound logical, but that does not mean it is. Quantitative research is not the same as qualitative research, different measures, different ways of expression. Kata is essentially a qualitative experience, not a quantitative one. However, what occurs is that the IJF strangely enough attempts to implement quantitative evaluation of qualitatative processes. This is not possible. The problem, obviously, is that the qualitatative appreciation is very difficult to standardize and very difficult to compare across different couples. If I evaluate three couples as follows:

A. This was superb, excellent expression of jû.
B. This was fantastic, excellent expression of realism.
C. This was awesome, excellent action/reaction.

Who now should be the winner, couple A, B or C ?  But if I give couple A 9.5, couple B. 9.25, and couple C. 9, there is no doubt, and it is made easy. Only, can I be sure that what I wrote in A, B, and C is accurately reflected in the numbers I listed ?

I very much like the fact that despite being an active competitor you show the spine to be critical about the rules. That deserves a lot of courage knowing that always someone could retaliate. I find again that in your criticism you are very close to mine !

Positioning

You are right, again. "Wrong positioning on the tatami is regarded as a medium mistake". Why ?  What evidence is there to say that wrong position is a medium mistake ?  What is the IJF's basis for such dogma ?  When is a positions 'wrong' ?  When you are 30 cm off the desired spot ?  or 1 tatami or 2 meters, or turned to the wrong side ?  From what point does it become a "wrong positioning" ?

The example you mention is correct, and I agree with you. There are other concerns. Is a 'different' position detrimental to the kata or does it follow from natural action/reaction ?

If you are doing katame-no-kata and have just completed kuzure-gesa-gatame and you decide to move all the way to tô-ma, get up and go sit at uke's head for kami-shihô-gatame and suddenly realize that you actually want to do kata-gatame, well, then that is obviously a problem and a mistake.

Luckily, nage- and katame-no-kata have rather simple and strict patterns, so it is hard to be really in the wrong position unless tori goes stand in uke's spot or the other way around, and in jû-no-kata if you are not in the right position you usually can't do the technique. So, really wrong position are an issue in kata with either very liberal patterns (goshinjutsu) or very complex patterns (koshiki-no-kata, but the latter is not yet on the IJF program).

The conclusion you arrive at is, I think, often true, but not the only one. You write "one finds out, that the cut for the medal-ranks is around 80% which is an average score of 8 points. So there are only very few people in the world at the moment, who can demonstrate a full kata without any "medium" mistakes". That is not entirely true. What is missing there is : "who can demonstrate a full kata without  --what the judges deem to be--  a medium mistake". Besides knowing the mathematical and statistical problems that skew the marks make the actual number of 80% at which you arrive invalid. What it really means is that there are people who can do these kata at a significantly different level than others, but those who can with a 90% level of expertise and those with a 70% expertise likely are both also forced within the group of 80% whereas they shouldn't. You can only make the conclusion you propose IF the 80% actually accurately reflects a level of 80%. It doesn't.

You mention "Of course there are a lot of other people believing, that they can do it, but a reality check usually opens their eyes."  (...)

You are probably correct, BUT it depends on what that reality check is. Vladimir Horowitz used to say he was irritated by critics and didn't need the as he knew well enough what he did himself and was a far harsher critic of himself than people imagined. The simple matter of carrying out a "reality check" is not so easy to achieve. You have been very successful in kata competitions, especially jû-no-kata. But what if I would say, "well, we are going to do this somewhat differently this time. You will compete, but not with your usual partner. Instead, I will determine your partner and introduce you to him 2 hours before your competition". Would you accept that as a basis for a reality check on your level of jû-no-kata ? You do realize that the partner I picked out for you is a blue belt of 140 kg, who has done the kata only twice before. I've been there in scenarios perhaps not that extreme but still not common. Years ago I jumped in to help out someone from another country far more junior and whose partner needed to undergo emergency surgery. We didn't medal, I thought it was very satisfactory that we got that far, but nevertheless some people tried to use the incident to suggest that after all this proved that our kata skills weren't so good as we wanted people to believe. Maybe they are right. I am not sure what I want people to believe when it comes to kata skills, but no doubt my own kata skills are not as good as I want the to be. In fact, there is nothing I am as good in as I want to be. My point is "reality check" required a check of reality and not a set-up horror scenario either. There are incidents known where the great Italian pianist Arthuro Benedetti Michelangeli after entering the stage started taking his piano apart, played a couple of notes, took it apart again, and then got up and left to never return leaving behind a stunned audience. It happened actually more than one. When I was living in Japan, Michelangeli was announced to perform in Ôsaka, so I got ready to purchase tickets to attend one of his extremely rare performances. Only, Michelangeli never played. In consequence, Japanese customs confiscated his piano for breach of contract, and Michelangeli vowed to never set foot again in Japan, and he never did. For him as an artist of a supreme level, Japan's surgical and cold economic reaction was beneath the understanding of what a true artist represents. Does the fact that he either did not show up or only played a couple of notes allow us to consider this as a "reality check" that he couldn't really play the piano very well ?

No doubt kata as a demonstration or performance rarely is in conditions that are ideal for everyone, but they should at least reach a certain level, and people should receive some level of accomodation if you want a "reality check" to really reflect "reality" and not "sabotage". Besides that, you also must consider that for some this "reality check" simply is no longer possible due to old age and physical impairments.

Some years ago I was on a nage-no-kata course with Abe Ichirô at that time still 9th dan. I was helping out together with Satô Tadashi, and at one point Abe wanted to correct kata-guruma, so he wanted to do it but couldn't get up. He tried again, but had to give up. Is that a "reality check" that he can't do nage-no-kata ?  Could we if supposedly he had demonstrated the whole kata assign him just a "medium mistake" or a "big mistake" ?  After all if you do the same during your contest, you will be given such a penalization, no ? So, Abe couldn't do it while most of us can. Does this then mean we are all much better?  If it doesn't mean that, then what does it mean and how do you correctly address that an calculate that in. By the way, if I were a judge that time and I would have seen Abe do it, I doubt that anyone else would have gotten a score as high as him. You know why ?  His body position, use of his hara, kuzushi, timing, it was all there. He couldn't complete it, but there is no way one had to correct his position. I remember him also performing ô-soto-gari as part of sode-dori in kime-no-kata. No one of the performers I usually see in kime-no-kata performs it so correctly in terms of body position, center of gravity and other things. It was jûdô, plain and simple.

One could to the same extend elaborate about some of Mifunes's performances. After all there are times he steps back with his left foot in stead of his right, times he realizes he starts walking too soon, and steps back again, all kinds of things. So are these all mistakes "small" or "medium" and can we then legitimately concluded that most of our kata couples today all perform their kata much better than Mifune ?  I assure you they don't, but one also has to have reached the level that one can see beyond those mechanics and take them for what they are. They are not even relevant anymore at that level. Who cares whether he steps back first with his left or right foot. How would that even be relevant to the core or objective of the exercise. The stepping is a convention, and implementation of rational but for the rest it is irrelevant to the core of the kata.

As long as the judging system is not equipped to distinguish these issues, and have the people who have long transcended the constraints of the mechanical approach to kata, it can't be evaluated and constantly produces false results. one could say that that result is winning or not winning a medal, but it is not limited to that. If the objective of kata is indeed improving someone's jûdô then such erroneous conclusion holds judgement on the level of the jûdôka involved. In this case that would imply that most people who participate in kata today are than much better jûdôka than Mifune, since few will ... "make the mistake of" stepping back with the left foot first.

That being said, as you know, when we are critical, we often get angry responses saying "can you do it better". So, it is only fair that we look for improvements rather than to be just negative. I think that some problems can be resolved, but I think others can't as some of the problems are inherent to the nature of the beast.

When I look back at the phenomenon of kata competitions and want to make an evolution I come to the following conclusions. I too competed in kata, but I found it very frustrating. You know why ?  I could not find any partner available whose kata level I deemed sufficient, or they already formed part of a known kata couple and were thus "occupied". The offers I did receive I rejected, all of them. You know why I started competing ?  I did to create and external motivation towards having to know every possible detail and research everything about it. Even though I competed in it far less than you, I did achieve the motivational part and researched kata to the bone. To some extent I recognize the same in you. your participation in kata competition has brought with a whole plethora of different things and you actively research, discuss, communicate, teach. But I don't think that your development in response to participating in kata competition is how the average people develop. I know many other kata performers besides yourself, and while all of them are devoted, and practice and want to win, I don't think I know of many other people who have grown so much due to this activity. If the effect was the same in everyone, I would probably embrace it more. At the end of the day, I don't think that what I observe really is the effect or merit of kata competition but of yourself. Why ?  Well, you already developed as a very good competitive jûdôka when you were still fighting shiai. I see this more as a product of personality and having been exposed to the right teachers than that this is the merit of kata competition. In other words, even if kata competition would not have existed, you probably would have developed in this field in a way far above average too. It's all speculation, but it is still what I believe.

CK Sensei wrote, quote..............."your participation in kata competition has brought with a whole plethora of different things and you actively research, discuss, communicate, teach. But I don't think that your development in response to participating in kata competition is how the average people develop. I know many other kata performers besides yourself, and while all of them are devoted, and practice and want to win, I don't think I know of anyone else who has grown so much due to this activity. If the effect was the same in everyone, I would probably embrace it more. At the end of the day, I don't think that what I observe really is the effect or merit of kata competition but of yourself. Why ? Well, you already developed as a very good competitive jûdôka when you were still fighting shiai. I see this more as a product of personality and having been exposed to the right teachers than that this is the merit of kata competition. In other words, even if kata competition would not have existed, you probably would have developed in this field in a way far above average too. It's all speculation, but it is still what I believe".

I agree entirely with this point. I also know from direct debates with WDax Sensei that he knows his judo from A to Z. With great reverence and respect to him I do think that because he can distinguish kata from choreography that all people involved in kata competition are the same. As CK points out this is so far from the truth.

I also know that there are now 'kata judo clubs', these clubs focus on kata at the expense of all other parts of judo. An overactive ambition or desire to win in shiai is what reduced judo shiai to a torro-torro pat-a-cake pat-a-cake wrestling bout. It is why judo means to Joe public that judo is no longer a martial way but a wrestling sport and not a very skilful one at that.

I also know that some kata competitors now concentrate on kata to the degree that they exclude other parts of judo training. This win the medal attitude is not suited to shiai and certainly not suited to the philosophy of kata. Imbalance in ones training means an imbalance in ones life.

Kata cannot be perfect. No two partners are the same. No two partners even perform the same action the same twice in a row. The variables in kata are so vast unless they are understood as described by CK sensei, then what exactly are those judges judging? They are NOT judging they are comparing And comparing one performance after another is missing the point of kata by a mile. IF kata is to be judged it has to be judged on its principles being demonstrated not the time allocated or the perfection of keeping to a set line on the tatami.

I used the analogy of music many years on the old JF. Each orchestra has the same score, same number of players in the orchestra but with 5 different conductors. Each conductor will have the same orchestra play 5 different renditions of exactly the same score. That is art. Art cannot always be judged when placed side by side. What is more what I may like another may not. That is also art. To take this recognition of art to the finite level what a person well educated in music will hear are five different renditions but accept all are equally as brilliant even though different.

An honest question for our friends in kata competition. if I where to partake in a local event do you realise where I would come? Bottom of the list. HOW could that be when I have been taught by the best kata and judo teachers in the world going back 52 years and studied, taught kata all over the world? HOW would I fail the kata test? I have every confidence CK Sensei would also be laughed of the tatami also. Would you consider either of us kata uneducated and incapable of performing the kata? What is it we would do that would cause us to fail?
My own answer would be that neither of us would try to bend a kata to fit a mathematical equation that scored good points but was, in reality, absolutely false and of zero benefit to our judo learning and progression plus failed to obey the set regulations of dynamic action reaction that is the art of judo. If two renditions of kata look the same something is very wrong.

Very complex subject and one that needs cool headed debate and thought.

Mike

Hanon wrote:I have every confidence CK Sensei would also be laughed of the tatami also.  

Hey, hey, hey !!  I am not usually "laughed off the tatami" and I don't think that would happen. "Retaliated against" that is something else, but "laughed off the tatami", that I don't think so.  

Many, many years ago at the Kôdôkan after Yamamoto Shirô, now 9th dan had taught a class about Koshiki-no-kata, I just couldn't take it anymore and told that what he was teaching made no sense whatsoever. I then demonstrated the kata, and it went very quiet as it was obvious that I intentionally ignored all of he nonsense he had been teaching. I assure you that nobody was laughing. I also can tell you that it was the last time that he was in charge of Koshiki-no-kata and that he was replaced by Daigo. Of course, I can't say or prove that he was replaced because of that, who knows, but ... let's be fair ... is there anyone who regrets that it is Daigo who is now in charge ?  I don't, and so far I haven't met anyone yet who said "Gee, I wish they would take Daigo off the KNK and replace him by so or so".

At least one other forum member here was witness to a Goshinjutsu peformance, which, sure, was not perfect at all, but one thing I can assure you is that all activity on all four mats stopped and it went very quiet and everybody was watching. Is the number of people watching evidence that oneself is beyond every doubt or improvement ?  Of course not, but I doubt the same would happen if the objective was to mainly count the number of steps, be concerned about my position, and make sure that after step number 2, I was looking into the desired direction. It's living kata, and it is not so frequent kata that is alive.

Luckily, our friends wdax, but also heikojr are both fine men who too are honestly committed. Wdax is also correct when he suggests that the system isn't perfect, but in all fairness there is (sometimes) improvement. At least in jû-no-kata there are some judges that can see through the facade of mere aesthetics.

I also wanted to comment on something else I had forgotten to mention: the Kôdôkan DVDs.

I don't think that it is recommended to consider any of them "standards" in a sense that they are how it should be. But I also think that wdax didn't say that either. I just want to go one step further and point out that they really are "instructional DVDs". Would anyone dare to suggest that an instructional piano or music session is how you want it during the concert or recital.

The Kôdôkan DVDs show unnatural kata, something that is the result of an edited recording, and that is devoid of any emotion. Part of it is culture, but part of it is because it is instruction. To this day, I'd still like my Beethoven played by Kempff, Fischer, Backhaus, Richter, , Schnabel, Serkin, Arrau or conducted by Furtwängler or Kleiber, instead of listening to an instructional Beethoven CD. Yes, you are right there is no Japanese in that list who might well be able to copy all those mechanics and meanwhile maintain an emotionless face, yet unable to play anything remotely remniscent of Beethoven. One does not need to be theatrical either, but there needs to be spirit. The Kôdôkan DVDs have no spirit since they were also not made for that purpose. They were made for lower stage instruction of mechanics. Not everyone sees this, so can one be surprised that in today's kata performance we see kata-guruma performed in a way that tori stands ther, ukemi falls on his shoulders , he stand ups has uke lying there en then drops him. So, he went through most of the mechanical steps but it hasn't anything to do with kata-guruma. The judging system is not compensating for that either, because it cannot detect any mistake when compared to the Kôdôkan DVD, and it doesn't subtract for lack of spirit either, because they now believe that that truly is what Kata-guruma is about. Does that make any sense ? What I am saying, is that as wdax has correctly pointed out judges may subtract marks for lack of spirit, but is there "lack of spirit" to many of them when there is no difference between how the performers do it and how they do it on the Kôdôkan DVD. In other words, if not on the Kôdôkan DVD and not shown by many performers, then what are the standards of "spirit" or "lack of spirit" for the judges. After all, one needs to know what "proper spirit" is before one can detect its presence or lack thereof. One has to have learnt it, and even if relatively old or physically imparied, one should still be able to show spirit.

This is a problem, and one with no easy solution. From one side, the judge should be allowed to give a 2 or something for that reason, but from the other side, the competitor now will argue that he has no idea why the judge gave him such a low score because he did it exactly as on the DVD and is not aware of any fault. This is a problem, and if you are too vague, and say, "we want good jûdô", it still won't be helpful. In some kata it becomes even more complicated since the objective becomes abstract. In jû-no-kata "jû" obviously is important, but there also is already a presence of muga mushin, perhaps not to the xtent of KNK. These are very advanced concepts, and I do not believe that the majority of the kata judges can even begin to grasp that concept in its full understanding as how precisely it applies to kata. So how can they evaluate it, when one knows that in terms of importance it is far more important than the mechanics of a kata ?

Also of some interest is that I was told years ago by a very reliable source within the Kôdôkan that they were exploring the idea of creating a new kata as a successor to Kôdôkan goshinjutsu. Kôdôkan goshinjutsu is modern self-defense, but "modern" obviously does not mean 2013. "Modern Japan" refers to something starting in the second half of the 19th century. Don't forget that the Medieval period in Japan lasted until the 19th century, which is 400 years longer than in Europe ! So "modern" is not the same as "contemporary". Goshinjutsu probably is not "contemporary". There are no defenses in there against someone attacking you with an iPhone, with a taser, with the chain of a bike, with a phone book, of with an injection needle filled with Polonium-210 ... obviously ... so no not "contemporary" self defense. I mentioned this before on the old JudoForum, but then of course, someone who also visited the Kôdôkan asked someone and that person had never heard of the idea, hence conclusion ... what I said was not true. Now, I don't know what the present status is of the project and if it has been buried completely, I don't know. I would imagine that given the shit they had with goshinjutsu, and the fact that they today are not very interested anymore in anything but sports, I doubt that the project will be realized any time soon.

Care to elaborate on.................."I would imagine that given the shit they had with goshinjutsu,"............. 

Okay you would not be laughed off the tatami, that was a poor joke. You would be shot while still on it8)

This is a very sensitive subject. Not sure how this thread could turn?  

Mike

C.K.,

we are 100% on the same page, but talking a about slightly different things, which of course belong together. I was trying to explain the basic ideas behind the current scoring-system, while you are explaining the problems of the implementation of these ideas in a judging-system.

I see two major problems, which lead to the fact, that the final score for a kata is always only an approximation:

- There will always be some imperfections of the system
- There will always be a human factor

1.) Correctly you point out, that it is almost impossible to exactly quantify the quality of a movement. We don´t have measurement devices so the only thing left is an expert rating, which in itself is a problem, I will address below.

Additionally one of judo´s goal is to reach a state of expertise where all movements are done with a complete body/mind integration. This is almost impossible to observe, so how can it be quantified? The answer is: it´s simply not possible.

2.) The human factor! There are some issues:

- We can only see an action, but the quality of the technique is set by the forces between Uke and Tori. So the perception of the judges is always indirect.

- You never find five judges of equal knowledge, so the aggregation of the individual scores to to final result is questionable (your example five men with IQ of 130 vs one man with IQ of 180)

- Selecting of judges is always a compromise (because they are sent by their federations and maybe you need the best judges for Nage-no-Kata at the same time for Katame on the other tatami)

- Judges knowing some teams well - and others not

- Judges sometimes tend to give scores similar to their collegues

- etc.

Knowing this, we can start to develop a system, that provides the best possible approximation. We need:

- guidelines and definitions for the expert-rating (see my initial post)
- breaking them down and apply them to all actions in all kata (if I include Koshiki-no-Kata we have we have without opening and closing all together 107 forms) and for each of them a lot of different variations, "accident" and errors.
- training of judges, so that they can apply this all in the best possible way.

Given to the fact, how problematic kata-teaching in the past was (and still in present is) this is a huge task, which takes a lot of time. Maybe in 5 or 10 years, we are were we should be. In Europe it is really improving - taking a look at the scores at the WC in Kyoto prevents me from saying the same about other continents.

Where are we now?

• We have the general guidelines - see above.
  
• We have some judges, who really can apply these guidelines in their scores
  
• We still have some parts of the guidelines and the application to improve
  
• There are still a lot of judges, who need more training and experience
  
• And everybody knows, that their always will be certain limitations in the judging

It is interesting this.  I haven entered kata competitions and taken medals, but that was long before the current popularity of kata competition. It is an emotive subject, but I am always impressed with the knowledge and sincerity that wdax-sensei writes about the issue.

Just an anecdote for me.  I was first taught the Kodokan Goshinjutsu by John Cornish - a 7 dan judoka, 8 dan aikidoka, a Kodokan kenshusei who had Tomiki and Kotani as his teachers.  Every aspect of being uke to JC was painful - every technique hurt, was effective and compact.  Cornish learnt his Goshinjutsu from the original principle architect of the kata, yet his techniques, in many cases bear not that much resemblance in terms of their execution and finish to the current judging standard.  If I performed a Kodokan Goshinjutsu in that manner I would be told, that this and that was wrong.  Is it really wrong?

I like to think that all the Kodokan kata have latitude in their execution.  Too much deviation and it becomes unrecognisable, something else, and wrong.  However I am not convinced that the current fractal, highly granular means of scoring is right too, as perhaps it does not take into account the "special sauce" that distinguishes a good by-the-book performance, from a great one!

The other problem I have is the dropping of the high and low outlier scores. What if they are correct? Say if the person who gave the low, or high score was Daigo? Statistically this approach is flawed and drives all scoring towards the centre and does not necessarily produce the right winner.

Jonesy wrote:(....)
I like to think that all the Kodokan kata have latitude in their execution.  Too much deviation and it becomes unrecognisable, something else, and wrong.  However I am not convinced that the current fractal, highly granular means of scoring is right too, as perhaps it does not take into account the "special sauce" that distinguishes a good by-the-book performance, from a great one!

This leads to another very important point. Kata is IMHO for the development of one´s judo useless, if you do not try to transfer the lessons you have learned in kata into other parts of judo - mainly randori and self-defence. So everybody must deviate from the textbook-kata into free exercise. But there is a need of a basis, where we can return - otherwise we loose the fundament of (our) judo. In kata-competition INHO the fundament should be demonstrated, not the personal deviation - of course a perfect adaption of the fundament to one´s body (but I think nobody questions this).

Jonesy wrote:The other problem I have is the dropping of the high and low outlier scores. What if they are correct? Say if the person who gave the low, or high score was Daigo? Statistically this approach is flawed and drives all scoring towards the centre and does not necessarily produce the right winner.

Sometimes the score of the best judge is dropped.... but in practice, this produces better results, then asses all five scores. But I said very often better three really competent judges than three competent plus two other less competent.....

wdax wrote:
You never find five judges of equal knowledge, so the aggregation of the individual scores to to final result is questionable

Why do we add the total scores to determine the winner? The scoring key provides a way for each judge to form an overall assessment of each team. At the end of the competition, each judge should have one vote for each team in the competition. A strong bias of one judge should not overshadow the scores of the other judges. Similar to shiai competition, a strong opinion of ippon from the referee does not influence the score if the two judges say no score. The majority of no score prevails.

A better system would be one where the judges’ scores are converted to ordinals. The team that wins is the one that gets the most first place votes from the judges based on their individual scores. This way, a judge that might be considered an extremely easy or hard marker does not distort the overall scoring. It should be the ordering of the teams for that competition that is important not the raw scores.

This is a problem in our area as we can often only find three available judges. You have to keep all three scores. One of the judges may be significantly less experienced and may have a significant impact if they miss something big. By using ordinals, you can have some control over the bias of a weak judge.

cp2000 wrote:

Why do we add the total scores to determine the winner? The scoring key provides a way for each judge to form an overall assessment of each team. At the end of the competition, each judge should have one vote for each team in the competition. A strong bias of one judge should not overshadow the scores of the other judges. Similar to shiai competition, a strong opinion of ippon from the referee does not influence the score if the two judges say no score. The majority of no score prevails.

A better system would be one where the judges’ scores are converted to ordinals. The team that wins is the one that gets the most first place votes from the judges based on their individual scores. This way, a judge that might be considered an extremely easy or hard marker does not distort the overall scoring. It should be the ordering of the teams for that competition that is important not the raw scores.

This is a problem in our area as we can often only find three available judges. You have to keep all three scores. One of the judges may be significantly less experienced and may have a significant impact if they miss something big. By using ordinals, you can have some control over the bias of a weak judge.

Have you studied the mathematical and statistical analysis that has appeared in several scholarly papers that exist on ice skating and gymnastics ? These two sports have a far longer history than such scoring in judo. I would recommend that you and the IJF do. When you do and you understand what they are trying to convey, you will see that certain problems cannot be overcome as they are important mathematical issues. These are not the kind of things that need to be addressed by "I think" or "I feel"; there is some hard evidence that supports the impossibility of the system.

In your reasoning you make some of the same mistakes as the IJF does. You write "A strong bias of one judge should not overshadow the scores of the other judges." Why shouldn't it ? You are assuming that "a strong bias of one judge" would be a bias in error. That is not certain at all. The term 'bias' has a pejorative connotation suggesting that it is willfully incorrect; perhaps 'deviation' should be a better choice. One judge completely deviating from the other 4 could be right and all four could be wrong. It is comparable to asking a question on this forum and arguing that everyone's opinion is equally valid which would imply that that common denominator is the right answer. That is absurd. The correct answer in many cases may be that of a single person, likely the best qualified person with the most insight to the detriment of everyone else. When you take 4 of the average kata judges and as a 5th I would put Daigo-sensei in there, or let's say Okano Isao-sensei, who has no kata judging certifications at all and who is 'only' a 6th dan which may be lower than all of the other kata judges, then ... I am very sorry ... but chances are considerable that the opinions of "all 4 qualified kata judges" are probably as valuable as trying to clean up the Fukushima disaster with a toothbrush.

Apart from the above mathematical problem the scoring itself is problematic and this at two levels. The first level is the rules that dictate the scoring. The second level is the application of the scoring as dictated by the rules. It is possible to give examples of a near perfect performance of kata that would score far less than a very mediocre performance. An example of this would be to intentionally perform the entire nage-no-kata with two steps instead of three with perfect timing, perfect tsukuri, perfect kuzushi and perfect kake and control.

The quality of the judges is problematic in itself. In many cases the top kata compeititors are going to have a far higher level of understanding of kata than the majority of the judges. The level of understanding of kata among kata judges is very poor, as it is among the average jûdôka. Today's jûdô education lacks that what is necessary to achieve that level, and the same applies to those giving out the qualifications. You have a system that is rotten from top to bottom. It is not possible to properly assess kata based on a set of rules; but even if it were, it still would require a superior analysis of that what is taken place. Most jûdôka do not have this. Another example is that kata in this way are performed as a mostly aesthetic discipline. Where would kata judges have obtained such level of understanding of aesthetics as it isn't Western aesthetics that govern budô but Japanese aesthetics, which is a rather complicated area and requires years of study, considerable reading of and study of academic texts on Japanese aesthetics and field study of the whole plethora of Japanese literature, chadô, bijutsu, kabuki, nô to grasp the different from in which some of those qualities are expressed. This is not knowledge that even the average educated Japanese native still has today, just to illustrate the level that is required to do this properly.

To have the current system do what it does, it needs to be brought down to the lowest possible level of understanding, which is exactly what is happening, with its detrimental effect affecting anyone who participates and whose understanding exceeds that level. If it exceeds that level only a bit, it will still be grasped by the best judges whose own personal level exceeds that of the average judge. If it exceeds the level considerably, its proper judgment will be up in the air out of reach of most.

Kata in jûdô as was defined by its founder is per definition an ikimono. Neither the IJF, nor the judges (at least in their judging role; maybe some do individually) get that. If I would tell this to the IJF Committee or a group of kata judges, they would all sit there, and maybe, just maybe, once in a blue moon one would actually put up his hand and ask "could you please explain what ikimono means, because I am not familiar with that term". That is the problem one faces. Not only is there a lack of knowledge, but even an absence or refusal to admit that lack of knowledge.

What is the ultimate criterion to judge kata ? It's the same as the criterion for randori. The criterion to assess randori is not whether you won, not even whether you scored ippon. The intent of randori and its assessing criterion is whether your jûdô improved. That is also the criterion for kata. It is meant to improve your jûdô, and that improvement depending on the nature of that kata needs to be appreciated in a context of rentai-hô, shôbu-hô, and shûshin-hô, and allowing isshinhô. This is in a relative reflective way to oneself impossible to assess in a contest since it is a longitudinal process. One could conduct a comparative approach between different performers, but this is somewhat counter-educational as in essence --if accurately applied-- would only indicate the difference in evolution between different people. This difference we should already know if the dan-rank would have been assigned in an accurate way. In that case we know that the evolution of a 5th dan is/should be further than that of a 3rd dan. Yes, wishful thinking. Nevertheless, the comparative approach is problematic because it fails to grasp that what is at issue, i.e. the evolution of oneself. After all it is very well possible that your personal evolution has been steep, amazing but that you still are less evolved than someone else who has not shown any evolution over years. Under that proposed judgment it falsely suggest that in education it would be more important to score more than someone else than it is for you to learn and to evolve. Sure, there will be devil's advocates who will claim that when someone else is better (the competitive model) that this will stimulate you to do better too. This is not certain at all and in this context is even false since you have no evidence as to who is actually 'better', only evidence to someone "scoring higher" ... which makes the problem even more serious as the person scoring higher could in effect be worse.

So, it is not possible to assess in a contest the evolution of an individual unless one takes performance in previous contests in consideration. Considering those limitations, the only possibility is a comparative approach, which either uses the other participating teams as standard, or which uses a separate standard, for example, a Kôdôkan DVD as standard. This in itself is problematic not only because it is contradictory to kata as an ikimono, but also a very serious misunderstanding. The Kôdôkan DVDs are not at all some mythic standard that should be achieved. What they are is a beginner's tool that tries to convey the mechanistic basis of kata on which kata needs to be built. I would hope that people to no actually perform their kata as is on those DVDs, just like I hope that people do not try to copy the randori of someone else recorded on a DVD when they do randori.

To end my response, I would like to refer to a statement by the Chair of the EJU's Kata Committee made during the recent EJU's Symposium in Montpellier. When asked by professor Sacripanti to reflect on the couple of years of kata competition which have now been in existence for several years, he said that it was a complete failure and that it did not do what thy had hoped for, and instead is counterproductive. Later in a personal conversation with me he said that in the way they have been doing kata competition they tried something out because they had no alternative and really didn't know what else to do. He remained remarkably quiet when I asked him "how 'bout starting by requiring some competent people". He said that they had asked the Kôdôkan for help (why I am not sure, but this is a problem in itself since it assumes that all experts would be at the Kôdôkan which is hardly true; in reality even in Japan, most of the true experts are not even associated with the Kôdôkan, and this for obvious reasons ...), but that they didn't provide much help. When I asked him who, and whether they had asked Daigo-sensei, he said that they had met Daigo in Kyôto but that he was not interested in helping. My conclusion was that indeed with the rank of 10th dan wisdom comes ...

O.K. that is not constructive, but I am sure that Daigo-sensei in an attempt to circumvent having to insult anyone, realizes that in a rotten system you can't make it work by here and there adjusting something. I don't know this, but it may be that Daigo-sensei has his doubts the whole concept even though, remarkably, the Kôdôkan has its yearly All Japan Kata Championships too, something I don't quite get the reason for its existence of. Perhaps they see it more as a stimulant towards continued practice rather than a crowning of champions, and at least the environment and those present are far more serene than at IJF or EJU kata championships and tournaments. Still, factually, its existence both in Japan and abroad remain a remarkable thing in the light of Kanô's views, but given the sportification of jûdô that has been going on for many decades now in Japan and abroad, it shouldn't be a surprise.