Expected Accuracy
Expected accuracy is the degree of confidence you can have that Tiberias will classify your test text accurately. The expected accuracy score for your experiment is displayed on the Analysis of Classes Page.
This topic describes how Tiberias determines the expected accuracy score during class analysis.
How does it work?
Given the text classes you have defined and the feature sets you have selected, Tiberias begins its analysis of the features in each of the text classes. Tiberias assigns weights to each of the hundreds (and often thousands) of features in each text class as it builds the best possible classification model.
The weights that Tiberias assigns to each feature are based on the following:
- how often a feature appears in each class
- the prevalence of each feature in one text class relative to its prevalence in another
Features that appear in relatively high concentration in one text class but hardly at all in another, receive a high weighting for classification.
For example: When we compare Genesis and Psalms for analysis, the word ויאמר receives a high weight as a Genesis marker, because it is overwhelmingly more prevalent in Genesis than in Psalms, while לדוד will receive a high weight as a marker of Psalms, as it appears exclusively in that text class.
What do the numbers mean?
What can we understand from an expected accuracy score of 100% vs. a score of 50%?
Intuitively, it is easy to tell the difference between a passage from Genesis and a passage from Psalms, and not unsurprisingly the classification model that Tiberias builds to analyze these two book successfully classifies the chapters of both books 100% of the time. But not all text classes are as distinct.
Consider two text classes, one consisting of the even chapters of Genesis, and one the odd chapters of that book. When Tiberias analyzes these two text classes, it achieves an expected accuracy percentage of only 50%. That is because the division of Genesis into text classes of even and odd chapters is entirely random. Tasked with creating a classification model for this random division of the texts, Tiberias can do no better than the flip of a coin.
What can expected accuracy tell us about the biblical texts under analysis?
Let’s start with what it can’t tell us.
It is critical to understand the limits of what the expected accuracy percentage can tell us about the biblical texts under analysis.
Two books by two different authors—in any language—may well be analyzed by the algorithm employed by Tiberias, and yield an expected accuracy percentage of 100%. But a high expected accuracy percentage cannot automatically be interpreted as a sign that the texts under analysis were composed by different authors.
Example: To understand why this is so, consider the following: You write a book about art history and it is published by an American publisher. A second printing of the book is published by a British publisher, who converts all of the American spellings to British ones. Nearly all of the linguistic material found in the two editions is identical. But every page of the American edition contains the word "color” while in the British edition these appear universally as “colour.” The algorithm underlying Tiberias will achieve an expected accuracy of 100%, although the two books are virtually identical because this single tell-tale marker appears on every page of the book.
The conclusion here is critical to understand: The expected accuracy percentage is not a measure of how similar the two books are in style. Rather, it is a measure of how easy it is to classify a text as belonging to one class or another.
The expected accuracy cannot be relied upon by itself to determine that two text classes are indeed from the same text (in the case of a low expected accuracy of classification), or that they were written by two separate authors (in the case of a high expected accuracy of classification). If you identify differences in content and in ideology between two text classes, a high expected accuracy percentage between the two texts can contribute to an argument that the two books were written by different agents. This is an excellent example of how interpretation of the data is ultimately the responsibility of the investigating scholar.
How to boost expected accuracy
To the extent possible, you should aim to make your text classes and test texts as large as possible. The more data Tiberias has to work with, the greater the likelihood that it will achieve a high expected accuracy percentage from its analysis of the text classes, and the greater the likelihood that it will be able to classify your test text with high confidence.
The likelihood of a high expected accuracy percentage may also be greater if you balance the size of your text classes. When text classes are highly imbalanced, the expected accuracy may be somewhat lower than when you define two text classes comprised of similar chapters, but closer to a balanced ratio.
Learn more about the SVM machine learning algorithm employed by Tiberias.