Posts
Introduction:
Bort, is the new and more optimized version of BERT; which came out this october from amazon science. I came to know about it today while parsing amazon science's news on facebook about bort. So Bort is the newest addition to the long list of great LM models with extra-ordinary achievements.
Why is Bort important?
Bort, is a model of 5.5% effective and 16% total size of the original BERT model; and is 20x faster than BERT, while being able to surpass the BERT model in 20 out of 23 tasks; to quote the abstract of the paper,
' it obtains performance improvements of between 0.3% and 31%, absolute, with respect to BERT-large, on multiplepublic natural language understanding (NLU) benchmarks.'
So what made this achievement possible?
The main idea behind creation of Bort is to go beyond the shallow depth of weight pruning, connection deletion or merely factoring the NN into different matrix factorizations and thus distilling it. While methods like knowledge distillation, weight and parameter prunning, connection removal and matrix factorizations have proved significant and created essentially interesting results; like distillbert( which is 40% lesser size, 60% faster and almost 95% accurate as BERT); here in amazon, researchers went ahead and created a more depth.
They tried to find out a solution where they not only essentially shed the model, but become able to reparameterize its structure, and thus they reformed the problem of finding an optimized new network from a neural network.
In a 2020 october paper, Mr. de wynter of alexa, found approximate solution to the computationally hard problem of finding out optimal architectural parameters from a deep neural network.
While the details of this paper is extremely riddled with theoretical computer science and hard maths for me to understand, this paper establishes a approximate algorithm which in turn lets you trade off the brute force time of reaching the optimal solution versus the amount by which you are left off. The process also ensures that only in such a pareto optimal way, you will be on path to be near optimal architectural parameters; in all other case it will be suboptimal at most.
Using this paper's finding, amazon's applied scientist Daniel j.perry and de wynter, found out the optimized architecture of the existing big BERT model. They optimized it using another algorithm called agora, which helps optimize a model by taking a help of a development set. While that algorithm in itself is again complicated, it is, to layman's term, helped the pre-Bort model reach to its Bort-level accuracies, using further developmental training set's help.
Finally they reached the Bert level accuracies and surpassed it in most cases. It is not necessary to say at this point of the article that this is a very different and significant change rather than random trial/error or other empirical procedures people have been following to optimize on Bert for quite a few years from Bert's origination now.
Codes and implementations:
While the codes for the original research are available under the repository Alexa's code for Bort from the authors directly, Huggingface doesn't yet support Bort.
Someone has opened a PR for the new model; but A.d.wynter mentioned
that it is not in their current roadmap to provide a code for that;
although he would like it. Now that is 28 days ago this comment was
done. So I guess we still have to wait a bit for a huggingface version
of the model. But if one is interested then one can download the model
from the alexa repo and work on it I guess.
Final thoughts:
While having Bort out there helps medium level researchers and their
companies like mine to get more power in their hands; research wise
questions can now be raised on what improvements wynter's algorithms can
bring on GPT-x( x>2) and models like google's performer and Big Bird
models.
I will try to contact myself both the authors of Bort and let's see if I can get some of my answers. I will come up with a better analysis of both the papers and if possible showcase some new results if pre-trained models are out in market already. Till then, thanks for reading!
Further resources:
(c) The optimal subarchitecture extraction(OSE) paper
Comments
Post a Comment