GPT-3 is a transformer neural net topology that can be used for natural language processing (NLP) tasks. It is an unsupervised learning approach with the core idea that if you feed the net enough text, it eventually will be able to find patterns. It is developed by special research company OpenAI that is backed by Silicon Valley's who is who. OpenAI has been working and publishing on language models for artificial intelligence for quite a while. Its predecessor GPT-2 was already quite impressive (running on 1.5 billion parameters) but GPT-3 (running on 175 billion parameters) is knocking the socks off. If you wan't to dive into what parameters are Jesse Vig has written a post that explains it nicely.
The texts that it generates are almost indistinguishable from what an original author would have written and its internal architecture of understanding is quite astonishing. If you feed it some input it can predict the rest of the text. In this tweet it was given the first half of a blog post on "How to run an effective board meeting" and GPT-3 wrote a 3-step process on how to recruit board members. Check it out:
Omfg, ok so I fed GPT3 the first half of my
— delian (@zebulgar) July 17, 2020
"How to run an Effective Board Meeting" (first screenshot)
AND IT FUCKIN WROTE UP A 3-STEP PROCESS ON HOW TO RECRUIT BOARD MEMBERS THAT I SHOULD HONESTLY NOW PUT INTO MY DAMN ESSAY (second/third screenshot)
IM LOSING MY MIND pic.twitter.com/BE3GUEVlfi
People are starting to tinker with GPT-3 and creating really interesting experiments. In this tweet someone is creating Layouts right from specification:
This is mind blowing.
— Sharif Shameem (@sharifshameem) July 13, 2020
With GPT-3, I built a layout generator where you just describe any layout you want, and it generates the JSX code for you.
W H A T pic.twitter.com/w8JkrZO4lk
He refined it by creating React components from specification:
I just built a *functioning* React app by describing what I wanted to GPT-3.
— Sharif Shameem (@sharifshameem) July 17, 2020
I'm still in awe. pic.twitter.com/UUKSYz2NJO
Another guy was creating a Figma plugin to create layouts from a specification:
This changes everything. 🤯
— Jordan Singer (@jsngr) July 18, 2020
With GPT-3, I built a Figma plugin to design for you.
I call it "Designer" pic.twitter.com/OzW1sKNLEC
But how reliable does it work? Kevin Lacker has written a nice post "Giving GPT-3 a Turing Test" where he's checking out the rough edges. Max Woolf has also written a post "Tempering Expectations for GPT-3 and OpenAI’s API" where he is diving into limitations - especially noting that GPT-3 is not a public model but it's locked up behind a proprietary API.
Some might argue since the paper was published you can replicate the training of an equivalent model - but as Lambda has pointed out it is not that easy. To be specific: It would cost 355 years and $4,600,000 to train GPT-3 on the lowest-priced GPU cloud on the market. OpenAI actually build a super computer with 285,000 CPU cores, 10,000 GPUs and 400 gigabits per second of network connectivity to make it work (I think that's the engineering secret sauce that you would need to replicate GPT-3 and it's a shame that Europe doesn't have any research facility that's even trying to replicate what has been achieved at OpenAI).
What can I do with GPT-3?
Beside the examples that has been listed above, think about content tasks like:
- Text Summarisation
- Rephrasing Content
- Unique Content Creation
All these tasks are pretty manual tasks that could be half or fully automated in the near future (bye-bye unique content for SEO). But GPT-3 is not only for linguists - since it's eaten the internet for learning it's capable of understanding programming languages too. So other use cases could be:
- Create Entity Relationship models from written use cases
- Context sensitive auto complete, where an IDE could detect your intent and then suggest the boilerplate for you
GPT-3 is still locked behind a closed API. But soon it will be available for the public (for fun and profit). So what would you build with GPT-3?
We're open for business so feel free to contact us for your next project.
Photo by Franki Chimaki