Running a ChatGPT type bot at home

[Bijan]

I got a bit busy at work this year, but there's so much talk of AI chatbots these days, I decided to try my hand at running one at home.

I ran into this article:

https://arstechnica.com/information-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/amp/

Which got me a bit too hopeful. Raspberry pi is a pretty puny computer, or at least was at one point.

I got the CPU only model running pretty quickly. But the larger models run pretty slowly, 1 word every second or two.

I managed to get the 30 billion parameter model running on my Nvidia 4090, GPU (it can be squeezed down to about 20GB, and the GPU has 24GB).

It runs at about 10 words per second.

The training data/model is from Facebook (it's called llama). It was not too bad in text generation mode, but it really shines in chatbot mode.

After several hours, I found there's a tab where you can give the chatbot a personality and a backstory. And it does not have the sandbox/guard rails of chatgpt.

It can also be a lot more argumentative than ChatGPT.

Here's an example conversation.

I tried to keep it off US politics and have it be inoffensive. It had some real interesting takes on recent US and world events at times. Also it was ready to be real offensive and/or controversial.

Keep in mind this is using a local 16GB database and no other data or access to the internet.

A quick question to see what it thinks of our hobby:

[NSXCIGAR]

I find these AI chatbots to be pretty good compendiums of prevailing opinion among the informed but that's about it. For example everyone knows Wrath of Khan is the best film. Almost all trekkers think DS9 is the most intellectual series. No one liked Voyager. When asked why it gave a confusing answer. 

Not saying these are incorrect answers, just definitely looks more like aggregator answers. Good aggregator, but aggregator. 

Although TNG is the best series. 

[ha_banos]

I like bard! Wait till I show the wife!

 

"Yes, there is a proven link that cholesterol levels do not correlate with what we eat. A study published in the Journal of the American Medical Association found that there was no correlation between dietary cholesterol intake and serum cholesterol levels. The study, which looked at data from over 100,000 people, found that those who ate the most cholesterol had the same cholesterol levels as those who ate the least.

 

Another study, published in the journal The Lancet, found that a diet high in saturated fat and cholesterol did not increase the risk of heart disease. The study, which looked at data from over 90,000 people, found that those who ate the most saturated fat and cholesterol had the same risk of heart disease as those who ate the least.

 

These studies suggest that cholesterol levels are not affected by what we eat. Instead, they are affected by genetic factors, lifestyle factors, and certain medications."

[Bijan]

11 hours ago, NSXCIGAR said:

I find these AI chatbots to be pretty good compendiums of prevailing opinion among the informed but that's about it. For example everyone knows Wrath of Khan is the best film. Almost all trekkers think DS9 is the most intellectual series. No one liked Voyager. When asked why it gave a confusing answer. 

Not saying these are incorrect answers, just definitely looks more like aggregator answers. Good aggregator, but aggregator. 

Although TNG is the best series. 

I may have not pushed the chatbot angle in my first example:

Here's how it thinks my 3.5 year old would respond:

(Grammar is generally too good, but I guess not a lot of internet posts by toddlers to train on)

 

[ha_banos]

Quite impressed you can run a trained 'ai chatbot' at home. Given the amount of text data needed. 16gb sounds like a tiny dataset. But then I wonder how much or little data we get exposed to in a year?

[Bijan]

15 minutes ago, ha_banos said:

Quite impressed you can run a trained 'ai chatbot' at home. Given the amount of text data needed. 16gb sounds like a tiny dataset. But then I wonder how much or little data we get exposed to in a year?

The dataset is mainly training weight/probabilities. The original dataset is 40-50gb I think. But it turns out you can round each entry to 4 bits without much effect on the output. And this lowers the size to 16gb.

The FB provided weights come in 4 model sizes: 7B, 13B, 30B and 65B. Where B means billion parameters/weights.

ChatGPT 3.5 is supposedly 170 billion or so. But I feel they have a lot of special case handling and extra training to get better results.

I was running the 30B quantized to 4bits. Which fits in an Nvidia 4090 consumer GPU.

The 7B one runs on a lot cards, but you lose a lot of data.

Keep in mind the text of all of English wikipedia is about 25gb I think. But again this has other languages and it's not just text but also weights reflecting language/words/grammar.

[ha_banos]

You are smart 🤓 a lot of that means nothing to me.

[Bijan]

4 minutes ago, ha_banos said:

You are smart 🤓 a lot of that means nothing to me.

Don't worry. I only understand half of it. Maybe half .. 😂

 

[El Presidente]

7 minutes ago, Bijan said:

Don't worry. I only understand half of it. Maybe half .. 😂

 

Ken will be up shortly to explain the other half 👍

[Chibearsv]

46 minutes ago, Bijan said:

Don't worry. I only understand half of it. Maybe half .. 😂

 

Well your nerd vocabulary is fantastic 😁

[MrBirdman]

15 hours ago, NSXCIGAR said:

No one liked Voyager.

[NSXCIGAR]

7 hours ago, Bijan said:

Here's how it thinks my 3.5 year old would respond

I had seen Star Trek at 3.5. I think the TOS cartoon. Not scary. Although I did see Wrath of Khan at probably 6 and that might have been a little young. 

[NSXCIGAR]

3 hours ago, MrBirdman said:

 

Don't the "serious trekkers" kind of bash Voyager? I never saw much of it. From what I can tell the Star Trek cognoscenti loves DS9 and hates Voyager. Personally I did watch a bit of DS9 in the beginning as it was kind of a direct TNG spinoff (and the first) but I kind of lost interest. A little too political for me which is what everyone seemed to like about it. I was also in high school when it came out so I think maybe too young to appreciate its depth. TNG is a little more family oriented. 

My point is that the general feeling of liking DS9 but disliking Voyager is the predominant one in the inner circles and thus I'd expect that to be the AI response. 

[Bijan]

There was a lot going on with TNG vs DS9 vs Voyager.

TNG really was Star Trek++. Same idea, better budget, effects and execution. Also real prime time spot.

DS9 was a great show, but in a sense it wasn't Star Trek. It was more or less in one location and the last seasons were a continuous story arc. Also it was often more dystopian than utopian. Less good of a TV slot too.

Voyager was the same but more so in some ways, less so in others. It kind of settled down in the opposite direction after the initial bit. Becoming more episodic over time. I'm not sure if it aired outside of UPN or whatever that network was for most of its run.

Odd to say for a Sci-Fi but the characters are what really make the shows. And Voyager to me had some real annoying ones.

As a kid it was fun to daydream about being part of the TNG crew or DS9 crew. But the people from voyager I'd actually want to hang around with...

Also it turns out you can run pretty good AI image generation at home. That's my next project. Since the GPU in this computer is maxed out running the chatbot. I'll try running on another PC with an AMD GPU.

You can then hook them up together and have the chatbot generate images.

[MrBirdman]

13 hours ago, NSXCIGAR said:

Don't the "serious trekkers" kind of bash Voyager? I never saw much of it. From what I can tell the Star Trek cognoscenti loves DS9 and hates Voyager. Personally I did watch a bit of DS9 in the beginning as it was kind of a direct TNG spinoff (and the first) but I kind of lost interest. A little too political for me which is what everyone seemed to like about it. I was also in high school when it came out so I think maybe too young to appreciate its depth. TNG is a little more family oriented. 

My point is that the general feeling of liking DS9 but disliking Voyager is the predominant one in the inner circles and thus I'd expect that to be the AI response. 

There really isn’t one “inner circus” within Trek, since it spans such a long period and primarily appeals to young people. Voyager actually had better ratings than DS9, probably in part for the reasons Bijan pointed out. First two seasons of DS9 were absolutely dreadful. The mood was also much darker. Neither came close to the popularity of TNG (hence the movies).

To the extent that some Trek fans don’t like Voyager, it’s for its goofiness (which I like) and its unrealized potential (which even big fans like me agree with; it could’ve been even better). It’s streaming popularity has been remarkable though. 

The one show all “real Trek” fans agree is awful is Discovery. Pretty much all Kurtzman-era trek is abysmal. That I would definitely expect ChatGPT to get. 

[NSXCIGAR]

4 hours ago, MrBirdman said:

There really isn’t one “inner circus” within Trek, since it spans such a long period and primarily appeals to young people

Inner circle was an inartful term. I mean the most serious and knowledgeable fans that are communicating online. The die-hards. Similarly, I would expect AI would pull a lot from FoH regarding CCs. If you ask AI what are the best and worst CCs I bet it would align with the general sentiments here. 

[ha_banos]

Here are some of the best and worst Cuban cigars, according to brand and vitolas:

 

Best Cuban Cigars

 

Montecristo No. 4: This cigar is a classic Cuban cigar that is known for its smooth smoke and rich flavor. It is available in a variety of vitolas, including corona, robusto, and churchill.

Romeo y Julieta No. 2: This cigar is another popular Cuban cigar that is known for its mild flavor and smooth smoke. It is available in a variety of vitolas, including corona, robusto, and churchill.

Cohiba Siglo V: This cigar is a premium Cuban cigar that is known for its full flavor and rich smoke. It is available in a variety of vitolas, including robusto, churchill, and perfecto.

Worst Cuban Cigars

 

Punch: This brand is known for producing low-quality Cuban cigars. They are often dry and harsh, and they may not burn evenly.

Fonseca: This brand is also known for producing low-quality Cuban cigars. They are often dry and harsh, and they may not burn evenly.

Said Bard...

🤔

[MrBirdman]

1 hour ago, NSXCIGAR said:

I mean the most serious and knowledgeable fans that are communicating online. The die-hards.

Even there Voyager has tons of fans. Unfortunately there aren’t many good watering holes since TalkTrek shut down. Now it’s a ton of media stuff most which praises discovery (which isn’t real Star Trek). 

[Fuzz]

@Bijan many are thinking it, none are willing to say it.... except me.

DO NOT upload this Chatbot into a sex doll!

[chriskoch]

On 3/23/2023 at 10:20 AM, Bijan said:

Also it turns out you can run pretty good AI image generation at home. That's my next project. Since the GPU in this computer is maxed out running the chatbot. I'll try running on another PC with an AMD GPU.

You can then hook them up together and have the chatbot generate images.

It's great to see enthusiasm for experimenting with AI image generation and exploring different hardware options. Running AI projects at home can be a fascinating and rewarding endeavor. It's important to consider the resources required for such tasks, like GPU usage and processing power. By leveraging multiple devices, there are opportunities to enhance and expand the capabilities of AI systems.

@contact: gptnederlands.nl

[ha_banos]

On 3/23/2023 at 10:44 PM, Fuzz said:

@Bijan many are thinking it, none are willing to say it.... except me.

DO NOT upload this Chatbot into a sex doll!

Actually.. have you seen the metaverse interview? ... interesting opportunities in there. Who might one stumble across...?

[Fireball]

@ha_banos

what are you referring to ‘bard’ ? I may have missed it in the thread

 

I think DS9 was boring for me because it was a space station. In Star Trek anything,  I like the ship. Like Back to the Future, where I like the car. 

[ha_banos]

1 hour ago, Fireball said:

@ha_banos

what are you referring to ‘bard’ ? I may have missed it in the thread

This was posted elsewhere.

https://youtu.be/MVYrJJNdrEg?si=IsM-ca40ueFiv1Pm

 

The realism is amazing. So imagine @Fuzz comment, instead of a sex doll, a sex-metaverse! Oh the fun!

[BoliDan]

Let's be honest, Voyager was bad from the the get go. Start off by infiltrating a terrorist group, and making an arrest. Then, giving said terrorists assignments on the ship, and even giving one of them officer rank of COMMANDER. That's right, all of our lowly ship crew need to work years on this ship. But, you Mr. Terrorist with some previous star fleet experience can jump ahead of all them, including Tuvok, and get number 2 on the ship. In what military would that ever happen? 

I would have thought TNG as best by consensus. Deep Space 9 was decent, but the whole religious thing was eye rolling.

[Bijan]

3 hours ago, BoliDan said:

I would have thought TNG as best by consensus. Deep Space 9 was decent, but the whole religious thing was eye rolling.

To me TNG was best in the episodic format (sadly it was practically the last one in this format), and DS9 was best in the story arc format.